1
$\begingroup$

I have an algorithm that produces a set of real outputs given real inputs. For practical purposes, let's say I have two inputs and one output, and the algorithm can be represented by the function $\phi: \Re² \rightarrow \Re$.

I need to calculate $\frac{\partial^2 \phi(u_1,u_2)}{\partial u_1\partial u_2} \Bigr|_{\bar{u_1}\bar{u_2}}$ but I don't have an explicit formula for $\phi$. My only option is to run the algorithm to obtain the output given the input values.

My original idea was to run the program with a series of values around $\bar{u_1}$ and $\bar{u_2}$, then calculate $\frac{\partial \phi}{\partial u_1}$ at the different fixed $u_2$ values, and finally taking $\frac{\partial }{\partial u_2}$ to that (i.e. taking three values for $u_1$ and three for $u_2$ this would be running the algorithm for all the combinations $\{\bar{u_1} - \delta, \bar{u_1} , \bar{u_1} + \delta\} \times \{\bar{u_2} - \delta, \bar{u_2} , \bar{u_2} + \delta\}$, computing the derivative of $\phi$ with respect to $u_1$ for each of the three fixed $u_2$ values, and finally taking the derivative of that with respect to $u_2$ .)

However I notice that the calculated derivate depends heavily on the spacing between the values of the inputs $u_1$ and $u_2$ (i.e. $\delta$.) Moreover, I don't see it converging as I make the spacing smaller.

Is this a numerical problem or is it that the function is simply not differentiable? How can I tell?

In the following picture there's the resulting plot for $\delta = 0.01$ (left) and $\delta = 0.02$ (right). The derivative was calculated in $\texttt{R}$ using the $\texttt{splines}$ package.

enter image description here

EDIT: If I plot $\phi$ with 21 values for each independent variable I get,

With $\delta = 0.05$

enter image description here

With $\delta = 0.001$

enter image description here

  • 1
    There is an obvious problem with the evaluation of the $\phi$ function. Try reducing the steps to see if you get anything smoother. If not, question the algorithm. If the algorithm is out of the picture, drop the idea of derivating the wild function at all. – 2017-02-28

2 Answers 2

2

In general, you can't tell. More specifically, one could formally prove that there is no algorithm that will correctly answer such a question for all $\phi$ (representable as outputs of algorithms).

In practice, it does not matter: if your function is only differentiable in a tiny, tiny region around the point of interest, so tiny that you start having numerical stability problems when trying to get into it, that differentiability is useless to you -- it's as if it were not there.

1

Just an idea.

Why not to try to compute nine points on a square, the central point being the one for which you want the second derivative (this is what you propose) and now write $$f(x,y)=a+bx+cy+dx^2+ey^2+gxy$$ and use matrix calculation just as for a multilinear regression.

I suppose that this would be more stable than computing derivatives of derivatives and the order of operations will not affect the result.

  • 0
    Wouldn't I still have the same problem with the size of $\delta$? – 2017-02-28
  • 0
    @milo. By the way, are you aware of $svcm(.)$ in $R$ ? It does multidimension splines. As I said, I do not enjoy the idea of computing derivatives of derivatives. – 2017-02-28
  • 0
    @milo. Have a look at http://www.holoborodko.com/pavel/numerical-methods/numerical-derivative/central-differences/ there are very interesting things for your needs. – 2017-02-28
  • 0
    I added the plot of $\phi$ to the question. Would a polynomial fit help in this case? – 2017-02-28
  • 0
    After looking at the plot you added, I am crying ! If there are such apparent discontinuities, I am afraid by the problem. Did you try the methods given in the link I gave earlier ? In any manner, the step size need to be small (say $\Delta x=0.001\times x$). – 2017-02-28
  • 0
    The plot is much more decent with $\delta = 0.001$ (I added a picture). I'm still worried about the obvious discontinuities when $\delta = 0.05$ – 2017-02-28
  • 0
    Let us [continue this discussion in chat](http://chat.stackexchange.com/rooms/54497/discussion-between-claude-leibovici-and-milo). – 2017-02-28