3
$\begingroup$

It is mentioned in some literature that we should always use central difference when computing the derivatives of an image instead of forward or backward difference. Does anyone knows why is that?

Central difference = $\frac{df(x)}{dx} = \frac{f(x+h) - f(x-h)}{2h}$

Forward difference = $\frac{df(x)}{dx} = \frac{f(x+h) - f(x)}{h}$

Backward difference = $\frac{df(x)}{dx} = \frac{f(x) - f(x-h)}{h}$

  • 0
    There are much more ways to approximate the derivative : search for '*derivative filter design*'. And you can use the one you'd like, if you understand what are the difference (phase, frequency response, effect on the probabilistic model for the signal). So the advantage of the central difference is that it has 0 phase : its Fourier transform is real. The 2 others (being the central difference shifted and strechted !) have a linear phase, it is not bad too. In general, we tend to avoid filters with chaotic phase (varying too much with the frequency) – 2017-01-30

3 Answers 3

4

A few things come to mind:

  • For smooth $f$, the central difference scheme is second order in $h$, whereas the other two you mentioned are first order in $h$. In other words, if $f$ is smooth, the (real space) error for the centered difference scheme is $O(h^2)$ whereas for the forward/backward schemes it is $O(h)$. Thus at fixed sufficiently small $h$, the central difference will have a better (real space) error than the other two. (Specifically, if $f \leq c_1 h$ and $g \leq c_2 h^2$ then $g
  • Central differencing uses the same number of points as the other two you mentioned, so there is no loss in efficiency compared to those.
  • There are higher order methods even than central differencing, and actually some of these may be even better. But very high order methods are not practical in floating point arithmetic, because they get their high order from subtraction of large nearly equal numbers, which causes loss of precision. "Middle order" methods (of order 2 through 6) are generally preferred when using double precision, because they do a good job of balancing accuracy in exact arithmetic with roundoff error when the calculations are done in double precision.

There are some other aspects of this that are more specific to signal processing. If this doesn't answer your question by itself then I can mention a little bit about that.

Upon request, here is a figure comparing the (signed) error in the central, forward, and backward derivative estimates for $e^x$ at $x=1$. The centered difference is in black, the forward difference is in blue, and the backward difference is in red.

enter image description here

  • 0
    I don't see what you mean. – 2017-01-30
  • 0
    @user1952009 What's unclear? – 2017-01-30
  • 0
    A typical derivative filter with greater length is $[-\frac{1}{3},-\frac{2}{3},\frac{2}{3},\frac{1}{3}]$. – 2017-01-30
  • 0
    @user1952009 I'm not sure I see your point. Yes, we use filters with more sample points at times. My second point was just that if you want to only use two points, you don't have to drop second order to do so. – 2017-01-30
  • 0
    I don't see what you mean with "second order" or with "for smooth $f$, the central difference scheme is second order in $h$". Theoretically, for analyzing a discrete filter replacing $\delta'$ the true derivative filter, we look at its frequency response and compare it to $i \omega$ – 2017-01-30
  • 0
    @user1952009 In real space, the central difference scheme approximates $f'(x)$ with an error which behaves as $\Theta(h^2)$ as $h \to 0$. (This is standard terminology in numerical analysis.) You could make a comparison on the Fourier side if you prefer, which may be desirable in the signal processing setting, but this is not the only way to do the analysis. (By the way, the derivative filter is $-\delta'$.) – 2017-01-30
  • 0
    Ok I just understood, you are looking at the error as $h \to 0$. But then $|f'(x) - \frac{f(x+2h)-f(x)}{2h}| < h^2 \sup_x |f''(x)|$ is very close to $|f'(x) - \frac{f(x+h)-f(x-h)}{2h}|$, it doesn't change the order of the error. And in real life $h$ is fixed, so we look at the total error $\int_a^b |f'(x) - \frac{f(x+h)-f(x-h)}{2h}|^2 dx$ with respect to the frequency. And the derivative filter is $\delta'$ because $(f \ast \delta)' = f' \ast \delta = f \ast \delta'$ (there is some time reversal in the convolution) – 2017-01-30
  • 0
    @user1952009 Oh, right, I mixed up convolution with just application of a distribution for a moment. And no, you're wrong: the central difference is higher order than the forward difference if $f$ is smooth enough. But yes, in reality $h$ is fixed, but again once $h$ is sufficiently small but finite the central difference scheme is more accurate than the forward or backward schemes. – 2017-01-30
  • 0
    The central difference with $h$ is almost the same as the forward difference with $2h$ : it doesn't change the order of the error – 2017-01-30
  • 1
    @user1952009 No, this is just blatantly wrong. $f(x+h)=f(x)+hf'(x)+1/2 h^2 f''(x)+O(h^3)$. $f(x-h)=f(x)-hf'(x)+1/2 h^2 f''(x)+O(h^3)$. Therefore $f(x+h)-f(x-h)=2hf'(x)+O(h^3)$, so the centered difference has an error of $O(h^2)$ for three times continuously differentiable functions. Doing essentially the same calculation reveals that the forward difference has an error of $O(h)$ for twice continuously differentiable functions. So for reasonably small $h$, the centered difference scheme will perform better on the real space side. – 2017-01-30
  • 0
    ??? Come on do the calculation rigorously, both $|f'(x) - \frac{f(x+h)-f(x-h)}{2h}|$ and $|f'(x) - \frac{f(x+2h)-f(x)}{2h}|$ are $< h \sup_y |f''(y)|$. The only difference is that in the 1st case we can bound with $h \sup_{|y-x| < h} |f''(y)|$ while in the second it is $h \sup_{|y-x-h| < h} |f''(y)|$ – 2017-01-30
  • 1
    @user1952009 No, $f(x+2h)=f(x)+f'(x)2h+O(h^2)$. Thus taking the difference and dividing by $2h$ gives you an error of $O(h)$. With the centered difference (again for *three* times continuously differentiable functions) the error is merely $O(h^3)$, because the second derivative terms cancel one another. – 2017-01-30
  • 1
    Again *the second derivative terms cancel one another* in the centered case: $(f(x)+hf'(x)+1/2 h^2 f''(x)+O(h^3))-(f(x)-hf'(x)+1/2 h^2 f''(x)+O(h^3))=2hf'(x)+O(h^3)$. This is exceptionally basic numerical analysis, I'm not sure why I have to explain it 5 times to someone who clearly knows some heavy math. – 2017-01-30
  • 0
    No ok I got it. I didn't except it changed something assuming it is $C^3$ (and you should see that your answer is really impossible to understand alone) – 2017-01-30
  • 0
    @user1952009 I said "smooth $f$" to begin with, so that you have all the derivatives you want. I guess I can clarify what "second order" means if you insist, but this really is standard terminology... – 2017-01-30
  • 0
    I explained : in signal/image processing we look at $h$ fixed, not $h \to 0$. And the forward difference is just the central difference shifted, so all we have to do is take the phase in account. In this context, it wasn't really possible to see what you meant with "second order" and "higher order methods" (the vocabulary of numerical analysis for approximating integrals and differential equations) – 2017-01-30
  • 1
    @user1952009 The fact that $h$ is fixed does not matter. Once $h$ is small *enough*, centered difference will perform better (in real space error metric). I see what you mean about one being a shift of the other, so that on the Fourier side one can convert between the two through a phase factor...but this can be viewed as a phase *discrepancy*, i.e. a *flaw* in the forward/backward schemes which happens to be easy to correct on the Fourier side. I also disagree that there is enough context here to really be certain of what exactly the OP wants. – 2017-01-30
  • 0
    @Ian. thanks for the clear explanation. Can you elaborate a bit more on the part where the central difference will have a better (real space) error than the other two? I understand about the $O(h^2)$ and $O(h)$ part. Maybe how $O(h^2)$ can give a better error? If you have a diagram, that will be perfect! – 2017-01-31
  • 1
    @Ray.R.Chua Is this better? – 2017-01-31
1

You already have got a couple of good relevant points, so I'm just gonna add one I haven't seen so far among the answers.

  • The result of an operator with a well defined center pixel is on the same grid where you could argue that forward or backward difference are off by a fraction of 1/2 samples in either dimension (compared to the in-grid), this could be impractical for many reasons and in many circumstances.

Here we can see how the grid moves from before and to after forward x-difference and forward y-difference respectively: None of the 3 grids are the same! enter image description here

And for the central difference scheme all of them align nicely on top of each other: enter image description here

And as many things in engineering and science estimating a differential is one of many operations which need to be compatible or align in some sense for them to be useful.

EDIT

As proposed by @Ian in the comment to avoid the grids not being aligned, one could use (expressed in the functions Z-transform):

$$\mathcal{Z}\{d_k(x_1,x_2)\} = 1-{z_k}^2$$ But we see that it will simply be a lazy filtering by one grid step of the central difference: $$\mathcal{Z}\{d_k(x_1,x_2)\} = \underset{\text{lazy}}{\underbrace{z_k}}\underset{\text{central diff.}}{\underbrace{({z_k}^{-1}-{z_k}^1)}}$$

So in practice we would be calculating the exact same thing, just moving the result one step to the left (or down).

  • 0
    I don't see what you mean here: why not just have $h$ be the same as the grid size? – 2017-01-30
  • 0
    Yes then we would at least get from one set to the same set of middle points in the convolution results. So in some sense that is better. But in practice it really just becomes a permutation of the central difference filter in the respective dimensions. – 2017-01-30
  • 0
    @mathreadler, what is $z^{-1}_{k}$ here? – 2017-01-31
  • 0
    It's coefficient is the filter tap at coordinate -1 for dimension k and 0 for all other dimensions (their exponents are all 0 so they vanish). – 2017-01-31
0

One of the reasons, not unique to image processing, is that the central difference produces a 2nd order error, while the others - 1st order, so usually the central difference schemes have better convergence and/or stability properties.

  • 0
    Not always, you may need more regularization for higher frequencies. – 2017-02-02