Find the homography matrix that maps a 2D image point to a 3D world point on a known plane

Question

Given a pinhole camera with known 3x4 calibration matrix $P$ how do we find the 3x3 homography matrix $H$ that can map 2D image points to 3D world points given that the 3D points are constrained to a known world plane $\pi$?

Hartley and Zissermann's Multiple View Geometry section 8.1.1 gives a simple case with the world plane $Z=0$.
$$x=PX=\begin{bmatrix}p1 & p2 & p3 & p4\end{bmatrix} \begin{pmatrix}X \\ Y \\ 0 \\ 1\end{pmatrix} =\begin{bmatrix}p1 & p2 & p4\end{bmatrix} \begin{pmatrix}X \\ Y \\ 1\end{pmatrix} $$
Here the 3x3 homography that maps world points to image points is given by $H=\begin{bmatrix}p1 & p2 & p4\end{bmatrix}$ and the inverse $H^{-1}$ maps homogeneous pixels to world points $$X=H^{-1}x$$ In my case the world points are constrained to a known plane that is parallel to $Z=0$ but offset some known amount $a$. I tried to solve: $$x=PX=\begin{bmatrix}p1 & p2 & p3 & p4\end{bmatrix} \begin{pmatrix}X \\ Y \\ a \\ 1\end{pmatrix}$$ But I'm not sure how to proceed in this case because I can't invert the 3x4 matrix on the right. If it exists I'm looking for a solution that gives $H^{-1}$ with parameter $a$ so I don't have to invert a matrix for each point I want to map.

Answer 1

You can just change coordinates so that your parallel plane becomes the plane $Z=0$.

$P\begin{bmatrix}X\\Y\\Z\\1\end{bmatrix}=PA^{-1}\begin{bmatrix}X\\Y\\Z-h\\1\end{bmatrix}$ where $A=\begin{bmatrix}1&0&0&0\\0&1&0&0\\0&0&1&-h\\0&0&0&1\\\end{bmatrix}$.

I think then you can proceed as normal with $PA^{-1}$ by continuing with $Z-h=0$ (i.e. $Z=h$.)

That's the argument that first occurs to me anyway: change world coordinates to make your plane coincide with the $Z$ plane. I have not seen it written down, and I have not applied it, but that seems reasonable.

Answer 2

If you write

$$x=PX=\begin{bmatrix} |&|&|&|\\ p_1 & p_2 & p_3 & p_4 \\ |&|&|&| \end{bmatrix} \begin{pmatrix}X \\ Y \\ a \\ 1\end{pmatrix}$$

then the right hand side contains two variables and two constants. Since you are dealing with homogeneous coordinates, there is a third degree of freedom in scaling the whole vector. A result can be mapped back to an input of this form by employing these three degrees of freedom.

But in your notation, it looks like four independent elements. The reason why they have to be fixed is that the last two elements always have ratio $a:1$, for your chosen constant $a$. To make this clearer, you can combine these two constants into a single coordinate:

$$x=\begin{bmatrix} |&|&|\\ p_1 & p_2 & ap_3+p_4 \\ |&|&| \end{bmatrix} \begin{pmatrix}X \\ Y \\ 1\end{pmatrix}$$

Now the coefficient $a$ has moved to the matrix, which is now $3\times3$ and can be inverted (at least in general). If you compute the inverse, and then want to go back to your original coordinates, you get the first, second and fourth coordinate, and the third is $a$ times that fourth coordinate.

Answer 3

There’s a bit of a misunderstanding in your question about what $\mathtt H$ does. It doesn’t map from world coordinates to image coordinates. For it to do that, it would need to be a $4\times3$ matrix, not $3\times3$. Instead, it maps from some coordinate system of the plane $\mathbf\pi$ to image coordinates, i.e., it represents a map from $\mathbb{RP}^2$ to $\mathbb{RP}^2$, not $\mathbb{RP}^3\to\mathbb{RP}^2$. By the same token, $\mathtt H^{-1}$ doesn’t map to world coordinates, but to this coordinate system on the plane.

In order to build $\mathtt H$ you first have to choose a coordinate system for $\mathbf\pi$. Using Zisserman’s notation from §3.2.1, this means that you have a full-rank $4\times3$ matrix $\mathtt M$ that maps from these coordinates to world coordinates, i.e., $\mathbf X = \mathtt M\mathbf x$. Once you have this, getting the homography matrix is straightforward: the projected image of this point is $\mathtt{PM}\mathbf x$. We now insist that the camera center not lie on the plane so that the $3\times3$ matrix $\mathtt H = \mathtt{PM}$ has full rank, otherwise the image of $\mathbf\pi$ collapses to a line through the principal point. If you want to map from image to world coordinates on this plane, the appropriate matrix is then $\mathtt M\mathtt H^{-1} = \mathtt M(\mathtt{PM})^{-1}$. This construction is an example of the point transfer method that comes up again later in the epipolar geometry section.

For your example plane, a natural choice of coordinate system is $$\mathtt M = \begin{bmatrix}1&0&0\\0&1&0\\0&0&a\\0&0&1\end{bmatrix}.$$ Using the standard camera matrix $\mathtt P = [\mathtt I\mid\mathbf 0]$, the resulting homography matrix is $$\mathtt H = \begin{bmatrix}1&0&0\\0&1&0\\0&0&a\end{bmatrix}.$$