Intuitively, $\theta$ represents longitude on the (hemi-)sphere, and $\phi$ represents co-latitude. If you let $a$ vary, then it would describe altitude.
To see that these coordinates actually describe the hemisphere is straightfoward: all you have to do is check that
$$\begin{align*}
x & = a \sin \phi \cos \theta \\
y &= a \sin \phi \sin \theta \\
z &= a \cos \phi
\end{align*}$$
actually satisfy $x^2 + y^2 + z^2 = a^2$.
But how would one actually derive these coordinates? Essentially, it works the same as in 2-dimensional polar coordinates: draw the right triangle and use "soh cah toa."
So which right triangle do we draw? Let $O$ denote the origin, let $P = (x,y,z)$ denote the point on the (hemi-)sphere in question, and let $Q = (x,y,0)$ denote the projection of $P$ directly down onto the $xy$-plane. Look at the triangle $\Delta OPQ$. Mathworld's page on spherical coordinates has an illustration of this.
The height of this triangle (side $PQ$) has length $z$, the base (side $OQ$) has length $r = \sqrt{x^2 + y^2}$, and the hypotenuse (side $OP$) is the radius of the sphere $a$. Then by "soh cah toa,"
$$\begin{align*}
r &= a \sin \phi \\
z &= a \cos\phi.
\end{align*}$$
Now we examine the point $Q$, and use 2-dimensional polar coordinates. Since the length of $OQ$ is $r = a\sin\phi$, we have
$$\begin{align*}
x &= (a \sin \phi)\cos \theta \\
y &= (a \sin\phi)\sin\theta,
\end{align*}$$
which is what we wanted.