Albraically a projective space is $\frac{V \setminus \underline{0}}{\sim}$, where $\sim$ is an equivalence relation defined as $\underline{x} \sim \underline{y} \Leftrightarrow \underline{x} = \rho \underline{y} \; \rho \in \mathbb{R}^{*}$, $V$ is a vector space, $\underline{0}$ is the null vector and $\mathbb{R}^* = \mathbb{R} \setminus \{ 0 \}$.
Basically, what the definition says is that a projective space is a vector space without the zero vector and where all the vectors that multiples of each other are identified. This leads to topological representations of projective spaces such as the projective line, which can be identified as a circle (if you add the "point at infinity" on a line, it closes).
As for your intuition, it is right. What happens in a projective space is that you "add" some "points at infinity" which identify a direction (all parallel lines "end" in the same point).
Like aaron points out, since we qutient out all multiples, the projective space has a dimension less than the vector space we construct it from.
EDIT: re-reading the OP's comment, I see I answered only partially. The point that the horizon is a circle is derived from a topological model of the 2 dimensional projective space (projective plane). You can see it as your "normal" cartesian plane, where you add a "circle" at infinity, where all the antipodal points are identified. This means that the points lying on the circumference on the first quadrant are the same as the points on the circumference on the third (same for the second and fourth quadrant). This is just a description of what aaron said: you're taking all lines from the origin and identifying their directions (their "points at infinity") so that all parallel lines end up at the same point. (I could draw something on the computer, but it would look really silly, because I can't program anything that would do it for me automatically, so I'd have to use MS Paint) So, yes, the horizon is actually a circle and "in perspective" parallel lines meet "at infinity", which was the motivation for artists to start and think mathematically of perspective (this is, I think, one of the historical origins to the projective plane).
As for the projective line, you should try to think to add one single point to a regular line, let's say "at infinity". What happens is that the line starts from this point very far away and ends in this same point (intuitively, a line has only "one infinity" to go to) so it closes in a circle.
Is this more explanatory? :)