Yes, the affine space is the correct concept to look at for the distinction of points and vectors.
However, there's a simple way you can get an uniform definition of vectors and points: Define your space as $S = \mathbb R^n\times\{0,1\}\subset\mathbb R^{n+1}$. Now you define the ordinary vector space operations on $R^{n+1}$, but with a restriction: The operations are only defined ibn $S$, and only if the result again ends up in the set $S$. You'll find that for the elements $v\in S$ with $v_{n+1}=0$ you basically face no restrictions; indeed, they are just forming the vector space $\mathbb R^n$. However, if $v_{n+1}=1$, there are severe restrictions: For example, you cannot multiply them with an arbitrary number $\lambda$, because $\lambda v_{n+1}=\lambda\notin\{0,1\}$. You also cannot add them (because then $v_{n+1}+w_{n+1}=2$. However, you can add to it a vector with $v_{n+1}=0$. You can also do affine combinations of them, that is, for two vectors $v,w$ you can create $\lambda v+(1-\lambda)w$ (because, again, the last component becomes $1$), and similarly for more than two vectors (note that this is now an "atomic" operation because the intermediate steps are not defined; however you can rewrite it so that all intermediate steps are defioned, too, as $v + \lambda(w-v)$). You also can subtract two of those vectors with last component $1$, which gives you a vector with last component $0$.
Now you probably can already guess what those vectors with $v_{n+1}=1$ represent: They represent the points in $\mathbb R^n$! You can add an $\mathbb R^n$-vector (i.e. an $S$-vector with last component $0$ to a point, and get another point. You can subtract two points, and get a vector. And the affine combinations of two points form the straight line through those points (and similarly, if you have three points not on a straight line, you get a plane from their affine combinations, and so on).
Also note that there's an $1:1$ relation between points and vectors: For each vector $(v,0)$ there's a point $(v,1)$ which you get by adding the vector $v_0$ to the origin $(0,1)$.
OK, now it looks as if the origin was somehow special: After all, we've got an unique association from the (special) null vector. However this is not really the case: We could do the same restriction using any other point $(o,1)$ as origin; then we get the $1:1$ association $(v,0)\leftrightarrow(v+o,1)$. Note that all above constructions are completely unaffected by this choice, because all the vector space operations which leave the last component fixed are actually operations which leave the constant vector fixed; also, where the last component cancels out, so does the fixed vector. Note that by changing the basis, using $(o,1)$ instead of $(0,1)$ as $n+1$st basis vector, you even recover the original form $(0,1)$ for the new origin expressed in the new basis.