This is actually a result that holds for rings: if an element has more than one right inverse, then it has infinitely many right inverses.
Suppose that there are at least two elements in $A$, so that $\alpha$ cannot have a left inverse. Let $\beta_1$ be any element of $A$. Consider the map $f\colon A\to A$ given by $f(\beta) =\beta\alpha - \mathrm{id} + \beta_1$.
Note that this is well-defined: the image of $\beta$ is an endomorphism of $V$ (composition and sum of linear transformations is linear), and $\alpha\Bigl(\beta\alpha - \mathrm{id} + \beta_1\Bigr) = \alpha\beta\alpha -\alpha + \alpha\beta_1 = \alpha - \alpha +\mathrm{id} = \mathrm{id}$ so if $\beta$ is a right inverse for $\alpha$, then so is $f(\beta)$.
Next: $f$ is one-to-one: if $f(\beta)=f(\beta')$, then $\beta\alpha - \mathrm{id} + \beta_1 = \beta'\alpha - \mathrm{id} + \beta_1$ which yields $\beta\alpha = \beta'\alpha$. Now composing with $\beta_1$ on the right we obtain $\beta=\beta'$, since $\alpha\beta_1=\mathrm{id}$.
Finally, $f$ is not onto: we can never get $\beta_1$ in the image. For if $f(\beta)=\beta_1$, then $\beta\alpha - \mathrm{id} + \beta_1 = \beta_1$, hence $\beta\alpha=\mathrm{id}$. But that would mean that $\beta$ is both a left and a right inverse for $\alpha$, which is impossible since $\alpha$ has at least two right inverses.
Thus, $f\colon A\to A$ is one-to-one and not onto, which proves that $A$ is infinite.