Denote by $\mathrm{sp}(A)$ the Gelfand spectrum of $A$. First, show that the Gelfrand transform $\hat{x} : \mathrm{sp}(A) \rightarrow \sigma(x)$ of $x$ is in fact a homeomorphism of topological spaces. This also implies that you can identify $C(\sigma(x))$ with $C(\mathrm{sp}(A))$ with the pullback.
Now, given $\lambda \notin \sigma(x)$, you know that $(x - \lambda)$ is invertible and so $(x - \lambda)^{-1} \in A$. Because $A$ is generated by a single element $x$, this implies that there are polynomials $p_n(z)$ such that $p_n(x) \to (x - \lambda)^{-1}$, inside $A$. Taking the Gelfand transform, you also have $\widehat{p_n(x)} \to \widehat{\frac{1}{x - \lambda}}$ inside $C(\mathrm{sp}(A))$. Using the identification above, you obtain in fact a sequence of complex polynomials $p_n(z)$ that converges uniformly over $\sigma(x)$ to the holomorphic function $\frac{1}{z - \lambda}$.
Assume that $\mathbb{C} \setminus \sigma(x)$ is not connected and decompose it as $ \mathbb{C} \setminus \sigma(x) = \Omega_{\infty} \sqcup \Omega $ where $\Omega_{\infty}$ is the open unbounded connected component and $\Omega \neq \emptyset$ is an open bounded (not necessarily connected) set which is the union of all other connected components of $\mathbb{C} \setminus \sigma(x)$. Define $K = \sigma(x) \sqcup \Omega$ (the spectrum together with the holes). Then, as $\partial{\Omega} \subset \sigma(x)$, the set $K$ is closed and bounded, hence compact and we have $\partial{K} = \partial{\sigma(x)}$. This, together with the maximum modulus principle, implies that for any polynomial $p(z)$ we have $ \max_{z \in \sigma(x)} |p(z)| = \max_{z \in K} |p(z)|. $
Let $\lambda \in \Omega$. Take the sequence $p_n(z) \to \frac{1}{z - \lambda}$ we have constructed above that converges uniformly over $\sigma(x)$. Let $m = \max_{z \in \sigma(x)} |{z - \lambda}|$. Then, we have $n$ such that $ \max_{z \in \sigma(x)} |p_n(z) - \frac{1}{z - \lambda}| < \frac{1}{m}. $ But this, together with the maximum modulus principle, implies that $ \max_{z \in \sigma(x)} |(z - \lambda) p_n(z) - 1| = \max_{z \in K} |(z - \lambda) p_n(z) - 1| < 1.$ Plugging in $z = \lambda$ we have $1 < 1$ which contradicts our decomposition.
Let us put some context to this argument. In general, given a compact subset $K \subset \mathbb{C}$, the complex polynomials are not going to be dense in $C(K)$. For one thing, if a function is a uniform limit of polynomials on $K$, then on $K^{\circ}$ (if it is non-empty), it will be holomorphic. But there are further topological restrictions. If $K$ is a compact set "with holes", then if you take any holomorphic function with a pole inside a hole, you won't be able to approximate it uniformly on $K$ with polynomials. On the other hand, if $K$ doesn't have a hole, then any function which is continuous on $K$ and holomorphic on $K^{\circ}$ can be approximated by polynomials. This is Mergelyan's theorem, see Chapter 20 of Rudin's "Real and Complex Analysis". What we have done here is to show that if there are holes, and if $\lambda$ is inside such a hole, then because of the special structure of $A$, it implies that the function $\frac{1}{z - \lambda}$ can be approximated uniformly by polynomials, which implies that in fact there cannot be any holes.