The singular value decomposition of $A$ gives us orthogonal matrices $U, V$ and a diagonal matrix $S$ such that
$A = U S V^T$
Since $U$ and $ V$ are orthogonal, we have $\|U\| = \|V\| = 1$. Therefore, for all vectors $x$, we have
$\|A x\| = \|S x\|$
Since $S$ is the diagonal matrix containing the singular values of $A$ (which by definition are the roots of the eigenvalues of $A^T A$), the $x'$ which maximizes $\|S x\|$ is the unit vector $e_1 = (1, 0, \dots, 0)$ assuming the singular values in $S$ are sorted in descending order of magnitude.
Let $s$ be the largest singular value, then we have
$\|A\| = \|A x'\| = \|S x'\| = \|S e_1\| = \|e_1 s\| = s$
So, the norm of $A$ is indeed the largest singular value $s$ of $A$, which is the root of the largest eigenvalue of $A^T A$.