Let's find the upper central series. For ease of notation, let
$$ T_k = \begin{pmatrix} 1 & k\\0 & 1\end{pmatrix}$$
and
$$ R = \begin{pmatrix} 0 & 1 \\ 1 & 0\end{pmatrix}$$
be translation and reflection matrices, with inverses $T_k^{-1} = T_{-k}$ and $R^{-1}=R$. The center of $GL(2,\mathbb{Z}$) must surely commute with all $T_k$ (that is, for all $k$), and also with $R$. If we let
$$ X = \begin{pmatrix} a & b \\ c & d\end{pmatrix}$$
then $X$ being in the center implies
\begin{align}
T_kXT_{-k} &= X\\
RXR &= X
\end{align}
This is enough to imply $b=c=0$ and $a=d$ [to see this, just do the multiplcations and equate matrix coefficients]; that is, the center is $Z_1=\{I, -I\}$, with $I$ the $2\times2$ identity matrix.
What about the second center, $Z_2$? These are elements that are central "mod $Z_1$", or elements $X$ such that for all $Y\in GL(2,\mathbb{Z})$, we have
$$ YXY^{-1} = \pm X$$
[Note: the choice of $+$ or $-$ on the right hand side is dependent on $Y$.]
Again, using $T_k$ and $R$, show that $X$ would still have to be $\pm I$. This means $Z_2=Z_1$, and thus our upper central series has stalled (and hasn't become the whole group); the group $GL(2,\mathbb{Z})$ thus is not nilpotent.
EDIT: Adding in more details.
First center
If $X$ is in the center of $GL(2,\mathbb{Z})$, then it commutes with all matrices. In particular, we have
$$ T_kXT_{-k}=X$$
for all $T_k$. Expanding that out, we get four equations:
\begin{align}
a+ck &= a\\
b+k(d-a-ck) &= b\\
c &= c\\
d-ck &= d
\end{align}
Since this holds for any $k$, we get from the first equation $a+ck_1=a+ck_2$ for $k_1\neq k_2$, which immediately implies $c=0$. Knowing $c=0$, the second equation similarly implies $a=d$. To get $b=c$, we expand $RXR=X$:
\begin{equation}
\begin{pmatrix}
d & c\\
b & a
\end{pmatrix} = \begin{pmatrix} a & b\\c & d\end{pmatrix}
\end{equation}
Thus, our center is $\{I, -I\}$ (we actually need to check these commute with all matrices, but that's easy).
Second center
The second center is defined as $Z_2(G)/Z_1(G)=Z(G/Z_1(G))$; that is, it consists of the matrices that commute with all others, "mod $Z_1$". This means that if $X\in Z_2$, and $Y\in GL(2,\mathbb{Z})$, then there exists $Z_Y\in Z_1$ such that
$$ YXY^{-1} = Z_YX$$
The element from $Z_1$ will probably depend on $Y$; the point is that it always exists. (If you want to think about this more abstractly, the above is just saying that $[Z_2, G]\subset Z_1$).
Let's find the second center. If $X\in Z_2$, then for every $k$, there exists $Z_k\in Z_1=\{I, -I\}$ such that
$$ T_kXT_{-k} = Z_kX$$
Since there are only two choices for $Z_k$ (namely, $I$ or $-I$), and since there are infinitely many $k$s, we can choose $k_1, k_2$ such that $Z_{k_1}=Z_{k_2}$, and thus just as above, we get (after possibly cancelling a negative sign if $Z_{k_1}=Z_{k_2}=-I$)
$$ a + ck_1 = a + ck_2$$
which again implies $c=0$. Using $R$ we have
\begin{equation}
\begin{pmatrix}
d & c\\
b & a
\end{pmatrix} = Z_R\begin{pmatrix} a & b\\c & d\end{pmatrix}
\end{equation}
where $Z_R$ is $I$ or $-I$. Thus we get $b=\pm c=0$, and $a=\pm d$.
That means the only thing left to show that $Z_2=Z_1=\{I, -I\}$ is to show that $a=-d$ cannot happen. That is, we need to show the matrix
$$ \begin{pmatrix}
-1 & 0\\
0 & 1
\end{pmatrix}
$$
is not in $Z_2$. I'll leave this as an exercise (hint: try checking what happens with $T_1$).