I calculated the same polynomial and I got
$$P(X)= X^2 (X-1)^3 (X-4) \,.$$
Note that $tr(A)=7$ has to be the sum of eigenvalues.
Just to get you started:
$$\det(A-tI)= \det \pmatrix{2-t & 1 & 1 & 1 & 1 & 1 \\ 1 & 1-t & 0 & 1 & 0 & 1 \\ 1 & 0 & 1-t & 0 & 0 & 1 \\ 1 & 0 & 0 & 1-t & 0 & 0 \\ 1 & 0 & 0 & 0 & 1-t & 0 \\ 1 & 0 & 0 & 0 & 0 & 1-t}$$
Subtract the 6th row from 4th and 5th: $$\det(A-tI)= \det \pmatrix{2-t & 1 & 1 & 1 & 1 & 1 \\ 1 & 1-t & 0 & 1 & 0 & 1 \\ 1 & 0 & 1-t & 0 & 0 & 1 \\ 0 & 0 & 0 & 1-t & 0 & t-1 \\ 0 & 0 & 0 & 0 & 1-t & t-1 \\ 1 & 0 & 0 & 0 & 0 & 1-t}$$
Now, $(1-t)$ common factor on rows 4 and 5.
$$\det(A-tI)= (t-1)^2\det \pmatrix{2-t & 1 & 1 & 1 & 1 & 1 \\ 1 & 1-t & 0 & 1 & 0 & 1 \\ 1 & 0 & 1-t & 0 & 0 & 1 \\ 0 & 0 & 0 & 1 & 0 & -1 \\ 0 & 0 & 0 & 0 & 1 & -1 \\ 1 & 0 & 0 & 0 & 0 & 1-t}$$
Next add Column 4 and column 5 to Column 6, and you can get a smaller $4 \times 4$ determinant....