It appears from the context in the book that the correct problem is $ A^2 + B^2 = A B + B A = 0. $ The middle step is that $(B-A)^2 = 0,$ so we name the nilpotent matrix $N=B-A.$ Wait, I think that is enough. Because it is also true that $(A+B)^2 = 0.$ So $A+B$ and $B-A$ both have trace $0.$ So $tr \; \; 2B = 0.$ That finishes characteristic other than 2. We don't need full Jordan form for nilpotent matrices, just a quick proof that $N^2 = 0$ implies that the trace of $N$ is zero. Hmmm. This certainly does follow from the fact that a nilpotent matrix over any field has a Jordan form, but I cannot say that I have seen a proof of that.
Alright, in characteristic 2 this does not work, in any dimension take $ A = B, $ $ A = B \; \; \; \mbox{then} \; \; A^2 + B^2 = 2 A^2 = 0, \; AB + BA = 2 A^2 = 0. $
In comparison, the alternate problem $ A^2 + B^2 = A B + B A = I $ has the same thing about nilpotence, however in fields where $2 \neq 0$ and $2$ is a square we get a counterexample with $ A \; = \; \left( \begin{array}{rr} \frac{1}{\sqrt 2} & \frac{-1}{2} \\ 0 & \frac{1}{\sqrt 2} \end{array} \right) $ and $ B \; = \; \left( \begin{array}{rr} \frac{1}{\sqrt 2} & \frac{1}{2} \\ 0 & \frac{1}{\sqrt 2} \end{array} \right) $