If any two of $x,y,z$ are $0$, all the equations say $0 = 0$. On the other hand, if only one is $0$, it's easy to see that another must also be $0$.
So now let's assume none of them are $0$. Then we can cancel a $y$, $z$, or $x$ respectively from each equation, leaving $ \eqalign{(x^2 + z^2) y &= 5 x z\cr (x^2 + y^2) z &= 17 x y\cr (y^2 + z^2) x &= 20 y z\cr}$
Next, notice that changing signs of any two of the variables preserves all three equations. Thus we may assume, say, $x > 0$ and $y > 0$, and get the other solutions by symmetry.
Now $x z \le (x^2 + y^2)/2$ (because $x^2 - 2 x z + y^2 = (x-y)^2 \ge 0$), and similarly $-x z \le (x^2 + y^2)/2$, i.e $|x z| \le (x^2 + y^2)/2$.
So from the first equation we get $|y| = \frac{5 |x z|}{x^2 + y^2} \le \frac{5}{2}$ Since at this point $y$ is a positive integer, the only possibilities are $y= 1$ and $y = 2$.
If $y = 1$, the second equation says $z (x^2 + 1) = 17 x$. Since $x^2 + 1$ and $x$ are relatively prime, this implies that $x$ divides $z$. So let $z = m x$. We then have $m (x^2 + 1) = 17$. Since $17$ is prime, one of $m$ and $x^2 + 1$ must be $1$ and the other $17$. But $x \ne 0$ so it must be $x^2 + 1 = 17$ (i.e. $x=4$), and $m= 1$ so $z=4$. But $x=4,y=1,z=4$ doesn't satisfy the first or third equation.
The remaining possibility is $y=2$. Then the second equation says $z (x^2 + 4) = 34 x$. If $x$ is odd, so is $x^2 + 4$, and that is relatively prime to $x$ so it divides $34$. The only odd factors of $34$ are $1$ and $17$, so we'd need $x^2 + 4 = 17$, but that doesn't work since $17-4=13$ is not a square. So $x$ must be even: say $x = 2 t$. Now the second equation, after dividing by $4$, becomes $z (t^2 + 1) = 17 t$. As in the previous paragraph, we conclude that $t=4$ (so $x=8$) and $z=4$. Thus we have $(x,y,z) = (8,2,4)$, which is a solution. By symmetry (changing any two signs) we also have solutions $(-8,-2,4), (-8,2,-4)$ and $(8,-2,-4)$.
Summing up, we have the solutions $(x,y,z) = (0,0,z)$, $(0,y,0)$, $(x,0,0)$, $(8,2,4)$, $(-8,-2,4), (-8,2,-4)$ and $(8,-2,-4)$.