The zeta function is trivially a solution to many differential equations
You are correct that we can construct differential equations for which $\zeta(s)$ is a solution in this manner. But what you have. . .
$y'(s) = \displaystyle \sum_{n=1} ^\infty \frac{\log(n)}{n^s}$
. . . is quite a complicated equation. Much more complicated that, for example, a finite linear equation like. . .
$A_n(s)y^{(n)} + A_{n-1}(s)y^{(n-1)}+ \ldots + A_{1}(s)y' + A_{0}(s)y(s) = 0$
. . . where $A_i(s)$ are some functions. The paper points out that $\zeta(s)$ does not solve any finite linear equation. Then it asks how complicated we need to make an equation in order for $\zeta(s)$ to solve it. Does it need to be as complicated as yours or can we do better?
The answer given in the paper is that $\zeta(s)$ solves an infinite linear equation of the form
$ \ldots + A_n(s)y^{(n)} + A_{n-1}(s)y^{(n-1)}+ \ldots + A_{1}(s)y' + A_{0}(s)y(s) = 0$.
This equation is simpler than yours in that there are no functions of $s$ other than $y(s)$ and its derivatives. However it is more complicated in the sense that it includes the higher derivatives of $y(s)$.