The equation is from the Bruck-Ryser-Chowla theorem, which says: \[x^2-(k-\lambda)y^2-\lambda z^2 (-1)^{\frac{v-1}{2}}=0\] has a solution, where $x,y,z$ are not all zero. (Note: the OP is missing some minus signs.)
We can search for all possible values of $(v,k,\lambda) \mod 8$ that satisfy the congruences $v \equiv 1 \pmod 2$, $k \equiv 5 \pmod 8$ and $\lambda \equiv 3 \pmod 4$ along with the equation \[\lambda(v-1)=k(k-1),\] which is true for symmetric BIBDs, modulo $8$.
Here's some GAP code to save me the legwork:
for k in [0..7] do if(k mod 8=5) then for lambda in [0..7] do if(lambda mod 4=3) then for v in [0..7] do if(v mod 2=1) then if((lambda*(v-1)) mod 8=(k*(k-1)) mod 8) then Print([v,k,lambda],"\n"); fi; fi; od; fi; od; fi; od;
From this we find that $(v,k,\lambda) \mod 8 \in \{(5,5,3),(5,5,7)\}$. Since $v \equiv 5 \pmod 8$, Bruck-Ryser-Chowla theorem tells us that, if a symmetric $(v,k,\lambda)$-design exists, then \[x^2-(k-\lambda)y^2-\lambda z^2=0\] has a solution where $(x,y,z) \neq (0,0,0)$.
We check the Bruck-Ryser-Chowla identify modulo $8$, using the following GAP code:
for k in [0..7] do if(k mod 8=5) then for lambda in [0..7] do if(lambda mod 4=3) then for x in [0..7] do for y in [0..7] do for z in [0..7] do if((x^2-(k-lambda)*y^2-lambda*z^2)=0 and [x,y,z]<>[0,0,0]) then Print([k,lambda]," ",[x,y,z],"\n"); fi; od; od; od; fi; od; fi; od;
which returns nothing.
So, the only solutions are when $(x,y,z) \mod 8 = (0,0,0)$. If that is the case, we can divide $x$, $y$ and $z$ through by $8$ to obtain a smaller solution. By finite descent, we would eventually obtain a solution where $(x,y,z) \pmod 8 \neq (0,0,0)$. But no such values of $(x,y,z)$ exist.