I believe that there is a misprint in the problem statement. Just as you said, the general solution of $ y''+\lambda y=0 $ has the form $ y=A\cos\sqrt{\lambda}x+B\sin\sqrt{\lambda}x\,, $ (we assumed here that $\lambda>0$). However, this only satisfies the differential equation, not boundary conditions. Therefore, we need to insert this general solution into the boundary conditions and choose parameters $A$, $B$ and $\lambda$ from there. The described substitutions yield $ y'(0)=B\sqrt{\lambda}=0\,, $ $ y(1)+y'(1)=A(\cos\sqrt{\lambda} -\sqrt{\lambda}\sin\sqrt{\lambda})+B(\sin\sqrt{\lambda}+\sqrt{\lambda}\cos\sqrt{\lambda})=0\,. $ First of these equations means that $B=0$ (remember, we assumed that $\lambda>0$). The second equation then simplifies into $ A(\cos\sqrt{\lambda}-\sqrt{\lambda}\sin\sqrt{\lambda})=0\,. $ $A=0$ is not interesting, because it would give us a trivial solution. Hence, we must ensure that the second factor is equal to zero. It can be rearranged as $ \sqrt{\lambda}\tan\sqrt{\lambda}=1\,. $ If you introduce $\mu=\sqrt{\lambda}$, then $\lambda=\mu^2$ and $\mu$ solves $\mu\tan\mu=1$.