I believe that there is a misprint in the problem statement. Just as you said, the general solution of
$$
y''+\lambda y=0
$$
has the form
$$
y=A\cos\sqrt{\lambda}x+B\sin\sqrt{\lambda}x\,,
$$
(we assumed here that $\lambda>0$). However, this only satisfies the differential equation, not boundary conditions. Therefore, we need to insert this general solution into the boundary conditions and choose parameters $A$, $B$ and $\lambda$ from there. The described substitutions yield
$$
y'(0)=B\sqrt{\lambda}=0\,,
$$
$$
y(1)+y'(1)=A(\cos\sqrt{\lambda}
-\sqrt{\lambda}\sin\sqrt{\lambda})+B(\sin\sqrt{\lambda}+\sqrt{\lambda}\cos\sqrt{\lambda})=0\,.
$$
First of these equations means that $B=0$ (remember, we assumed that $\lambda>0$). The second equation then simplifies into
$$
A(\cos\sqrt{\lambda}-\sqrt{\lambda}\sin\sqrt{\lambda})=0\,.
$$
$A=0$ is not interesting, because it would give us a trivial solution. Hence, we must ensure that the second factor is equal to zero. It can be rearranged as
$$
\sqrt{\lambda}\tan\sqrt{\lambda}=1\,.
$$
If you introduce $\mu=\sqrt{\lambda}$, then $\lambda=\mu^2$ and $\mu$ solves $\mu\tan\mu=1$.