I have to solve the following problem.
Task-description: A sphere with the radius of $10$ contains a pyramid with the maximal lateral surface ($M_{max}$). Find $M_{max}$.

I need two terms. One of them is the following, but I don't find a second one.

I have to solve the following problem.
Task-description: A sphere with the radius of $10$ contains a pyramid with the maximal lateral surface ($M_{max}$). Find $M_{max}$.

I need two terms. One of them is the following, but I don't find a second one.

Let $S=(0,0,1)$, let $P$ be one of the base points, and let $\alpha:=\angle(OSP)$. Then the length of the ascending edges is $2\cos\alpha$, and the length of the base diagonal is $2\sin(2\alpha)$. It follows that the side walls of the pyramid are isosceles triangles with base $\sqrt{2}\sin(2\alpha)$ and height $$h=\sqrt{4\cos^2\alpha-\sin^2(2\alpha)/2}\ .$$ Therefore the total lateral surface $M$ is given by $$M=4{1\over2}\sqrt{2}\sin(2\alpha)\, h=8\sin\alpha\ \cos^2\alpha\sqrt{2-\sin^2\alpha}\ .$$ Putting $\sin\alpha=:s$ we now have to maximize the function $$f(s):=s(1-s^2)\sqrt{2-s^2}\qquad(0\leq s\leq 1)\ .$$ Doing the computations gives $s_{\rm opt}=\sqrt{1-1/\sqrt{2}}$.
Let $O$ be the midpoint of the line segment $AB$. (I use the same notation for the points as in your picture.)
You have correctly noticed that $$h_a^2=h^2+\frac{a^2}4,$$ which is equation obtained from the right triangle $SNO$.
You can use equation $$r^2=(h-r)^2+\frac{a^2}2$$ which you obtain from the right triangle $MNA$. (Just notice that the distance $|MN|$ is $h-r$.)
The above equation is equivalent to $$\begin{align} r^2&=h^2-2hr+r^2+\frac{a^2}2\\ 0&=h^2-2hr+\frac{a^2}2\\ 2hr&=h^2+\frac{a^2}2\\ \frac{a^2}2&=2hr-h^2=h(2r-h) \end{align}$$
So you have $h_a^2=h^2+\frac{a^2}4=h^2+\left(hr-\frac{h^2}2\right)=hr+\frac{h^2}2$.
Maximizing $2ah_a$ is equivalent to maximizing $$(ah_a)^2=(2hr-h^2)(2hr+h^2).$$ Or, if you denote $t=\frac{h}r$, you get $$(ah_a)^2=r^4(2t-t^2)(2t+t^2),$$ which reduces the problem to maximizing $(2t-t^2)(2t+t^2)$.
BTW are you sure you want to maximize $2ah_a$ and not $2ah_a+a^2$? (The second one is the surface including the base. It seems that this would be a more difficult problem.)
Note that you can get $\frac{a^2}2=2hr-h^2=h(2r-h)$ also from triangle $SAS'$, where $S'$ is the point opposite to $S$; if you use right triangle altitude theorem. (The triangle $SAS'$ is right triangle according to Thales's theorem. The hypotenuse $|SS'|=2r$ is divided into two parts of lengths $h$ and $2r-h$.)
Added later:
The function $f(t)=(2t-t^2 )(2t+t^2)=t^2(4-t^2)$ attains maximal value $f(t)=4$ for $t^2=2$, i.e. $t=\sqrt2$. (See Wolframalpha. This can be also seen directly, by maximazing the function $t^2(4-t^2)$ -- perhaps with the substitution $s=t^2$ -- or by by applying AM-GM inequality as $t^2(4-t^2)\le \frac{t^2+(4-t^2)}2=2$.)
This corresponds to $(ah_a)^2=4r^2$, $ah_a=2r$ and the lateral surface area $$M=2ah_a=4r.$$ We also have $h=tr=\sqrt2r$.
If we compare this with Christian Blatter's solution, he gets maximal value $f(s_{\rm opt})=\frac12$, see wolframalpha. This gives $M=8f(s_{\rm opt})=4$.
We can also notice that the optimal angle is $\frac\pi8$, since $s_{\rm opt}=\sqrt{1-1/\sqrt{2}}=\sqrt{\frac{2-\sqrt2}{\sqrt2}}=\sin\frac\pi8$, see Wikipedia.