Suppose that the upper righthand corner of the rectangle is at the point $\langle x,y\rangle$. Then you know that the area of the rectangle is, as you say, $4xy$, and you know that $\frac{x^2}{a^2} + \frac{y^2}{b^2} = 1\;.\tag{1}$
Thinking of the area as a function of $x$, we have $\frac{dA}{dx}=4x\frac{dy}{dx}+4y\;.$ Differentiating $(1)$ with respect to $x$, we have
$\frac{2x}{a^2}+\frac{2y}{b^2}\frac{dy}{dx}=0\;,$ so $\frac{dy}{dx}=-\frac{b^2x}{a^2y}\;,$ and $\frac{dA}{dx}=4y-\frac{4b^2x^2}{a^2y}\;.$
Setting this to $0$ and simplifying, we have $y^2=\dfrac{b^2x^2}{a^2}$. From $(1)$ we know that $y^2=b^2-\frac{b^2x^2}{a^2}\;.$ Thus, $y^2=b^2-y^2$, $2y^2=b^2$, and $\dfrac{y^2}{b^2}=\dfrac12$. Clearly, then, $\dfrac{x^2}{a^2}=\dfrac12$ as well, and the area is maximized when
$x=\frac{a}{\sqrt2}=\frac{a\sqrt2}2\quad\text{and}\quad y=\frac{b}{\sqrt2}=\frac{b\sqrt2}2\;.$