Without loss of generality, you can assume that your circle of radius $r$ is centered in $(0,0)$. In that case, you can parametrize your rectangle in terms of the angle $\theta$ between a diagonal and the $x$-axis, as in the picture below:

Then, as shown above, the area of the shaded triangle is
$$A_T(\theta) = \frac{1}{2} \big(r\cos(\theta)\big) \big(r\sin(\theta)\big)$$
and, hence, the area of the rectangle is
$$A_R(\theta) = 8\cdot A_T(\theta) = 4 \big(r\cos(\theta)\big) \big(r\sin(\theta)\big) = 2r^2\sin(2\theta).$$
Finally, to maximize it, as you mentioned, you just need to look for the $\theta$ such that $\frac{dA_R}{d\theta}(\theta)=0.$
Note that $A(\theta)$ gives you the area of the rectangle in terms of $\theta$. The derivative $\frac{dA_R}{d\theta}$ expresses the rate at which the area increases or decreases when variating $\theta$. So, if $\frac{dA_R}{d\theta}=0$ for some $\theta$, it means that you are in a minimium or a maximun, because there is no variation at that point:

It is like climbing a mountain: you know you are on the top when there slope there is no slope remaining to climb.
Once you know where the critical point is, you will need to determine whether it is a minimum or a maximum. For that you have to check the sign of the second derivative: if $\frac{dA_R}{d\theta}(\theta)>0$ for that value of $\theta$, then it is a minimum; if $\frac{dA_R}{d\theta}(\theta)>0$, then it is a maximum.