An idea:
Draw the diagonal $\;BD\;$ and take a close look at triangle $\;\Delta ABD\;$. Deduce that $\;F\;$ is the intersection point of the medians of this triangle and thus
$$AF=\frac23r\;\left(\implies FC=\frac43r\;\right)\;,\;\;r=\text{ the circle's radius}$$
Likewise, using Pythagoras and with $\;x=\sqrt2\,r=$ the square's side:
$$DE=\frac{\sqrt5}2x\implies FE=\frac13DE=\frac13\frac{\sqrt5}2\sqrt2\,r=\frac{\sqrt{10}}6\,r\;,\;\;AE=\frac12x=\frac r{\sqrt2}$$
So now you know all the sides of $\;\Delta AEF\;$ and you can calculate its area, for example with Heron's formula for the semiperimeter.