For a "formula" we would first have to discuss what constitutes an answer, but I made a picture to make it clear that the condition "inside" is not a very natural one.

I used geogebra. Note that when the curve crosses the line CD or AB , you do not have equal angles anymore, instead the smaller angles sum to 180 degrees, but it is fine again when the curve crosses again.
Furthermore, note that if you move the vertex A slightly, the part of the curve that passes through $A$ and $B$ becomes detached and formes a little oval curve.
If you want to see an equation: $\frac{((a1 - x) (b1 - x) + (a2 - y) (b2 - y))}{\sqrt{((a1 - x)^2 + (a2 - y)^2) ((b1 - x)^2 + (b2 - y)^2)}} = \frac{((c1 - x) (d1 - x) + (c2 - y) (d2 - y))}{\sqrt{((c1 - x)^2 + (c2 - y)^2) ((d1 - x)^2 + (d2 - y)^2)}}$
It does not get better if you square it.
Non-convex quadrilaterals do not look different, they also can pass from an S-curve to a little oval plus another branch.
