Here is one way (not guaranteed to be the simplest, but easy enough for computers) to do this task in the plane.
Let $A$, $B$, $C$ and $D$ be the position vectors of the vertices of the square, written clockwise. Imagine for a moment that $P$ is a point near the center of the square. Connect $P$ to the vertices with line segments.
By taking the cross product of the vectors $(B-P)\times(A-B)$ you get an "upward" normal, reflecting the fact that the triangle $PBA$ (around the edges in that order) is oriented counterclockwise. The same thing can be done with $PCB$, $PDC$ and $PAD$. If you set up all four of these counterclockwise-oriented (upward) triangles, then no matter where your point $P$ appears in the interior, all of the cross products will come out pointing upward.
If someone accidentally gives you the vertices in counterclockwise order, nothing changes except that all the triangles are oriented clockwise (downward).
Now imagine you take $P$ and pull it over one of the sides: what happens to the triangles? If you pull it over a side, then one of the orientations reverses. If you pull it directly over a corner, two of the orientations reverse.
So: there is your test. After you have set up your equations to give you four upward oriented triangles for an interior point, plugging in any exterior point results in a mixture of orientations for the four triangles. If all four are up or all down, then the point is in the interior.