Question 1:
Number your points from 1 to 20. Pick 4 numbers from 1 to 20, and then work with corresponding points. Pick your numbers in increasing order. That way, you will never have duplicates (for example, checking 1,2,3,4 then later checking 2,1,4,3).
Start with {1,2,3,4}, then {1,2,3,5}, then {1,2,3,6}, ..., then {1,2,3,20}. At this point you increment the 3rd value: 1,2,4, .... Because we want increasing numbers, the first pick for the final value will be 5:
{1,2,4,5}, {1,2,4,6}, ... {1,2,4,20}
{1,2,5,6}, {1,2,5,7}, ... {1,2,5,20}
...
{1,2,19,20}
{1,3,4,5}, ... ... {1, 18, 19,20}
{2,3,4,5}, ... ... ... {17,18,19,20}
Question 2:
That is WAY too much work to do for this problem. You have additional information that can eliminate a lot of possibilities without having to check them all. If the distance from point A to point B is not the same as from point A to point C or from point B to point C, then you don't even need to look for a point D. These three will not be part of any square.
I suggest the following strategy:
Let $P_i = (x_i, y_i)$ be the $i^{th}$ point. Then let $s_{ij}$ be the distance squared from $P_i$ to $P_j$. So $s_{ij} = (x_i - x_j)^2 + (y_i - y_j)^2$. There is no need to take the square root to get the distance itself. In fact, doing so makes things a little harder.
Start by calculating $s_{1j}$ for $j = 2, 3, ...$. Each time you calculate a new value, compare to the previous values. You are looking for three indices $a, b, c$ such that $s_{1b} = 2s_{1a} = 2s_{1c}$. That is, $s_{1a} = s_{1c}$, while $s_{1b}$ has exactly twice that value. If you find such a trio, calculate $s_{ab}$ and $s_{bc}$. If both of these have the same value as $s_{1a}$ and $s_{1c}$, then the points $P_1, P_a, P_b, P_c$ form a square.
If you go through all $j = 2, ..., 20$ without finding such a trio, then $P_1$ is not part of a square. Discard it and repeat the process with the remaining 19 points.
