There is a nice section on Galois connections in George Bergman's An Invitation to General Algebra and Universal Constructions (available from his website). If you want to learn more about Galois connections, I highly recommend reading the entire chapter (except perhaps section 5.4, which is a complete digression).
The general setting is:
Let $S$ and $T$ be sets, and let $R\subseteq S\times T$ be a relation from $S$ to $T$. For any $A\subseteq S$ and $B\subseteq T$, let $\begin{align*} A^* &= \{t\in T\mid \forall a\in A (aRt)\}\subseteq T\\ B^* &= \{s\in S\mid \forall b\in B (sRb)\}\subseteq S. \end{align*}$ This gives us two functions, one from $\mathcal{P}(S)$ to $\mathcal{P}(T)$, and one from $\mathcal{P}(T)$ to $\mathcal{P}(S)$. The operations are:
Inclusion reversing: if $A\subseteq A'\subseteq S$ and $B\subseteq B'\subseteq T$, then $(A')^*\subseteq A^*$ and $(B')^*\subseteq B^*$.
Increasing: $A\subseteq A^{**}$ for all $A\subseteq S$ and $B\subseteq B^{**}$ for all $B\subseteq T$.
For all $A\subseteq S$, $A^{***}=A^*$; for all $B\subseteq T$, $B^{***}=B^*$.
In particular, the maps $A\mapsto A^{**}$ and $B\mapsto B^{**}$ give closure operators on $\mathcal{P}(S)$ and $\mathcal{P}(T)$. The closed elements of $\mathcal{P}(S)$ are precisely the sets of the form $B^*$, the closed elements of $\mathcal{P}(T)$ are precisely the sets of the form $A^*$, and the $*$ operation restricted to closed sets gives an anti-isomorphism (order-reversing bijection whose inverse is also order-reversing) between the complete lattice of ${}^{**}$-closed subsets of $S$ and of $T$.
Bergman notes:
A Galois connection between two sets $S$ and $T$ becomes particularly valuable when the ${}^{**}$-closed subsets have characterizations of independent interest.
Here are the examples he gives:
The "classical example". Let $S$ be the underlying set of a field $F$, $T$ the underlying set of a finite group $G$ of automorphisms of $F$. For $a\in F$ and $g\in G$, let $aRg$ mean "$g$ fixes $a$" (i.e., $g(a)=a$). The Fundamental Theorem of Galois Theory says that the closed subsets of $F$ are precisely the subfields of $F$ that contain the set $G^*$, and that the closed subsets of $G$ are precisely all subgroups of $G$.
Let $S$ be a vector space over a field $K$, and $T$ the dual space $\mathrm{Hom}_K(S,K)$. Let $xRf$ mean "$f(x)=0$". The closed subsets of $S$ are precisely the vector subspaces, the closed subsets of $T$ are precisely the vector subspaces that are closed in a certain topology (all subspaces when the dimension is finite, but when the dimension is infinite you get interesting stuff).
Let $S=\mathbb{C}^n$ (complex affine $n$-space), and $T=\mathbb{Q}[x_0,\ldots,x_{n-1}]$, the polynomial ring in $n$ indeterminates with rational coefficients. Let $(a_0,\ldots,a_{n-1})Rf$ mean $f(a_0,\ldots,a_{n-1})=0$. This is the starting point of classical algebraic geometry: the closed subsets of $\mathbb{C}^n$ are the solution sets of polynomial equations, and the Nullstellensatz characterizes the closed subsets of $T$ as the radical ideals.
Let $S$ be a finite dimensional real vector space, $T$ the set of pairs $(f,a)$ where $f$ is a linear functional on $S$ and $a\in\mathbb{R}$. Let $xR(f,a)$ mean $f(x)\leq a$. The closed subsets of $S$ are the convex sets.
Let $S$ be a finite dimensional real vector space, $T$ the set of linear functionals on $S$. Let $xRf$ mean $f(x)\leq 1$. The closed subsets on each side are the convex subsets that contain the zero vector.
Let $A$ be an abelian group (or a module over a commutative ring) and $S=T=\mathrm{End}(A)$ the ring of endomorphisms. Let $sRt$ stand for $st=ts$. Given a subring $X$ of $S$, $X^*$ is the commutant of $X$, an important subring studied by ring theorists.
Let $S$ be a set of mathematical objects, $T$ a set of propositions about objects of this sort, and $sRt$ stand for "object $s$ satisfies proposition $t$" (for logicians, $s\models t$). The closed subsets of $S$ are the axiomatic classes, the closed subsets of $T$ are the theories.
Added. It seems to me that Baez is staking a middle ground between the full generality of Galois connections and the special case of Galois Theory of fields, by considering only the situation in which $T$ is a set, $S$ is a subgroup of the group of bijections $T\to T$, and the relation is $fRt$ if and only if $f(t)=t$. In that case, you can always let $k=S^*$, and you do indeed get a correspondence between the closed subsets of $T$ and the closed subgroups of $S$ (though not every subgroup need be closed; this occurs for instance in the Galois group of an infinite field extension, where only subgroups that are closed in the profinite topology correspond to subfield of the extensions).