The bijection is between the open subgroups of finite index in $G$ and the finite subgroups of $\operatorname{Hom}_{\mathrm{cont}}(G,\mathbb{Q}/\mathbb{Z})$.
The key fact is that, for every $n>0$,
$$
\mathbb{Q}/\mathbb{Z}[n]=\{t\in\mathbb{Q}/\mathbb{Z}:nt=0\}
$$
is finite.
Suppose $H$ is an open subgroup of finite index $n$ in $G$. Then $\alpha(H)$ can be identified with $\operatorname{Hom}(G/H,\mathbb{Q}/\mathbb{Z}[n])$ (add the details).
Suppose $K$ is a finite subgroup of $\operatorname{Hom}_{\mathrm{cont}}(G,\mathbb{Q}/\mathbb{Z})$. Then $H=\bigcap_{\chi\in K}\ker\chi$ is open (as a finite intersection of open subgroups). Let $n>0$ such that $nK=0$. Then the image of $\chi$ is contained in $\mathbb{Q}/\mathbb{Z}[n]$, which is finite. Therefore $G/\ker\chi$ is finite. Since
$$
G/H\hookrightarrow\bigoplus_{\chi\in K}G/\ker\chi
$$
we conclude that $H$ has finite index.
Now we can prove that $\beta(\alpha(H))=H$ and $\alpha(\beta(K))$, for every open subgroup $H$ of $G$ with finite index and every finite subgroup $K$ of $\operatorname{Hom}_{\mathrm{cont}}(G,\mathbb{Q}/\mathbb{Z})$
Let $H$ be a subgroup of finite index; consider $K=\alpha(H)$. If $\chi\in K$, then $H\subseteq\ker\chi$ and therefore $H\subseteq \beta(K)$. If $H\subsetneq\beta(K)$, the group $\beta(K)/H$ is nonzero, so there exists a nonzero group homomorphism $\gamma\colon\beta(K)/H\to\mathbb{Q}/\mathbb{Z}$. Since $\mathbb{Q}/\mathbb{Z}$ is divisible (that is, injective), this homomorphism extends to $\bar\gamma\colon G/H\to\mathbb{Q}/\mathbb{Z}$. By definition, $\chi=\bar\gamma\circ\pi\in K=\alpha(H)$ (where $\pi\colon G\to G/H$ is the canonical projection), which is a contradiction, because $\chi$ is nonzero on $\beta(K)$.
Let $K$ be a finite subgroup of $\operatorname{Hom}_{\mathrm{cont}}(G,\mathbb{Q}/\mathbb{Z})$. Set $H=\beta(K)$. By definition, we have $K\subseteq\alpha(H)$. Note that $\beta(\alpha(H))=H$. Thus $\alpha(H)$ can be identified with $\operatorname{Hom}(G/H,\mathbb{Q}/\mathbb{Z})$ and $K$ with a subgroup thereof. Since $\operatorname{Hom}(G/H,\mathbb{Q}/\mathbb{Z})$ is the character group of $G/H$, no proper subgroup of it separates the points of $G/H$.