Suppose that you’re given a function $f:X\to Y$ and asked to find a set $A\subseteq X$ on which $f$ is a bijection. $A$ is as large as possible if there is no $B\subseteq X$ such that
- $f$ is a bijection on $B$, and
- $A\subsetneqq B$.
In other words, if you expand $A$ to include even one more point of $X$, $f$ is not a bijection on the expanded set. This means that if $x$ is any point of $X$ that does not belong to $A$, there is already some point $a\in A$ such that $f(a)=f(x)$: if there were no such $a$, you could expand $A$ to $B=A\cup\{x\}$, and $f$ would still be a bijection on $B$.
Note that if $f$ is a bijection on all of $X$, then $X$ is automatically the largest possible subset of itself on which $f$ is a bijection.
Another point to note is that there may be more than one set $A$ that is as large as possible. To use an example that is not part of the problem, consider the function $f:\Bbb R\to\Bbb R:x\mapsto\cos x$. You know that the possible values of $\cos x$ are precisely the real numbers in the interval $[-1,1]$, so if this $f$ is a bijection on some $A\subseteq\Bbb R$, and $f[A]=[-1,1]$, then $A$ is as large as possible. Three such sets $A$ are $[0,\pi]$, $[\pi,2\pi]$, and $(-\pi/2,0]\cup[\pi/2,\pi]$, as you can see fairly easily if you draw the graph of $y=\cos x$.
By the way, there is a typo in part (c) of the problem: the function should read
$\begin{bmatrix}x\\y\end{bmatrix}\mapsto\begin{bmatrix}1&0\\1&2\end{bmatrix}\begin{bmatrix}x\\y\end{bmatrix}=\begin{bmatrix}x\\x+2y\end{bmatrix}\;.$