Preface added given the confusion exhibited by the OP in the comments.
We are trying to figure out whether or not there exist real numbers $x$ and $y$ such that $T(x,y) = (1,-4).$ (That's what it means for $(1,-4)$ to be in $R(T)$; it is in the range if there do exist such $x$ and $y$; it is not in the range if no such $x$ and $y$ exist). This is the same as asking whether or not there exist real numbers $x$ and $y$ such that $(2x - y , -8x + 4y) = (1,-4),$ which is the same as trying to solve the system of linear equations $\begin{array}{rcccl} 2x & - & y & = & 1\\ -8x & + & 4y & = & -4. \end{array}$
That is: the linear transformation can be represented by the $2\times 2$ matrix $\left(\begin{array}{rr} 2 & -1\\ -8 & 4 \end{array}\right),$ in the sense that $T(x,y) = (a,b)$ if and only if $\left(\begin{array}{rr} 2 & -1\\ -8&4\end{array}\right)\left(\begin{array}{c}x\\y\end{array}\right) = \left(\begin{array}{c}a\\b\end{array}\right).$
So, $(1,-4)$ is in the range of $T$ if and only if there exists $x$ and $y$ such that $\left(\begin{array}{rr} 2 & -1\\ -8 & 4\end{array}\right)\left(\begin{array}{c}x\\y\end{array}\right) = \left(\begin{array}{r}1\\-4\end{array}\right),$ which as you write is equivalent to solving the system with augmented matrix $\left(\begin{array}{rr|r} 2 & -1 & 1\\ -8 & 4 & -4 \end{array}\right).$ You will not be able to solve this by "inverting the matrix", because the matrix that corresponds to the linear transformation is not invertible. But you should be able to use Gauss-Jordan elimination (row reduction) to figure out whether or not this system has a solution.
Why are we trying to solve the system? Because finding a solution $(x,y)$ to this system is equivalent to finding a vector $(x,y)$ such that $T(x,y)=(1,-4)$. Any solution gives such a vector. And such a vector is a "witness" to the fact that $(1,-4)$ is in $R(T)$ (because $R(T)$ is the collection of all elements $(a,b)$ of $\mathbb{R}^2$ for which there exist a vector $(x,y)$ such that $T(x,y)=(a,b)$).
So: if the system has a solution, then $(1,-4)$ is in the range and the solution of the system is a witness to the fact that it is in the range. If you plug any solution to the system into $T$, you should get $(1,-4)$.
If the system is inconsistent, then that means there are no solutions, so $(1,-4)$ is not in the range.
Added. So you figured out by row reduction that the system is equivalent to the system with augmented matrix $\left(\begin{array}{cr|c} 1 & -\frac{1}{2} & \frac{1}{2}\\ 0 & 0 & 0 \end{array}\right).$ This is the matrix of a system of linear equations; you should be able to find the solutions to this system. Find a solution to this system: that means finding a specific value of $x$ and a specific value of $y$ that is a solution to this (and therefore to the original) system of linear equations. Those specific values of $x$ and $y$ that solve this system, which also solve the original system, should (if you've done everything right), when plugged into $T$, give you $(1,-4)$ as the output. And that will prove explicitly that $(1,-4)$ lies in $R(T)$.