The calculations are done in precisely the same way that one does Gaussian Elimination in vector spaces; to be more precise, you are performing the same kind of operations one does in computing Smith normal forms.
You start with $M = \left( {\begin{array}{ccc} -1 & x & 1 \\ x+5 & 2 & -3\\ 6 & 2 & x-4 \end{array} } \right).$ Now add $(x+5)$ times the first row to the second row. Multiplying the first row by $(x+5)$ would give $(-(x+5), x^2+5x, x+5)$, which added to $(x+5, 2, -3)$ gives $(0, x^2+5x+2, x+2)$: $ \left( \begin{array}{ccc} -1 & x & 1 \\ 0 & x^2+5x+2 & x+2\\ 6 & 2 & x-4 \end{array} \right).$ Adding $-6$ times the first row to the third row gives $\left(\begin{array}{ccc} -1 &x & 1\\ 0 & x^2+5x + 2 & x+2\\ 0 & 6x+2 & x+2 \end{array}\right),$ exactly what you have.
The point is that the module generated by $\mathbf{m}_1=(-1,x,1)$, $\mathbf{m}_2=(x+5,2,-3)$, and $\mathbf{m}_3=(6,2,x-4)$ is equal to the module generated by $\mathbf{n}_1=(-1,x,1)$, $\mathbf{n}_2=(0,x^2+5x+2,x+2)$, and $\mathbf{n}_3(0,6x+2,x+2)$: because $\begin{align*} \mathbf{n}_1 &= \mathbf{m}_1,\\ \mathbf{n}_2 &= \mathbf{m}_2 + (x+5)\mathbf{m}_1,\\ \mathbf{n}_3 &= \mathbf{m}_3 + 6\mathbf{m}_1; \end{align*}$ so $\langle \mathbf{n}_1, \mathbf{n}_2,\mathbf{n}_3\rangle\subseteq \langle\mathbf{m}_1,\mathbf{m}_2,\mathbf{m}_3\rangle$. On the other hand, $\begin{align*} \mathbf{m}_1 &= \mathbf{n}_1,\\ \mathbf{m}_2 &= \mathbf{n}_1 - (x+5)\mathbf{n}_1,\\ \mathbf{m}_3 &= \mathbf{n}_3 - 6\mathbf{n}_1; \end{align*}$ so $\langle \mathbf{m}_1,\mathbf{m}_2,\mathbf{m}_3\rangle\subseteq \langle\mathbf{n}_1,\mathbf{n}_2,\mathbf{n}_3\rangle$, giving equality.
What's important is that you are allowed to add multiples of one row to another row without changing the submodule generated by the rows. You can perform all the usual operations of Gaussian elimination except multiplying rows by nonzero constants: you should not multiply rows by constants that are not units.
If $R=\mathbb{Q}[x]$, you can multiply a row by any nonzero rational, but you cannot divide by nonconstant polynomials because they are not invertible in $R$. You can get to a diagonal matrix, which will be the Smith normal form.