You're right that 5 would be the adjacent side. So we have a right triangle, where the adjacent side is 5, the opposite side is -12, and the hypotenuse is 13.
$\sec \theta = \frac{1}{\cos \theta}$, so $\sec \theta$ is (hypotenuse)/(adjacent) $=\frac{13}{5}$.
$\cot \theta = \frac{1}{\tan \theta}$, so $\sec \theta$ is (adjacent)/(opposite) $=-\frac{5}{12}$.
Alternatively, note that $\sin ^2 \theta + \cos^2 \theta = 1$, so $(-\frac{12}{13})^2 + \cos^2 \theta = 1$, and $\cos \theta = \pm \frac{5}{13}$. Since we're in quadrant IV, we know that $\cos \theta$ is positive, so $\cos \theta = \frac{5}{13}$.
Then $\sec \theta = \frac{1}{\cos \theta} = \frac{13}{5}$.
And $\cot \theta = \frac{1}{\tan \theta} = \frac{\cos \theta}{\sin \theta} = -\frac{5}{12}$.