$$f(x) = \begin{cases} \frac{ \langle Mx, x \rangle}{\| x \|}, & \text{for } x \neq 0 \\\\ 0, & \text{for } x = 0 \end{cases}$$
$M$ is a symmetric matrix and $x \in \mathbb{R^n}$
Is there a particular property from linear algebra I need to keep in mind?
With $x = (x_1,x_2, \ldots,x_n) \in \mathbb{R}^n$ we have
$$\langle Mx,x \rangle = \sum_{j=1}^n \sum_{k=1}^n m_{jk}x_j x_k, \\ \|x\| = \left(\sum_{j=1}^n x_j^2 \right)^{1/2}$$
Let $(Mx)_i$ denote the $i$th component of $Mx$. For $x \neq 0$, we have the partial derivatives
$$\frac{\partial}{\partial x_i}\langle Mx,x \rangle = 2 (Mx)_i, \\ \frac{\partial}{\partial x_i}\|x\| = \frac{x_i}{\|x\|} , \\ \frac{\partial f}{\partial x_i} = \frac{1}{\|x\|}\frac{\partial}{\partial x_i}\langle Mx,x \rangle - \frac{\langle Mx,x \rangle}{\|x\|^2}\frac{\partial}{\partial x_i}\|x\| = \frac{2(Mx)_i}{\|x\|} - \frac{x_i}{\|x\|^3}.$$
Since the partial derivatives of $f$ are compositions of continuous functions, they are continuous in open neighborhoods of any point $x \neq 0$, the derivative $Df(x)$ exists at such $x$.
We can show that $f$ is not differentiable at $x = 0$ by showing that directional derivatives do not exist
For $x = 0$ and $u \neq 0$ in $\mathbb{R}^n$ and with $f(0) =0$, the directional derivative at $0$ is defined as
$$\begin{align}D_uf(0 ) &= \lim_{t \to 0} \frac{f(0 + tu) - f(0)}{t} \\ &= \lim_{t \to 0} \frac{\langle M(tu),(tu) \rangle}{\|tu\|t} \\ &= \lim_{t \to 0} \frac{t^2\langle Mu, u\rangle}{|t|\|u\|t} \\ &= \lim_{t \to 0} \frac{t}{|t|}\frac{\langle Mu, u\rangle}{\|u\|}\end{align}.$$
The limit does not exist since left- and right-hand limits are unequal:
$$\lim_{t \to 0+} \frac{f(0 + tu) - f(0)}{t} = \frac{\langle Mu, u \rangle}{\|u\|}, \\ \lim_{t \to 0-} \frac{f(0 + tu) - f(0)}{t} = - \frac{\langle Mu, u \rangle}{\|u\|}.$$