Since I am not completely sure about your background, I will review some linear algebra that will be useful for us. Apologies if the answer is too basic. I avoid the use of the Gram-Schmidt process in this answer.
You might be familiar that a Hilbert space is an inner product space (that is also complete w.r.t. the metric induced by the inner product, but this point is irrelevant for us). In your case, the inner product space is $L^2[0,1]$ with the inner product $(f,g) \mapsto \langle f, g \rangle := \int_0^1 f(x) g(x) dx$. I will call the elements of this function space "vectors".
Let $A$ be a given subspace and $h$ a vector not necessarily in $A$. It is a basic linear algebraic fact that the point $a$ of $A$ that is closest to $h$ is the projection of $h$ onto $A$, henceforth denoted $h_{\Pi}$. So in order to determine the distance of a vector $h$ from a subspace $A$, one should project $h$ onto the subspace $A$, and compute the distance of $h$ from its projection. To do this, we will decompose $h$ into a sum $h_{\Pi} + h_{\bot}$, such that $h_{\Pi} \in A$ and $h_{\bot}$ is orthogonal to $A$.
In the given example, $h= f_1$ and $A = \operatorname{Span} \{ x, x^2 \}$, so we can write $h_\Pi = ax + bx^2$ for some undetermined scalars $a, b \in \mathbb R$. We have the constraint that $h_{\bot}$ is orthogonal to $\{ x, x^2 \}$; i.e., $\langle h_{\bot}, x \rangle = 0$ and $\langle h_{\bot}, x^2 \rangle = 0$. Since $ h_{\Pi} + h_\bot = h, $ we then have $ \begin{eqnarray*} \langle h_\Pi, x \rangle &=& \langle h, x \rangle, \\ \langle h_\Pi, x^2 \rangle &=& \langle h, x^2 \rangle. \end{eqnarray*} $ That is, $ \begin{eqnarray*} \int_0^1 (ax + bx^2) x \ dx &=& \int_0^1 1 \cdot x \ dx, \\ \int_0^1 (ax + bx^2) x^2 \ dx &=& \int_0^1 1 \cdot x^2 \ dx. \end{eqnarray*} $ This gives us the linear system of equations: $ \begin{eqnarray*} \frac13a + \frac14b &=& \frac12, \\ \frac14a + \frac15b &=& \frac13; \end{eqnarray*} $ solving this gives $a$ and $b$.
Final steps. Using the computed values of $a$ and $b$, we can compute $h_\bot(x) = h(x) - (ax + bx^2)$. The required distance is just $ \| h_{\bot} \| = \left( \int_0^1 h_\bot(x)^2 dx \right)^{1/2}. $