In your example, the $x$-axis is tangent to the curve, the $y$-axis is what is called normal to the curve, and the line you describe coming straight up out of the page is what is called binormal to the curve. The normal line lies in the same plane as the curve, while the binormal direction is the cross product of the tangent and normal directions. Both the normal and binormal lines to a curve are perpendicular to the tangent line as lines, and that motivates why those lines are both called perpendicular to the curve.
If $\vec{r}:\mathbb{R}\to\mathbb{R}^3$, the unit tangent vector $\vec{T}(t)$ is defined as \frac{\vec{r}'(t)}{\|\vec{r}'(t)\|}. It is one unit long and points in the same direction as the curve "is moving".
$\vec{T}(t)$ itself changes as $t$ changes. In your example, as $t$ moves through $0$, $\vec{T}(t)$ points downward-right, then to the right, then upward-right. Anyway, that means we can consider $\frac{d}{dt}\vec{T}(t)$, which I'll just call \vec{T}'(t) for short. The unit normal vector $\vec{N}(t)$ is defined as \frac{\vec{T}'(t)}{\|\vec{T}'(t)\|}. It is one unit long, and points in the direction that $\vec{T}(t)$ is changing, which is the direction that the curve "is bending". In your example, that's straight up.
Locally, $\vec{T}(t)$ and $\vec{N}(t)$ define a plane that contains the second degree Taylor approximation of the curve. This plane is called the osculating plane.
In $\mathbb{R}^3$, there are $3$ dimensions. Locally, $\vec{T}(t)$ and $\vec{N}(t)$ account for two orthogonal directions. A third orthogonal direction can be written down as $\vec{T}(t)\times\vec{N}(t)$, which we call the unit binormal vector $\vec{B}(t)$. It is automatically one unit long and orthogonal to $\vec{T}(t)$ and $\vec{N}(t)$.
Lastly, in your example, $\vec{B}(t)$ is constant, pointing always straight up. Imagine a curve that took full use of all $3$ dimensions and writhes around more. $\vec{B}(t)$ would change as $t$ changes. In fact the more writhing there is, the more $\vec{B}(t)$ would change. This motivates why $\|\frac{d}{dt}\vec{B}(t)\|$ is called the torsion of the curve, denoted $\tau(t)$.
That's space curve vector calculus in a nutshell.