There is a problem of orientation here; therefore it is not enough to consider orthogonality only.
Let $\gamma:\ s\mapsto z(s)=\bigl(x(s),y(s)\bigr)$ be our curve parametrized by arc length and consider the point $z_0=z(s_0)$ of $\gamma$. At $z_0$ we set up a new orthonormal frame given by $e_1:=(\dot x,\dot y)$ and $e_2:=(-\dot y, \dot x)$, where I have suppressed the reference to $s_0$. Note that this system is again positively oriented, i.e., you get $e_2$ by rotating $e_1$ ninety degrees counterclockwise.
As $\dot x^2 +\dot y^2\equiv1$ we have $\langle e_1,\ddot z\rangle=\dot x\ddot x+\dot y\ddot y=0$ (this we knew before) which implies $\ddot z=\kappa e_2$ for some $\kappa\in{\mathbb R}$. What is at stake here is the sign of $\kappa=\langle e_2,\ddot z\rangle$.
To clear this question we shall look at the polar angle $\theta$ of $\dot z$ which is defined by $\theta:=\arg(\dot x,\dot y)$. As
$$\nabla\arg(\xi,\eta)=\Bigl({-\eta\over \xi^2+\eta^2},{\xi\over \xi^2+\eta^2}\Bigr)$$
we obtain by the chain rule
$$\dot\theta ={-\dot y\over \dot x^2 +\dot y^2}\ \ddot x\ +\ {\dot x\over \dot x^2+\dot y^2}\ \ddot y =\langle e_2,\ddot z\rangle =\kappa\ .\qquad\qquad(*)$$
This equation is the key to our problem. Number one, it tells us that the quantity $\kappa$ is nothing else but the (signed!) angular velocity by which the tangent vector $\dot z$ turns along $\gamma$ as $s$ increases, so we are allowed to call $\kappa$ the curvature of $\gamma$ at the point $z_0$.
Number two, we now look at the sign of $\kappa$, and this brings us to the solution of your original problem. If the curve $\gamma$ turns to the left at $z_0$ then its polar angle $\theta$ is increasing there, and the center $c$ of the oscillating circle at $z_0$ will lie to the left of $\gamma$, i.e. in the direction of $e_2$. On the other hand the equation $(*)$ tells us that $\kappa$ is positive, whence $\ddot z=\kappa e_2$ points in the same direction as $e_2$, which means in the direction where $c$ lies.
Similarly, if the curve $\gamma$ turns to the right at $z_0$ then its polar angle $\theta$ is decreasing there, and the center $c$ of the oscillating circle at $z_0$ will lie to the right of $\gamma$, i.e. in the direction of $-e_2$. On the other hand the equation $(*)$ tells us that $\kappa$ is negative, whence $\ddot z=\kappa e_2$ points in the same direction as $-e_2$ which again is in the direction where $c$ lies.
If, however the curve $\gamma$ is parametrized by "time" $t$ in the form $t\mapsto \tilde z(t)$ then its arc length $s$ is given by $$s(t)\ =\ \int_0^t\sqrt{x'^2(t)+y'^2(t)}\ dt\ ,$$
and the two functions $s\mapsto z(s)$ and $t\mapsto \tilde z(t)$ are related by
$$\tilde z(t)\ =z\bigl(s(t)\bigr)\qquad(0\leq t\leq T)\ .$$
Differentiating with respect to $t$ (denoted by $'$) gives, using the chain rule,
$$\tilde z'\ =\ \dot z\ s' = v\ e_1\qquad\qquad(3)$$
where $v:=s'=\sqrt{x'^2+y'^2}>0$ is the "velocity" along $\gamma$ measured by a speedometer. Differentiating (3) with respect to $t$ once more we get
$$\tilde z''=\ddot z\ s'^2 + \dot z\ s''\ .$$
We can write this in the following form:
$$\tilde z''=v^2 \kappa e_2 + a e_1\ ,$$
where $s''=v'=:a$ is the speed gain along $\gamma$. The last equation tells us that the vector $\tilde z''$ has two components: a component parallel to $\tilde z'=v e_1$ coming from the speed gain and a component orthogonal to $z'$ which is the centripetal acceleration. The word says it: This component is directed towards the center $c$ of the osculating circle, because ${\rm sgn}(\langle \tilde z'', e_2\rangle) ={\rm sgn}(\kappa)$.