There is a problem of orientation here; therefore it is not enough to consider orthogonality only.
Let $\gamma:\ s\mapsto z(s)=\bigl(x(s),y(s)\bigr)$ be our curve parametrized by arc length and consider the point $z_0=z(s_0)$ of $\gamma$. At $z_0$ we set up a new orthonormal frame given by $e_1:=(\dot x,\dot y)$ and $e_2:=(-\dot y, \dot x)$, where I have suppressed the reference to $s_0$. Note that this system is again positively oriented, i.e., you get $e_2$ by rotating $e_1$ ninety degrees counterclockwise.
As $\dot x^2 +\dot y^2\equiv1$ we have $\langle e_1,\ddot z\rangle=\dot x\ddot x+\dot y\ddot y=0$ (this we knew before) which implies $\ddot z=\kappa e_2$ for some $\kappa\in{\mathbb R}$. What is at stake here is the sign of $\kappa=\langle e_2,\ddot z\rangle$.
To clear this question we shall look at the polar angle $\theta$ of $\dot z$ which is defined by $\theta:=\arg(\dot x,\dot y)$. As $\nabla\arg(\xi,\eta)=\Bigl({-\eta\over \xi^2+\eta^2},{\xi\over \xi^2+\eta^2}\Bigr)$ we obtain by the chain rule $\dot\theta ={-\dot y\over \dot x^2 +\dot y^2}\ \ddot x\ +\ {\dot x\over \dot x^2+\dot y^2}\ \ddot y =\langle e_2,\ddot z\rangle =\kappa\ .\qquad\qquad(*)$ This equation is the key to our problem. Number one, it tells us that the quantity $\kappa$ is nothing else but the (signed!) angular velocity by which the tangent vector $\dot z$ turns along $\gamma$ as $s$ increases, so we are allowed to call $\kappa$ the curvature of $\gamma$ at the point $z_0$.
Number two, we now look at the sign of $\kappa$, and this brings us to the solution of your original problem. If the curve $\gamma$ turns to the left at $z_0$ then its polar angle $\theta$ is increasing there, and the center $c$ of the oscillating circle at $z_0$ will lie to the left of $\gamma$, i.e. in the direction of $e_2$. On the other hand the equation $(*)$ tells us that $\kappa$ is positive, whence $\ddot z=\kappa e_2$ points in the same direction as $e_2$, which means in the direction where $c$ lies.
Similarly, if the curve $\gamma$ turns to the right at $z_0$ then its polar angle $\theta$ is decreasing there, and the center $c$ of the oscillating circle at $z_0$ will lie to the right of $\gamma$, i.e. in the direction of $-e_2$. On the other hand the equation $(*)$ tells us that $\kappa$ is negative, whence $\ddot z=\kappa e_2$ points in the same direction as $-e_2$ which again is in the direction where $c$ lies.
If, however the curve $\gamma$ is parametrized by "time" $t$ in the form $t\mapsto \tilde z(t)$ then its arc length $s$ is given by s(t)\ =\ \int_0^t\sqrt{x'^2(t)+y'^2(t)}\ dt\ , and the two functions $s\mapsto z(s)$ and $t\mapsto \tilde z(t)$ are related by $\tilde z(t)\ =z\bigl(s(t)\bigr)\qquad(0\leq t\leq T)\ .$ Differentiating with respect to $t$ (denoted by $'$) gives, using the chain rule, \tilde z'\ =\ \dot z\ s' = v\ e_1\qquad\qquad(3) where v:=s'=\sqrt{x'^2+y'^2}>0 is the "velocity" along $\gamma$ measured by a speedometer. Differentiating (3) with respect to $t$ once more we get \tilde z''=\ddot z\ s'^2 + \dot z\ s''\ . We can write this in the following form: \tilde z''=v^2 \kappa e_2 + a e_1\ , where s''=v'=:a is the speed gain along $\gamma$. The last equation tells us that the vector \tilde z'' has two components: a component parallel to \tilde z'=v e_1 coming from the speed gain and a component orthogonal to z' which is the centripetal acceleration. The word says it: This component is directed towards the center $c$ of the osculating circle, because {\rm sgn}(\langle \tilde z'', e_2\rangle) ={\rm sgn}(\kappa).