Pick $\epsilon>0$ and choose $\delta>0$ that satisfies 1 and 2 above.
Suppose $x_0\not\in A$. Then, since $A$ is closed, there is a positive $\delta'\le\delta$ so that if $|x-x_0|<\delta'$, $x\not\in A$. Then, the Mean Value Theorem says
$$
\left|\frac{f(x)-f(x_0)}{x-x_0}-y\right|=\left|f'(\xi)-y\right|<\epsilon\tag{1}
$$
for some $\xi$ between $x$ and $x_0$, and therefore, $\xi\not\in A$.
Suppose $x_0\in A$ and $|x-x_0|<\delta$. If $x\in A$, then
$$
\left|\frac{f(x)-f(x_0)}{x-x_0}-y\right|<\epsilon\tag{2}
$$
If $x\not\in A$, let $a$ be the point in $A$ closest to $x$ so that $|x-a|+|a-x_0|=|x-x_0|$; that is, either $a=x_0$ or $a$ is between $x$ and $x_0$.
If $a=x_0$, then no point between $x$ and $x_0$ is in $A$ and the Mean Value Theorem then says
$$
\left|\frac{f(x)-f(x_0)}{x-x_0}-y\right|=\left|f'(\xi)-y\right|<\epsilon\tag{3}
$$
for some $\xi$ between $x$ and $x_0$, and therefore, $\xi\not\in A$.
If $a$ is between $x$ and $x_0$, then because no point between $x$ and $a$ is in $A$,
$$
\left|\frac{f(x)-f(a)}{x-a}-y\right|=\left|f'(\xi)-y\right|<\epsilon\tag{4}
$$
for some $\xi$ between $x$ and $a$. Furthermore, since $a\in A$,
$$
\left|\frac{f(a)-f(x_0)}{a-x_0}-y\right|<\epsilon\tag{5}
$$
Since $|x-a|+|a-x_0|=|x-x_0|$,
$$
\begin{align}
&\left|(x-x_0)\left(\frac{f(x)-f(x_0)}{x-x_0}-y\right)\right|\\
&=\left|(x-a)\left(\frac{f(x)-f(a)}{x-a}-y\right)+(a-x_0)\left(\frac{f(a)-f(x_0)}{a-x_0}-y\right)\right|\\
&<\epsilon|x-a|+\epsilon|a-x_0|\vphantom{\frac{f(x)-f(x_0)}{x-x_0}}\\
&=\epsilon|x-x_0|\vphantom{\frac{f(x)-f(x_0)}{x-x_0}}
\end{align}
$$
Therefore,
$$
\left|\frac{f(x)-f(x_0)}{x-x_0}-y\right|<\epsilon\tag{6}
$$
In conclusion, $(1)$, $(2)$, $(3)$, and $(6)$ cover all cases, and since $\epsilon>0$ was arbitrary, we get that $f'(x_0)=y$.