1. Well, it seems to me like the author did define the derivative at a point contained in a general set. Only, his definition does not guarantee that the derivative at such a point will be unique.
Naively, open set are special because at every point in an open set there is a lot of room. More precisely, if $a \in U$ and $U$ is open then you can find an open ball around $a$ that remains contained in $U$. Such an open ball around $a$ assures you that the equation
$$
f(a+v)-f(a)=T\cdot v+r(v)
$$
makes sense for all $v$ with sufficiently small $\| v \|$. In particular, it makes the limit
$$
\lim_{v\to 0}\frac{r(v)}{v}=0 \tag{$\star$}
$$
well-defined. Remember, $(\star)$ means that no matter what path $v$ takes in $\mathbb{R}^m$ to go to $0$, $\frac{r(v)}{v}$ will get as close as you want to $0$ following that path.
2. Now, let's consider the simple case $m=n=1$ to illustrate things.
If $X = \{0\}$, for example, then all you have is a function defined at one point. Suppose your function $f$ is such that $f(0)=0$.
Following the definition of the author, $f$ is differentiable at $0$ because it is the restriction of the constant function $g(t):=0$ defined on $\mathbb{R}$. The derivative $g'(0)$ is equal to $0$. So we could say $f$ has derivative $0$ at $0$.
Also following the definition of the author, $f$ is differentiable at $0$ because it is the restriction of the function $h(t):=t$ defined on $\mathbb{R}$. The derivative $h'(0)$ is equal to $1$. So we could say $f$ has derivative $1$ at $0$.
Now you see that since $0\neq1$ we lost the uniqueness of $f'$ at $0$.
This being said, sometimes, the "generalized" derivative will be unique. For example, consider the function $f:\mathbb{R}_{\leq0}\to\mathbb{R}$ defined by $f(t):=-t$. Note that the domain of $f$, which is $\mathbb{R}_{\leq0}$, isn't open in $\mathbb{R}$. However, at the point of interest $t=0$, the left derivative of $f$ exists and equals $-1$. Hence if $g$ is a function differentiable at $0$ such that $f$ is its restriction on $\mathbb{R}_{\leq0}$, then it must be the case that $g'(0)=-1$. Thus $f'(0)$ is unique and equals $-1$ according to the definition of the author.
By the way, in the last example. $(\star)$ doesn't make sense when $a=0$ because $r$ is not defined for $v>0$. Hence the limit in $(\star)$ can only be a left-side limit $\displaystyle\lim_{v\to0^-}$, which is a weaker condition than the full two-sided limit $\displaystyle\lim_{v\to0}$. What you get is the left derivative of $f$ at $0$. As you know, a function $f$ may have a left derivative at $a$ but fail to have a derivative at $a$.