This is now a very old question, but I want to correct an error in the original post. It is not the case that a relatively finite subset of a first countable space must have countable interior: the space $\omega_1$ with the order topology is a counterexample. Let $A = \{0\} \cup \{\alpha+1:\alpha \in \omega_1 \}$, the set of isolated points in $\omega_1$; $\operatorname{bdry}A = \omega_1 \setminus A$, the set of countable limit ordinals. Every infinite subset of $A$ has a cluster point in bdry $A$, so every nbhd of $\operatorname{bdry}A$ must be cofinite in $\omega_1$. That is, $A$ is an uncountable set of isolated points that is relatively finite in $\omega_1$
As for the original question, since a subset $A$ of a space $X$ is relatively finite in $X$ iff its interior is, it seems fair to say that relative finiteness is ‘really’ a property of open sets, and it's in that form (if at all) that I’d expect to find prior uses. I've not actually seen any, however.
As MartianInvader pointed out, in metric spaces it’s equivalent to having discrete interior and compact closure. More generally, in countably paracompact $T_3$ spaces it’s equivalent to having discrete interior and countably compact closure.
Proposition: Let $X$ be a $T_3$ space, and let $Y \subseteq X$ have discrete interior and countably compact closure; then $Y$ is relatively finite in $X$.
Proof: Let $W$ be an open nhbd of $\operatorname{bdry}Y$, and suppose that $Y \setminus W$ is infinite. Let $\{W_n:n \in \omega \}$ be a partition of $Y \setminus W$; points of $Y \setminus W$ are isolated in $X$, so $\{W\} \cup \{X \setminus \operatorname{cl}Y \cup \{W_n:n \in \omega \}$ is a countable open cover of $X$ that clearly has no finite subcover.
Theorem: Let $X$ be a countably paracompact $T_3$ space. If $Y$ is a relatively finite subset of $X$, then $\operatorname{cl}Y$ is countably compact. (In fact, $Y$ need only satisfy the weaker property that if $W$ is an open nbhd of $\operatorname{bdry}Y$, then $Y \setminus W$ is compact.)
Proof: Let $H = \operatorname{cl}Y$, and suppose that $H$ is not countably compact. Then $H$ has an increasing open cover $\mathcal U = \{U_n:n \in \omega\}$ with no finite subcover. Countable paracompactness of $X$ implies that $\mathcal U$ has an open refinement $\mathcal V = \{V_n:n \in \omega\}$ such that $\operatorname{cl}V_n \subseteq U_n$ for each $n \in \omega$. By passing to a (faithfully indexed) refinement if necessary, we may further assume that $\mathcal V$ is locally finite.
Let $V = \operatorname{int}Y$. Let $n(0) = 0$, and choose $x_0 \in V \cap V_0$. Suppose that for some $m > 0$ and all non-negative integers $k < m$ we have chosen $n(k) \in \omega$ and $x_k \in V$ so that (a) $x_{m-1} \in V \cap V_{n(m-1)}$, and (b) $n(i) < n(k)$ and $x_i \in V \cap V_{n(i)} \setminus V_{n(k)}$ whenever $0 \le i < k < m$. $\mathcal V$ is point-finite and has no finite subfamily covering $H$, so there is an $n(m) \in \omega$ such that $n(m) > n(m-1)$ and $\{x_k:0 \le k < m\} \cap V_i = \emptyset$ for all $i \ge n(m)$. Clearly we may now choose $x_m \in V \cap V_{n(m)}$, and the recursion goes through to yield a set $A = \{x_n:n \in \omega\} \subseteq V$ such that $x_k \in V \cap V_{n(k)} \setminus V_i$ whenever $i,k \in \omega$ and $i \ge n(k+1)$.
Now let $p$ be any point of $H$. $\mathcal V$ is locally finite, so $p$ has an open nbhd $W$ that meets only finitely many members of $\mathcal V$. Clearly $W \cap A$ is finite, so $p$ is not a cluster point of $A$. Thus, $A$ is a closed, discrete subset of $V$, and $X \setminus A$ is an open nbhd of $\operatorname{bdry}Y$ whose relative complement in $Y$ is not compact (and hence certainly not finite).