You mean for the set $U$ to be open.
A function between topological spaces is continuous iff the inverse image of every open set is open. If $D$ is a subset of a topological space $X$ then the open sets of $D$ in the subspace topology are exactly the sets of the form $D \cap V$ where $V$ is open in $X$. So a function $f:D\rightarrow Y$ is continuous iff for all $U$ open in $Y$, there exists $V$ open in $X$ such that $f^{-1}(U)=D\cap V$.
If you are only familiar with metric spaces and would like to understand why a function is continuous iff the inverse image of every open set is open, think about the standard definition of continuous functions on metric spaces.
A function $f:X\rightarrow Y$ between two metric spaces is continuous if and only if for all $x\in X$ and $\epsilon >0$ there exists $\delta >0$ s.t. if $x'\in X$ and $\text{d}(x,x') <\delta$ then $\text{d}(f(x),f(x')) <\epsilon$.
This basically means that for each $x$, the inverse image of every epsilon ball around $f(x)$ contains some delta ball around $x$.
If we know that the inverse image of every open set is open, then the inverse images of epsilon balls must be open. Choose a point $x\in X$. Let $\epsilon >0$ and let $B_\epsilon$ be a ball of radius $\epsilon$ and center $f(x)$. Then $f^{-1}(B_\epsilon)$ contains $x$, and is open. Thus $x$ will be an interior point of $f^{-1}(B_\epsilon)$. This means that there is some small delta ball around $x$ that is entirely contained in $f^{-1}(B_\epsilon)$. From here you can see that the definition of continuity is satisfied.
You can also check that if $D$ is a subset of $X$, and we view $D$ as a metric space with metric inherited from $X$, then $D$ has the subspace topology.