The structure sheaf is actually not so simple. On base open affines $U_f$, we define it to be $\mathcal{O}_X(U_f) = A_f$ (localization), but as you say this doesn't tell us what $\mathcal{O}_X(U)$ should be if $U$ is open but not affine.
If you look at Hartshorne, for example, you'll see that the better way to proceed is to first define the stalks of $\mathcal{O}_X$, and then take $\mathcal{O}_X(U)$ to be the ring of local sections (maps from $U$ to the stalks over points in $U$ satisfying some nice conditions). It is not too hard to check that on an open affine $U_f$, this is equivalent to $\mathcal{O}_X(U_f) = A_f$. This is very similar to sheafification of a presheaf.
Another way to think of this, if it helps, is that since any open set $U$ can be covered by affine open subsets $\{U_{f_i}\}_{i \in I}$, so a section of $\mathcal{O}_X(U)$ can be thought of as a collection of sections $s_i \in A_{f_i}$, satisfying the condition that $s_i$ and $s_j$ agree on the intersection of $U_{f_i}$ and $U_{f_j}$ (and we can cover $U_{f_i} \cap U_{f_j}$ by open affines if necessary, to make sense of this condition). The point is that since the base open affines $U_f$ form a base of the topology on $X$, any sheaf on $X$ is uniquely determined by its values on these base open affines.
There was a good MO thread on this, which can be found here.