Pick bases for $X$ and $Y$ say $\beta=(x_1,x_2,\dots,x_m)$ and $\gamma=(y_1,y_2,\dots,y_n)$. Suppose in addition that $X$'s basis is a completion of a basis for $S$, so that $(x_1,\dots,x_\ell)$ is a basis for $S$. Write $[x]$ for a vector $x$ written as a (column) coordinate vector. We can write down a coordinate matrix for $B$, call it $[B]$ -- it's $(i,j)$-entry is $B(x_i,y_j)$. Thus $B(x,y)=[x]^T [B] [y]$.
Requiring $B(S,y)=0$ means $[x_i]^T[B][y]=0$ for $i=1,\dots,\ell$. Put another way,
$ \left[ \begin{array}{c} [x_1]^T \\ [x_2]^T \\ \vdots \\ [x_\ell]^T \end{array}\right] [B] [y] = P[B][y] = \begin{bmatrix} 0 \\ 0 \\ \vdots \\ 0 \end{bmatrix} $
where $P$ is the matrix obtained by stacking the transposes of coordinate vectors of $S$'s basis. The dimension of the annihilator of $S$ is the same as the nullity of $P[B]$. Since $P$'s rows are linearly independent, $\mathrm{rank}(P)=\ell$ and so the rank of $P[B]$ is at most $\ell$. Thus its nullity is at least $n-\ell$. Therefore, $\dim(S)+\dim(ann(S)) \geq \ell+(n-\ell) = \dim(Y)$. To get equality we need $\mathrm{rank}(P[B])=\ell$.