Remember that if $X_1,\ldots X_n$ are a sample from the normal($\mu,\sigma^2$) distribution, then $\sum(X_i-\bar X)^2/\sigma^2$ has chi-square distribution with $n-1$ degrees of freedom.
To apply this to your problem, define $$Z_i:=\frac{Y_i-\theta}\theta.$$ Argue that each $Z_i$ has standard normal distribution. Next, use algebra to show that
$$\sum(Y_i-\bar Y)^2=\theta^2\sum(Z_i-\bar Z)^2.
$$
From this you should be able to conclude that $$U:=\frac{(n-1)S^2}{\theta^2}=\frac{\sum(Y_i-\bar Y)^2}{\theta^2}$$ has chi-square distribution with $n-1$ df, which is free of $\theta$, and is therefore pivotal.