Here's a proof of a more general result. Here I let $\alpha \in \Omega^1(M)$, I define $s_\alpha: M \longrightarrow T^\ast M,$ $p \mapsto (p, \alpha_p),$ and I write $M_\alpha$ for the image of $s_\alpha$. Recall that the canonical symplectic form $\omega_{\text{can}}$ on $T^\ast M$ is exact, $ \omega_{\text{can}} = - d\lambda,$ where $\lambda$ is the Liouville form given locally by $\lambda = \sum_{i = 1}^n \xi_i ~dx_i$ and globally by $\lambda_{(x,\xi)} = (d\pi_{(x,\xi)})^\ast \xi,$ where $\pi: T^\ast M \longrightarrow M$ is the bundle projection.
With the above setup, I claim that $s_\alpha^\ast \lambda = \alpha.$ To see this, recall that by definition, $\lambda_p = (d\pi_p)^\ast \xi$ where $p = (x, \xi) \in T^\ast X$ and $\pi: T^\ast X \longrightarrow X$ is the bundle projection. Then we have $\lambda_{s_\alpha(x)} = (d\pi_{s_\alpha(x)})^\ast \alpha_x,$ and hence \begin{align*} (s^\ast_\alpha \lambda)_x & = (ds_\alpha)^\ast_x \lambda_p \\ & = (ds_\alpha)^\ast_x (d\pi_{s_\alpha(x)})^\ast \alpha_x \\ & = (d\pi_{s_\alpha(x)} \circ (ds_\alpha)_x)^\ast \alpha_x \\ & = d(\pi \circ s_\alpha)_x^\ast \alpha_x \\ & = \alpha_x, \end{align*} since $s_\alpha$ is a section and thus $\pi \circ s_\alpha = \mathrm{Id}_X$. Therefore $s_\alpha^\ast \lambda = \alpha.$
Theorem. Let $\alpha \in \Omega^1(M)$. Then $M_\alpha$ is a Lagrangian submanifold of $(T^\ast M, \omega_\mathrm{can})$ if and only if $\alpha$ is closed.
Proof. Since $M_\alpha$ is the image of the closed embedding $s_\alpha$, we have that \begin{align*} M_\alpha \text{ is Lagrangian} & \iff s_\alpha^\ast \omega_\text{can} = 0 & & \\ & \iff s_\alpha^\ast d\lambda = 0 & & (\omega_\text{can} = -d\lambda) \\ & \iff ds_\alpha^\ast \lambda = 0 & & (d \text{ commutes with pullbacks}) \\ & \iff d\alpha = 0, & & (s_\alpha^\ast \lambda = \alpha) \end{align*} from which the result follows.