I'll consider the original question as posted, with irreducibility of a polynomial in $y$ over the field $F(a(x))$ as goal. The statement is not true as given: one needs the additional hypothesis that $p(x,y)\notin F[x]$, since otherwise $p(a(x),y)\in F(a(x))$ is a unit in $F(a(x))[y]$, and therefore neither reducible nor irreducible.
The first step is to show that $p(x,y)$ is irreducible as a polynomial in $y$ over the field $F(x)$, in other words irreducible as element of $F(x)[y]$. With the assumption that $p(x,y)\notin F[x]$ this amounts to showing that $p(x,y)$ is not reducible in $F(x)[y]$. This is the hardest part, but it is a standard result: a lemma attributed to Gauss says that is a polynomial (here in $y$) with coefficients in a Unique Factorization Domain (here $F[x]$) is irreducible, then its is also irreducible over its field of fractions (here $F(x)$).
The second step is to show that from $p(x,y)$ irreducible in $F(x)[y]$ one may conclude that $p(a(x),y)$ is irreducible in $F(a(x))[y]$. But this is mostly a formality, as in the answer by Lior B-S, once one realises that $a(x)$, like any non-constant polynomial, is transcendental over $F$. This transcendence allows defining a ring morphism $F(x)\to F(x)$ that fixes $F$ and sends $x\mapsto a(x)$ (it more generally sends $\frac{P(x)}{Q(x)}\mapsto\frac{P(a(x))}{Q(a(x))}$) and whose image is by definition the subfield $F(a(x))\subseteq F(x)$; since $F(x)$ is a field, this morphism is necessarily injective, and therefore defines an isomorphism of fileds $F(x)\to F(a(x))$. Applying this isomorphism to the coefficients induces an isomorphsims of polynomial rings $F(x)[y]\to F(a(x))[y]$ that sends of $p(x,y)\mapsto p(a(x),y)$ and preserves irreducibility.
For completeness I'll fix the loose ends in the second step. Any non-constant polynomial $a(x)\in F[x]$ is transcendental over $F$ because for any nonnzero $P(x)\in F[x]$ one has $\deg P(a(x))=\deg P\times \deg a$ and in particular $P(a(x))\neq0$: the contribution of the leading term of $P$ (which has that degree) cannot be cancelled by the contribution of any other term of $P$. And this allows defining the indicated morphism $F(x)\to F(x)$, because it ensures that $Q(a(x))\neq0$ for the denominator $Q(x)$: the image of every rational function is well defined.