Yes, there are only finitely many values of $a$ for which $p(x_1,...,x_n)+a$ is reducible.
In order to use the mighty tools of projective algebraic geometry, let us consider the homogeneization $P(x_0,x_1,...,x_n)$ of $p(x_1,...,x_n)$ ($P$ is an irreducible homogeneous polynomial of degree $d$) and ask whether $P(x_0,x_1,...,x_n)+ax_0^n$ is irreducible with only finitely many exceptions (The factorizability of the homogeneized polynomial is equivalent to the factorizability of the original polynomial).
The answer is yes.
Indeed in the projective space $\mathbb P^{N_d}\: (N_d=\binom {n+d}{d}-1)$ of homogeneous polynomials of degree $d$ in $n+1$ variables , the reducible ones form a union of finitely many irreducible strict subvarieties: the images of the morphisms $\mathbb P^{N_e} \times \mathbb P^{N_f}\to \mathbb P^{N_d}\; (e+f=d)$ given by the multiplication of a polynomial of degree $e$ by one of degree $f$.
The rest is easy: the projective line in $\mathbb P^{N_d}$ given by the family $P(x_0,x_1,...,x_n)+ax_0^n$ is not included in the mentioned union of subvarieties because the polynomial corresponding to $a=0$ is irreducible.
Hence it cuts that variety for only finitely many values of $a\in k$ and for the other values of $a$ both $P(x_0,x_1,...,x_n)+ax_0^n$ and $p(x_1,...,x_n)+a$ are irreducible.