If $\det(A - \lambda B) = 0$, there exists some vector $v$ such that $(A-\lambda B)v = 0$. Multiply through by $\bar{v}^T$, you get
$ \bar{v}^TAv = \lambda \bar{v}^TB v $
Since $A$ is Hermitian,
$ \bar{v}^TAv = v^T A^T \bar{v} = v^T \bar{A} \bar{v} \in \mathbb{R}$
so if $\bar{v}^T B v \neq 0$, the corresponding $\lambda$ is real.
BTW, your statement that
if $B$ is positive semidefinite, it is easy to see $p(\lambda)$ has only real roots
is false. Let
$ A = B = \begin{pmatrix} 0 & 0 \\ 0 & 1 \end{pmatrix} $
then $\det( A - \lambda B) = 0$ for any complex number $\lambda$.
It is also not enough that $B$ be non-degenerate. Let
$ A = \begin{pmatrix} 0 & 1 \\ 1 & 0\end{pmatrix} \qquad B = \begin{pmatrix} -1 & 0 \\ 0 & 1\end{pmatrix} $
The polynomial $p(\lambda) = -\lambda^2 - 1$ and has no real roots. This happens because the "eigenvector"s in this problem are the vectors $(1,1)^T$ and $(1,-1)^T$
Of course, a sufficient condition would be that $B$ is positive or negative definite. But this is far from necessary. A slightly more general result is Theorem 10.1 in the paper of Lancaster and Rodman
Theorem Let $A$ and $B$ be Hermitian. Suppose there exists a pair of real numbers $\alpha,\beta$ such that $C:= \alpha A + \beta B$ is positive semi-definite with $\ker C \subset \ker A \cap \ker B$, then $A$ and $B$ are simultaneously diagonalisable.
Note that this rules out the second example (for which there cannot be $\alpha,\beta$ to make $C$ positive semi-definite), but this is not sufficient to deal with the first example: if $\ker A\cap \ker B$ is non-empty, the polynomial $p(\lambda) \equiv 0$.
In general the problem is quite complicated, and the problem you want to ask is not the one that you asked. You should not be asking about roots of $p(\lambda)$ (since if $B$ is not non-degenerate, $p(\lambda)$ may have degree smaller than the full dimension), but you should be asking about whether $A$ and $B$ can be simultaneously diagonalized. For that I suggest you consult the linked paper and the references given in Section 10.