$\newcommand{\Span}[1]{\left\langle #1 \right\rangle}$Perhaps a not too complicated example is provided by the following group, where $p$ is any odd prime
$$
G = \Span{ a, b : a^{p^{2}} = b^{p^{2}} = 1, [a, b] = a^{p}}
$$
This has order $p^{4}$, and we have $G'= Z(G) = G^{p} = \Omega_{1}(G) = \Span{a^{p}, b^{p}}$, of order $p^{2}$. (Here $\Omega_{1}(G) = \Span{x \in G : x^{p} = 1}$ has exponent $p$.)
We claim
The subgroup $B = \Span{b}$ is non-normal, yet it permutes with all subgroups $H$.
It suffices of course to check this only for the non-normal subgroups $H$. We are immediately left with the case when $H$ is cyclic of order $p^{2}$.
- If $B \cap H = 1$, then clearly $B H = G = H B$.
- If $B \cap H = \Span{z}$ has order $p$, we may assume $z = b^{p}$. Choose $h \in H$ such that $h^{p} = z$. Then $B H$ contains $w = b^{-1} h \ne 1$, and $$w^{p} = (b^{-1} h)^{p} = b^{-p} h^{p} [h, b^{-1}]^{\binom{p}{2}} = 1$$ as $p$ is odd. So $B H$ contains $Z(G) = G'$, and thus is a subgroup.