I know that the function $(\mathbf{a}-\mathbf{b})'(\mathbf{a}-\mathbf{b})$ is convex in $\mathbf{a}$ ($\mathbf{a}$ and $\mathbf{b}$ are vectors, not scalars). Would $(\mathbf{a}-\mathbf{b})'(\mathbf{a}-\mathbf{b})\mathbf{a}'$ which is a cubic function be convex too? Is there a simple way to check this?
Edit (after martini's comment): the function is $(\mathbf{a}-\mathbf{b})'\mathbf{a}'S \mathbf{b}(\mathbf{a}-\mathbf{b})$ where the term in the middle $\mathbf{a}'S\mathbf{b}$ is a scalar (S is a square matrix and constant like $\mathbf{b}$).