Suppose $A,B$ are two $n \times n$ matrices with real entries. Are there some simple conditions on $A,B$ under which the function $ x \mapsto \sqrt{\|Ax\|\|Bx\|}$ from $\Bbb{R}^n \to \Bbb{R}$ is convex?
I use some numerical algorithm to calculate an optimum, and a condition for the convergence of the algorithm is the convexity of the given function. The algorithm converges for all functions of this form, but I did not manage to prove the convexity of the above function.