Given a covariance matrix $\left(\sigma^2 I + \lambda^{-1}K_{\theta}^{-1}\right)$ where $\sigma,\lambda \in \mathbb{R}$ (both positive), $I$ is the $n \times n$ identity matrix and $K_{\theta}^{-1}$ is known. Moreover, $K_{\theta}^{-1}$ is positive definite and symmetric. I want to find an expression for $\lambda$ that satisfies
$\frac{\partial}{\partial \lambda} (2\pi)^{n/2}\det\left(\sigma^2 I + \lambda^{-1}K_{\theta}^{-1}\right)^{-\frac{1}{2}}\exp\left[-\frac{1}{2}y^T\left(\sigma^2 I + \lambda^{-1}K_{\theta}^{-1}\right)^{-1}y\right] = 0$
Here, $n$ is fixed and $y$ is given.
I've been spending a lot of time trying to find an optimal $\lambda$. At the moment I'm not even sure it can be done but I would love to be shown wrong. Here's what I did so far. Since we can also find the derivative of the log of the expression, we have
$\frac{\partial}{\partial \lambda}\left[ -\frac{1}{2}\log\det\left(\sigma^2 I + \lambda^{-1}K_{\theta}^{-1}\right) - \frac{1}{2}y^T\left(\sigma^2 I + \lambda^{-1}K_{\theta}^{-1}\right)^{-1}y\right] = 0$ or $\frac{\partial}{\partial \lambda}\left[\log\det\left(\sigma^2 I + \lambda^{-1}K_{\theta}^{-1}\right)\right] + \frac{\partial}{\partial \lambda}\left[y^T\left(\sigma^2 I + \lambda^{-1}K_{\theta}^{-1}\right)^{-1}y\right]=0$
Concerning the first term, I've found that for a symmetric matrix $M = M(\alpha)$ for some scalar $\alpha$, it holds that $\frac{\partial}{\partial \alpha} \log\det M(\alpha) = \text{tr}\left(M(\alpha)^{-1}\frac{\partial}{\partial \alpha}M(\alpha)\right)$ Applying this to $\frac{\partial}{\partial \lambda}\left[\log\det\left(\sigma^2 I + \lambda^{-1}K_{\theta}^{-1}\right)\right]$ gives $\frac{\partial}{\partial \lambda}\left[\log\det\left(\sigma^2 I + \lambda^{-1}K_{\theta}^{-1}\right)^{-1}\right] = \frac{-1}{\lambda^2}\text{tr}\left[\left(\sigma^2 I + \lambda^{-1}K_{\theta}^{-1}\right)^{-1}K_{\theta}^{-1}\right]$ which I can simplify to $\frac{-1}{\lambda}\text{tr}\left[\left(\lambda\sigma K_{\theta} + I \right)^{-1}\right]$
Concerning the derivation of the second term, I've asked about this in a previous question, which you can read here: Finding the scalar derivative of a matrix product. It turns out that we can simplify $\frac{\partial}{\partial \lambda}\left[y^T\left(\sigma^2 I + \lambda^{-1}K_{\theta}^{-1}\right)^{-1}y\right]$ to $y^T\left(2\lambda\sigma^2 I + K_{\theta}^{-1} + \lambda^2 \sigma^4 K_{\theta}\right)^{-1}y$ Thus, I have $\frac{-1}{\lambda}\text{tr}\left[\left(\lambda\sigma^2 K_{\theta} + I \right)^{-1}\right] + y^T\left(2\lambda\sigma^2 I + K_{\theta}^{-1} + \lambda^2 \sigma^4 K_{\theta}\right)^{-1}y = 0$ which I would like to solve for $\lambda$. Is there any way to do this analytically or am I doomed to numerical methods? Thanks in advance.