Suppose i have a square matrix $A_{ij}$ and a test matrix $M_{ij}$. I want to find the integral curve that joins $M_{ij}$ to $A^{-1}_{ij}$ Following the gradient descent path of the following cost function:
$ F_{A}(M) := \sum_{ij} [( M A - I )_{ij}]^2 $
Of course, the curve does not always end in to $A^{-1}$ because sometimes $M$ lies in a local-minima basin (i.e. not connected to the global minima).
Locally, the gradient descent path is described by the gradient of $F_{A}$
$ \nabla_{M} F_{A} = 2 (M A - I ) A^{T} $
or in component notation (trying very hard to not use Einstein summation notation), looks like:
$ \frac{\partial F_{A}^{(ij)} }{ \partial M_{xy}} = 2 \sum_{k} ( \sum_{p} (M_{ip} A_{pk}) - \delta_{ik} ) A_{jk} $
now, it is pretty clear from the above expression that its third derivative is zero.
So if i try to compute the gradient descent path to higher orders, it seems that i'm left with a quadratic curve with no further corrections.
Question: what is wrong with this approach to obtain a matrix inverse? It seems strange to me that the gradient descent curve is a simple polynomial form. Maybe i'm doing something wrong? I feel like i'm missing something very obvious, but i can't think right now what is it.