Yours is not a bad guess, but, to begin with, there is a problem: $\mu_M\left( \frac{1}{x} \right)$ needs not to be a polynomial.
Nevertheless, I think we can improve your idea: for any degree $n$ polynomial $p(x)$, define its conjugate (I'm not sure if this guy has already a name in the literature: please, correct me [EDIT. According to Georges, this is called the reciprocal polynomial ]) as
$ \overline{p}(x) = x^n p\left( \frac{1}{x} \right) \ . $
Clearly, the conjugate of a polynomial is still a polynomial, and you can easily verify that:
- $\overline{\overline{p}}(x) = p(x)$. [EDIT: if $p(x)$ has non-zero constant term. See Georges' comment.]
- $\overline{p(x)}\overline{q(x)} = \overline{p}(x)\cdot\overline{q}(x)$
I claim that the result is the following: if $\mu_M (x)= a_0 + a_1 x + \dots + a_{n-1}x^{n-1} + x^n$, then
$ \mu_{M^{-1}} (x) = \frac{1}{a_0}\overline{\mu_M} (x) \ . $
In order to prove this, we'll need the following lemma.
Lemma. Let $M$ be an invertible matrix and $p(x)$ a polynomial such that $p(M) = 0$. Then $\overline{p}(M^{-1}) = 0$.
Proof of the lemma. Indeed, $\overline{p}(M^{-1}) = (M^{-1})^n p(M) = 0$.
Hence, since $\mu_M$ annihilates $M$, so does $\overline{\mu_M}$ with $M^{-1}$.
We have to prove that $\frac{1}{a_0}\overline{\mu_M}$ has the characteristic property of the minimal polynomial of $M^{-1}$. Namely, that it has no proper divisor which also annihilates $M^{-1}$.
So, assume there were two polynomials $p(x), q(x)$ such that
$ \frac{1}{a_0}\overline{\mu_M} (x) = p(x)q(x) $
and moreover $p(M^{-1}) = 0$. Then, taking conjugates in this last equality, we would obtain
$ \frac{1}{a_0}\mu_M (x) = \overline{p}(x)\cdot\overline{q}(x) \ . $
But, because of the lemma, $\overline{p}(M) = 0$. So, by definition of the minimal polynomial of $M$,
$ \mu_M (x) = \overline{p}(x) \qquad \text{(normalized)} \ . $
Taking conjugates again, we would have that, up to a constant,
$ \overline{\mu_M} (x) = p(x) \ . $