I think you need to be more specific regarding what you mean by instability because you have to choose that for Lyapunov, orbital, structural and input-output stability. There are also many kinds/classes of nonlinear equations that these can apply to.
Review these nice survey papers of stability and you'll see what is meant by instability based on the type of stability you are talking about. Additionally, see the bibliographies for books and other papers:
http://www.me.berkeley.edu/ME237/7_stability.pdf, and
http://www.ee.cityu.edu.hk/~gchen/pdf/C-Encyclopedia04.pdf .
Lastly, see the section titled "Stability of fixed points" at http://en.wikipedia.org/wiki/Stability_theory for stability statements that couple eigenvalues to a statement about the different ways to look at stability.
There are many wonderful books dedicated to this subject.
Is this what you were looking for or book references too?
Regards -A