I was thinking about the following problem that says:
Let $a,b\in \mathbb{R}$.Let $y=(y_1,y_2)^{t}$ be a solution of the system of $y'_1=y_2$,$y'_2=ay_1+by_2$. Then which of the following options is correct?
Every solution $y(x)\rightarrow 0$ as $x\rightarrow \infty$ if
(a) $a<0,b<0$
(b) $a<0,b>0$
(c) $a>0,b>0$
(d) $a>0,b<0.$
My attempts: From the second equation we see, $y_1''=ay{_1}'+by{_2}'=ay_2+by{_2}'$.The auxiliary equation of this D.E. is $m^2-bm-a=0$ and hence $m_1=(b+\sqrt(b^2+4a))/2$,$m_2=\frac{b-\sqrt{b^2+4a}}{2}$. Hence the solution is given by $y=c_1e^{m_1x}+c_2e^{m_2x}$.Now,$y_2$ will tend to 0 for any choice of $c_1$ and $c_2$ if and only if the real parts of m1 and m2 are negative(Thanks to Antonio Vargas sir).Now $m_2<0$ gives $b<\sqrt(b^2+4a)$ and so $b^2 and hence $a>0$. From here, i could not progess. Please help. Thanks in advance for your time.