The general solution to the equation $y''+by'+cy=0$ approaches to $0$ as $x$ approaches infinity if
- $b$ is negative $c$ is positive
- $b$ is positive $c$ is negative
- $b$ is positive $c$ is positive
- $b$ is negative $c$ is negative
The general solution to the equation $y''+by'+cy=0$ approaches to $0$ as $x$ approaches infinity if
Hint: to find the solutions of this differential equation, replace $y''$ by $r^2$ and $y'$ by $r$, and solve the quadratic polynomial $$r^2 + br + c = 0$$ to find the roots $r_1,r_2$.
The solution $y(x)$ of the differential equation is then given by $$y(x) = Ae^{r_1x} + Be^{r_2x}$$ if $r_1 \neq r_2$, and otherwise by $$y(x) = Ae^{r_1x} + Bxe^{r_1x}.$$ Note that if $r_1,r_2$ are complex, they are necessarily complex conjugates of the form $\alpha \pm \beta i$, and you can rewrite the solution as $$y(x) = e^{\alpha x}(A\cos(\beta x) + B \sin (\beta x)).$$
The signs of $b$ and $c$ will determine the signs of $r_1$ and $r_2$, which in turn determine the behaviour of your function as $x \to \infty$.
To seek a non-zero solution, put $y=e^{mx}$ ($m$ is arbitrary constant) in the given ODE
Then the auxiliary equation you get is $e^{mx}(m^2+bm+c)=0$
or $m^2+bm+c=0$ as $e^{mx}\neq 0$.
In case of real roots,what you want is that this quadratic equation must not have a positive real root for which $b>0$ and $c>0$ (by Descarte's rule of signs) is sufficient.
In case no real root exists, even then you need the real part of the root to be negative for which you must have $b>0$.
In both cases, option $3$ is the best choice.