I have this set of equations and I am trying to find $X(t)$ and $Y(t)$ analytically with initial values known such as $X(0)=X_0$ and $Y(0)=Y_0$. How should I approach to solve it?
$dX/dt=a \times X(t) + b \times Y(t)$
$dY/dt=c \times Y(t) + d \times X(t)$
Write your system of equations in matrix form:
$$\begin{pmatrix} \frac{dX}{dt} \\ \frac{dY}{dt} \end{pmatrix} =\begin{pmatrix} a & b \\ d & c \end{pmatrix} \begin{pmatrix} X \\ Y \end{pmatrix}$$
You can find the two eigenvalues $\lambda_1$ and $\lambda_2$ by letting $\det(A-\lambda I)=0$, and then evaluate corresponding eigenvectors $\vec{v_1}$ and $\vec{v_2}$ of the matrix $A=\begin{pmatrix} a & b \\ d & c \end{pmatrix}$ to give you the general solution to $X(t)$ and $Y(t)$.
Depending on your values of your eigenvalues $\lambda_1$ and $\lambda_2$ (which ultimately depend on the values of your constants $a,b,c,d$), there will be different general solutions.
There are 3 different solution cases:
Case 1: Real eigenvalues: $\lambda \in \mathbb{R}$
The solution will be of the form:
$$\begin{pmatrix} X(t) \\ Y(t) \end{pmatrix} =k_1 \vec{v_1} e^{\lambda_1 t}+k_2 \vec{v_2} e^{\lambda_2 t}$$
Case 2: Complex eigenvalues.
We denote a complex number by $\lambda=a+bi$.
Since $\lambda_{1,2}$ will both be complex, using $e^{i \theta}=\cos{\theta}+i\sin{\theta}$, we obtain the general solution:
$$\begin{pmatrix} X(t) \\ Y(t) \end{pmatrix}=k_1 \vec{v_1} e^{at} \cos{(bt)} + k_2 \vec{v_2} e^{at} \sin{(bt)}$$
Case 3: Real, repeated roots (i.e. $\lambda_1=\lambda_2$)
$$\begin{pmatrix} X(t) \\ Y(t) \end{pmatrix}=k_1 \vec{v_1} e^{\lambda t}+k_2 \vec{v_2} \cdot t e^{\lambda t}$$
Your initial values can be used to evaluate the constants $k_1$ and $k_2$.
Here is an Youtube video showing an example solve for Case 1.
Using brute force.
The system being $$X'=aX+bY \tag 1$$ $$Y'=cY+dX \tag 2$$ Form $(1)$ extract $Y$ $$Y=\frac{X'-aX}b\tag 3$$ Differentiate $$ Y'=\frac{X''-aX'}b\tag 4$$ Plug in $(2)$ $$\frac{X''-aX'}b=c\frac{X'-aX}b+dX \tag 5$$ $$X''-(a+c)X'+(ac-bd)X=0\tag 6$$ Solve $(6)$ for $X$ and go back to $(2)$ to get $Y$.
Rewriting the above as:
$$\begin{bmatrix} X'\\Y'\end{bmatrix}=\begin{bmatrix}a & b \\ d & c\end{bmatrix}\begin{bmatrix}X\\Y\end{bmatrix}$$
Let's assume for the moment that the determinant of the above matrix is nonzero. Then the matrix has two eigenvalues, $r_1$ and $r_2$, with two corresponding eigenvectors, $\vec{v_1}=(v_{1x},v_{1y})$ and $\vec{v_2}=(v_{2x},v_{2y})$. Then the solutions are given by:
$X=C_1v_{1x}e^{r_1t}+C_2v_{2x}e^{r_2t}$
$Y=C_1v_{1y}e^{r_1t}+C_2v_{2y}e^{r_2t}$
where the constants $C_1$ and $C_2$ are determined by initial conditions.
If the determinant of the above square matrix is zero, obtain the single eigenvalue $r$ and the eigenvector $\vec{v}=(v_x,v_y)$, along with the generalized eigenvector $\vec{w}=(w_x,w_y)$. Then the solutions are given by:
$X=C_1v_{x}e^{r_1t}+C_2w_{x}te^{r_2t}$
$Y=C_1v_{y}e^{r_1t}+C_2w_{y}te^{r_2t}$