Could anyone tell me When $Ax=0=Bx$ has a common non trivial solution when $A,B\in\mathbb{R}^{m\times n}$
Suppose $x_1\ne 0$ be that common solution, then what we get is $Ax_1=Bx_1=0$ but I am not getting any relation between them!
are they similar? does there exists non singular matrix $P$ such that $P^{-1}AP=B$?
There is a common non-trivial solution of $Ax=0$ and $Bx=0$
$\iff$ there is a non-trivial solution of $\pmatrix{A \\ B}x=0$
$\iff$ $\pmatrix{A \\ B}$ does not have full rank
This can be decided by row reduction.
They don't have to be similar. Let $A=\pmatrix{1&-1\cr1&-1\cr}$, $B=\pmatrix{1&-1\cr2&-2\cr}$. Then $x=(1,1)$ is in the nullspace of both matrices, but the two matrices are not similar, as they don't have the same eigenvalues.
The solutions of $Ax=0$ and $Bx=0$ are two vector subspaces of $\Bbb R^n$, say $W,U$. Clearly the intersection between them contain the zero vector, but if the intersection isn't trivial, then every $v\neq 0$ such that $v\in W\cap U$ is a common solution, in particular the entire subspace generated by $v$ is contained in the intersection and then you find infinitely many common solutions.
In general this happen also without any relations between $A$ and $B$.