If $A$ and $B$ are invertible, the following condition is necessary. If $A$ and $B$ are invertible and $A B^{-1}$ has distinct eigenvalues, it is necessary and sufficient:
For all integers $n$, we have $Tr{\Large (}[A,B] (AB^{-1})^k {\Large )} = Tr{\Large (}[A,B] (B^{-1} A)^k {\Large )}. \quad (\ast)$
Proof of necessity: I'll write out the case of $k$ positive; $k$ negative is similar. The left hand side of $(\ast)$ is $Tr((AX-BY) (A B^{-1})^k) = Tr (X (AB^{-1})^k A - Y (AB^{-1})^k B) = Tr (XAB^{-1} A B^{-1} \cdots A - Y A B^{-1} \cdots A)$ where we have used the cyclic symmetry of trace. Similarly, the right hand side of $(\ast)$ is $Tr((XA-YB) (B^{-1} A)^k) = Tr(X A (B^{-1} A)^k - Y B (B^{-1} A)^k) = Tr (XAB^{-1} A B^{-1} \cdots A - Y A B^{-1} \cdots A).$ The two expressions are identical. $\square$
Proof of sufficiency: Let's study the more general question of, given matrices $P$ and $Q$, when there exist $X$ and $Y$ with $AX-BY=P$ and $XA-YB=Q$. Let $V \subseteq \mathrm{Mat}_n^2$ be the set of pairs $(P,Q)$ which can be written in this way.
Clearly, $V$ is a linear space, so it is defined by some linear relations. Every linear function on $\mathrm{Mat}_n^2$ is of the form $(P,Q) \mapsto Tr(PT-QU)$ for some matrices $T$ and $U$. Let's figure out for which $(T,U)$ the function $Tr(PT-QU)$ vanishes on $V$.
This occurs if and only if $Tr((AX-BY) T -(XA-YB)U) = Tr (X (TA-AU)) - Tr (Y (TB-BU))=0.$ for all $X$, $Y$. This occurs if and only if $TA=AU$ and $TB=BU$.
So we deduce the more general, but less explicit, condition:
Your equation is solvable if and only if we have $Tr([A,B] T) = Tr([A,B] U)$ for all $T$ and $U$ such that $TA=AU$ and $TB=BU$.
Now suppose $A$ and $B$ are invertible. Then $T A B^{-1} = A U B^{-1} = A B^{-1} T$. So $T$ commutes with $A B^{-1}$ and $U$ can be computed from $T$ as $U=A^{-1} T A$. With the additional hypothesis that $A B^{-1}$ has distinct eigenvalues, the space of matrices that commute with $A B^{-1}$ is spanned by the powers of $A B^{-1}$. $\square$.
Some comments on dimension: Note that the sequence $(A B^{-1})^k$ satisfies a linear recurrence of length $n$, by the Cayley-Hamilton theorem. So condition $(\ast)$ for any $n$ consecutive values implies $(\ast)$ for all $k$. Also, condition $(\ast)$ is trivial for $k=0$. So condition $(\ast)$ cuts out a space of codimension $\leq n-1$ in the open set of matrices for which $A$ and $B$ are invertible and $A B^{-1}$ has distinct eigenvalues. The space of solutions to your equation thus has dimension $\geq 2 n^2 - (n-1)$.
I didn't check carefully, but I think that the condition $[A,B] \in \mathrm{Span}(A,B)$ has dimension $n^2 + n+1$, so much smaller.
A general comment: The set of $(A,B)$ such that this equation is solvable will be constructable, by Chevalley's theorem. But I see no reason to expect that it is closed. Without a result saying that the image is closed, I don't see a way to equip the image with the structure of an algebraic variety. Without this, questions about singularities and more subtle invariants don't make sense.