If you want the multiplication to be well-defined, then you must have A ∩ B = 1. If g in A ∩ B, how do you know if g = g⋅1 or g = 1⋅g. In particular, you need some very serious compatibility conditions for A ∩ B and φ.
If you want to say that G, rather than just AB, is a semi-direct product, then obviously you need G = AB.
If you want AB to be a subgroup of G, then you must have A is normalized by B, since clearly A is normal in the group structure you define on AB. Assuming you want G to be a semi-direct product, then that means A is normal in G.
As a specific example, consider G to be the symmetric group on 3 points, and let A be generated by (1,2) and B be generated by (2,3). Let φ be the unique function (which happens to be a homomorphism) from B to Aut(A). Then AB = { (), (1,2), (2,3), (1,3,2) } has order 4, and the group law you define is well-defined since A ∩ B = 1, but it gives the group a very weird multiplication: (1,2)⋅(2,3) = (2,3)⋅(1,2) in the new AB. The subgroup generated by A and B is actually G itself, of order 6. So we've somehow defined a group of order 4 as a subset of a group of order 6, and Lagrange tells us we do not have a subgroup.
To see the problems with overlap, consider two Sylow 2-subgroups of the symmetric group of order 4. Letting φ be an isomorphism (spooky choice, eh), I think you'll find the multiplication is not well-defined on the set AB.