Let $H$ be a self-adjoint $n \times n$ matrix with complex entries. $H$ gives rise to a continuous 1-parameter group of unitaries $t \mapsto U_t = \exp(itH) : \mathbb{R} \to U(n)$.
Let $A$ be some other $n \times n$ matrix. If $A$ commutes with $H$, then it commutes with $itH$ for all $t$ and hence with $U_t$ for all $t$ (by considering the series expansion). Thus we have $ U_t A U_t^* = AU_tU_t^* = A \ \ \ \text{for all }t \ \ \ (*)$ I suspect that the converse is true. That is, if $(*)$ holds, then $A$ must commute with $H$, but I'm not sure how to prove this and would appreciate some help.