For standard uses of RSA, the secret key is hard to find, and the public key is trivial. This is because most implementations use 65537 as the public key (the reason has to do with its primality and the efficiency of repeated squaring for $e=65537=2^{16}+1$).
Many systems will rechoose $p$ and $q$ if $\phi(pq)$ is not coprime to 65537, meaning that the public exponent is never anything else.
Several other answers note that $e$ and $d$ are interchangeable, but this ignores both practice and several important cryptanalytic results. In particular, even if we allow $e$ to vary, it can still be found given $d$ if it's sufficiently small. This is due to the fact that Coppersmith's algorithm can find small roots of polynomials modulo a composite of unknown factorization using LLL and some added cleverness. The bound keeps increasing, but (from memory) I believe if $e < n^{.33}$ it can be found from $n$ and $d$ (here $n=pq$). So saying $e$ and $d$ are equivalent is false (and dangerous). There is a conjecture that the "true" bound is $n^{.5}$, meaning that for security the secret exponent would have to be above $\sqrt{n}$.
Note: For those truly interested, Boneh has a nice survey of these results (and many others). And it's quite readable by non-cryptographers who have a moderately strong math background.