Your $6^4$ is correct. However, $6^2$ for the third end-swapping case is wrong: it really is $6^3$, as you originally had it. If the rotation is $\pi/2$ clockwise, the ends swap, the top and right side swap, and the bottom and left side swap, so you get to choose three colors, not two.
Here’s an elementary enumeration along the lines suggested by André Nicolas. Set the prism on end. Suppose first that the top and bottom faces get the same color; we’ll count the number of distinguishable ways to color the four sides.
Using just one color: $6$ ways.
Using two colors, $3$ sides of one color and $1$ of the other: There are $6$ ways to choose the color for the single face and then $5$ ways to choose the other color, for a total of $30$ ways. (Running total: $36$)
Using two colors, $2$ sides of each color: There are ${6 \choose 2} = 15$ ways to choose the two colors. Once the colors are chosen, they can be applied in just $2$ distinguishable ways: either opposite sides are the same color, or they are different colors. The total number of ways is therefore again $30$. (Running total: $66$)
Using three colors: In this case one color must appear twice, the other two once each. There are $6$ ways to choose the color that appears twice, and there are then ${5 \choose 2} = 10$ ways to choose the other two colors. The two faces of the same color may be either adjacent or opposite. In either case it makes no difference how the remaining two colors are applied to the other two sides, since we can swap ends, so we get a total of $120$ ways in this case, $60$ from each subcase. (Running total: $186$)
Using four colors: There are ${6\choose 4} = 15$ ways to choose the colors. Arbitrarily choose one of the colors to paint one side. There are $3$ ways to color the opposite side, and since we can swap ends, it makes no difference which way we apply the remaining two colors to the other two sides. This case yields a total of $15\cdot 3 = 45$ ways, for a grand total of $231$.
For each of these $231$ colorings of the four sides, there are $6$ ways to color the top and bottom faces, for a total of $1386$ colorings in which the top and bottom faces get the same color.
The analysis when the top and bottom faces get different colors is fairly similar. This time, however, it’s easier to account for the coloring of the top and bottom faces as we go along, noting that there are ${6\choose 2} = 15$ pairs of colors for the top and bottom faces.
Using just one color for the four sides: no change, so $6$ ways to color the sides and $6\cdot 15 = 90$ colorings altogether. (Swapping ends doesn’t affect the sides.)
Using two colors: no change, so $60$ ways and $60\cdot 15 = 900$ colorings; swapping ends affects the sides, but they can be restored by a rotation. (Running total: $990$)
Using three colors: As before, there are $6$ ways to choose the color that appears twice and then $10$ ways to choose the other two colors, and the two faces of the same color may be either adjacent or opposite. If they are opposite, it makes no difference how the remaining two colors are applied to the other two sides, so that sub-case yields $60$ ways and $60\cdot 15 = 900$ colorings. If they are adjacent, however, the order in which the other two colors are applied to the other two sides does matter, and we get $120$ ways. Since these $120$ ways of coloring the sides are not preserved by swapping ends, each apparently extends to $30$ different ways of coloring the prism, for a total of $120\cdot 30 = 3600$ colorings. However, this actually counts each coloring twice. Say that the two singleton side colors are $a$ and $b$, and that the top and bottom colors are $c$ and $d$. The coloring in which $b$ immediately follows $a$ in clockwise order and $c$ is on top is the same as the coloring in which $a$ immediately follows $b$ and $d$ is on top. Each of these colorings is therefore counted twice in the $3600$, once with the side colors in the order $ab$ and once with them in the order $ba$. This subcase therefore produces only $1800$ colorings. (Running total: $3690$)
Using four colors: There are still $15$ ways to choose the colors. As before, arbitrarily choose one of the colors to paint one side. There are still $3$ ways to color the opposite side, but we can no longer swap ends, so the order in which we apply the remaining two colors to the other two sides now matters. This case yields a total of $15\cdot 6 = 90$ ways. A little thought shows that it’s like the second three-color subcase, so it gives rise to $90\cdot 15 = 1350$ colorings and a grand total of $5040$ colorings with distinct top and bottom colors.
Thus, there are altogether $1386+5040 = 6426$ distinguishable colorings of the prism. As a check, the Polya Enumeration Theorem yields the same result.