Adam,
It seems to me, from this and many other comments, that you are approaching this (and previous mathematical courses) as subjects to be solely (or mainly) memorized. While it is certainly true that some things need to be memorized (e.g., the multiplication tables, some basic values for sine and cosine, etc), it is completely impractical (not mention nigh impossible) to memorize everything you seem to be trying to memorize. Not only is it very difficult, it is probably why you keep "losing it" after a bit of not using it.
Imagine that you have an exam in which you are required to write out the plot of a novel or movie. One way to do this would certainly be to memorize a description of each chapter (or scene) and then write out these descriptions in the exam. You won't remember what it is about a week later, though. Instead, the way we remember plots of novels and books is understanding the main ideas and mechanics of those plots, and then explaining them in our own words and using our own expressions and ideas.
Mathematics needs to be approached the same way, lest the sheer amount of material overwhelm you (as you are clearly overwhelmed). The key is to recognize and understand the key ideas that are at play, and then use them as organizational principles that allow you to figure out how to do all sorts of related problems. There are lots of things in basic math courses that students think they need to "memorize"; the vast majority of them I've never memorized at all (even when I teach the course)! Instead, I understand the main idea and I figure out the things as I need them on the fly.
In my experience, that is the kind of mindset that makes success in mathematics courses and allows you to retain the important things for later use. Just trying to memorize everything is going to (i) give you a headache; (ii) be extremely hard to do; (iii) make it very difficult to do well; and (iv) make it very easy to forget everything later. There's just no upside to it, even if it is the only approach you have ever used. I know it takes a lot of effort to change approach, but the rewards are worth it; and given how you seem to feel right now, I doubt any change would make you feel worse!
Really, one should be stingy with memorization; as Sherlock Holmes put it, your brain has only a finite amount of capacity, there's no point in filling it with whatever comes your way. Part of the key to success in mathematics is to recognize what needs to be memorized, and what needs to be understood instead. You are trying to approach it by memorizing pretty much everything, and that just doesn't work (as you've discovered).