I went back to school with only a slightly higher math background than yours. By the time I was ready to go back, my already neglected math skills had atrophied

I went back to school with only a slightly higher math background than yours. By the time I was ready to go back, my already neglected math skills had atrophied, but it wasn't like I never attended algebra class in high school. For me, I read a GRE prep book (it had concise reminders for basic things I should have known, and probably did at one point, but potentially forgot.) Following that I worked a college textbook for Calculus and enrolled for Calculus II at a local community college. I took all the math courses there and I'm now at a four year considering a math double major. Go figure, I thought I hated math. I still hit embarrassing gaps from time to time, but I'm at the point that filling them in isn't such an ordeal. It is strange though to be in a 400 level class and to be the only guy that gets caught up on some basic high school thing. At this point though all that stuff seems simple once looked at. Good luck with the mathematics. It's worth it.