I have struggled with the same thing (and continue to do so). One thing I have found to be very helpful is Ivan Savov's "No bullshit guide to math & physics". I

I have struggled with the same thing (and continue to do so). One thing I have found to be very helpful is Ivan Savov's "No bullshit guide to math & physics". It helps that it's written by a person who does a lot of tutoring, unlike the authors of many frightening ex-cathedra maths books. It basically builds its way up to calculus, in a way that's mostly easy to grok and in a way that's useful and interesting (which is where the physics bit comes in). It certainly hasn't made me a mathematician by any stretch, but it's helped me fill in a lot of gaps left behind by awful maths teachers in school, and it's helped rekindle an interest that I'd long since forgotten.