It's almost impossible having a decent understanding of the theoretical side of ML without having a decent understanding of linear algebra, multivariable calcul

It's almost impossible having a decent understanding of the theoretical side of ML without having a decent understanding of linear algebra, multivariable calculus and probability. So you need to start with those if you haven't the background and most ML courses (including Ng's) assume you know some of that. Going through the undergraduate material for these three topics is probably sufficient.