I've done the same, I'm a roboticist by training, but I've successfully self studied abstract algebra, topology, and complex analysis. It certainly helped that

I've done the same, I'm a roboticist by training, but I've successfully self studied abstract algebra, topology, and complex analysis. It certainly helped that I took proof based real analysis and linear algebra in school. Writing proofs is really hard to learn without a feedback loop of a professor. So if you don't have any experience with that, I'd start with an intro to proofs book first. but the group theory material is generally easier than something like topology or functional analysis, which I am certain I was only able to do from finding like minded people in the mathematics sub reddit and did basically a mock semester class on discord. specific to algebra, I really like the classic Pinter book, "A book of Abstract Algebra", which does a great job of mixing in exposition and history that was really beneficial for self study.