You should DEFINITELY write your own matrix library to learn the underlying principles, but once you want your renderer to actually do something with reasonable

You should DEFINITELY write your own matrix library to learn the underlying principles, but once you want your renderer to actually do something with reasonable speed you'll pretty quickly be replacing your homegrown matrix code with something like Eigen that handles SIMD for you.