We built a DBMS from scratch! Ok, it started before I joined, but the data at our startup was all basically in text files, not unlike a csv. There were data co

We built a DBMS from scratch! Ok, it started before I joined, but the data at our startup was all basically in text files, not unlike a csv. There were data coming from different sources and needed to be merged with existing. We also needed to do some arbitrage between data from multiple sources, so we wrote an always on server which will do all this in memory and dump the data to the files periodically. Unfortunately, the server can sometimes crash and data yet to be dumped was lost. So, next there was a checkpointing system, which will help recover data in case there was a crash. Slowly it grew into a full fledged data management behemoth. To be fair, this episode happened more than a decade back. The whole team were freshers straight out of college :) And that system had one of the most readable code, I have ever seen. It finally took a very senior tech person to come poking and find the inefficiencies. He convinced us of the goodness of MySQL and the whole workflow was then rebuilt around it. This exercise made us appreciate all the more what a DBMS does and wouldn't miss it for the world.