4 ms·
Proud to see my name (https://twitter.com/YingjunWu https://twitter.com/YingjunWu) mentioned in Andy's blog. I was Andy's visiting PhD at CMU and was the top 1
by yingjunwu 3y ago
Proud to see my name (https://twitter.com/YingjunWu https://twitter.com/YingjunWu) mentioned in Andy's blog. I was Andy's visiting PhD at CMU and was the top 1 contributor to Peloton (https://github.com/cmu-db/peloton https://github.com/cmu-db/peloton).
Today, building a database from scratch is extremely difficult, for several reasons:
1. it anyways takes a long time;
2. there are so many successful (open-source) databases;
3. hiring top engineers are so expensive.
4. you won't get enough attention unless your system is drastically better than existing ones.
An interesting observation is that very few database was built since 2020 - almost all the newly built databases were developed on top of existing databases (PostgreSQL, ClickHouse, etc).
I started building RisingWave (https://github.com/risingwavelabs/risingwave https://github.com/risingwavelabs/risingwave) in early 2021. The only reason we built the system from scratch was that none of the existing systems can address the problem we are solving - distributed SQL stream processing at cloud scale. We tried Flink but gave up, as it's too heavy and it's architecture was not designed for the cloud environment.
If you want to build a database from scratch, or are simply interested in databases, we may talk.
- iudqnolq 3y agoYou're obviously the expert here, but I was surprised that you found it notable very few databases have been released in the last three years. That seems like a very short timeframe. Per Wikipedia ClickHouse started as an experimental project in 2009 and was first released in 2016.