6 ms·

Building a new database management system in academia (2017)

83 points by greghn 3y ago

stakhanov 3y ago
This is an announcement from 2017 about "the next five years", which time period is now squarely in the past. Did the DBMS ever come into existence? (If so: link, please). If not: Why should we be interested in this announcement in 2023?
- paddw 3y ago
  This is Andy Pavlo, so he probably got sidetracked with https://ottertune.com/ https://ottertune.com/ Not sure what op's intention with this was
  - apavlo 3y ago
    Actually, it was a combination of three things: 1. OtterTune Start-up (https://ottertune.com https://ottertune.com) 2. Biological Daughter (https://twitter.com/andy_pavlo/status/1187841279260004355 https://twitter.com/andy_pavlo/status/1187841279260004355) 3. Pandemic When the pandemic first started, I had a bunch of CMU students reach out to me saying that their summer internships were rescinded and that they were looking for a project to work on so that they wouldn't have a gap in their CV. I ended up taking on any student that could program C++ even if they hadn't taken my DB class before. It as my way of trying to help. But our research group grew to about 35 people. That was not sustainable and the code quality suffered greatly. We ended killing the project and now all our self-driving work is done in the context of Postgres (https://db.cs.cmu.edu/papers/2023/p27-lim.pdf https://db.cs.cmu.edu/papers/2023/p27-lim.pdf). I also now realize that building the DBMS engine first then building the query optimizer second is the wrong order. Our future project is going to start with the optimizer first.
    - stakhanov 3y ago
      Awesome, thanks for the quick response.
    - pcthrowaway 3y ago
      Oh hey, please fix the link to macrobase; it redirects to pornhub
      - brazzledazzle 3y ago
        Oh wow
    - erichocean 3y ago
      > Our future project is going to start with the optimizer first. What's your opinion of recent attempts like LingoDB, that move the query optimizer into a traditional compiler stack, in this case, MLIR?
      - apavlo 3y ago
        LingoDB is an interesting system. Jana has done great work with it. I like projects that take unorthodox approaches to old problems. The problem with (most) query optimizers is that they take a one shot approach at optimization. I think an optimizer should be built from the groundup to support adaptive query optimization. Something similar to Berkeley's Eddies project from 20 years ago.
        zinclozenge 3y ago
        Do you know if there is anybody taking this approach? Alternatively, what would you consider to be the current state of the art when it comes to query optimizers?
        erichocean 3y ago
        One way to do that might be to merge the query optimizer and executor together and then execute them simultaneously in a self-adjusting code framework.[0] The relevant optimizer variables would then be set initially (using whatever mechanism already exists e.g. heuristics, stats, etc.) and then while the query executes, you continually update those optimizer variables. The self-adjusting code property would cause the query to self-adjust as it ran, while still producing the same end result. I'm sure there are details I'm missing here, but I do believe the general approach could do implemented in LingoDB (or similar) as a compiler transformation, so the actually cost-to-develop this approach would remain tractable. [0] I suspect you'd need to model the whole thing as a streaming network so that you can update the network parts as you go, effectively re-wiring the streams while not invalidating earlier results. So SAC+logic to map from one stream architecture to another. JITs that support de-optimization have to do something similar (with a lot of careful upfront design), so that's at least plausible.
      - zinclozenge 3y ago
    - cmrdporcupine 3y ago
      I'm curious, wondering if you could explicate why you feel starting from the query optimization end is key? I have my (amateur) guesses, but would love to hear your expert opinion.
    - zX41ZdbW 3y ago
      I'm doing a similar thing - invite every student who is interested, without interviewing or skill tests: https://github.com/ClickHouse/ClickHouse/issues/42194 https://github.com/ClickHouse/ClickHouse/issues/42194 It works if you target for ~10% outcome if you have a good CI system with a decent test coverage and a ton of fuzzing.
    - zX41ZdbW 3y ago
      Moreover, making as many as possible people to learn database engineering and production C++ experience - is one of my goals with ClickHouse.
    - lifepillar 3y ago
      I see that MVCC is still your preferred way of doing CC, and what academic research is mostly focused. I am wondering whether that’s an advantage for in-memory databases specifically. I was once discussing MVCC vs 2PL with an experienced Sybase and SQL Server guy, and he claimed that, when transactions are implemented properly and the database is well-designed (no surrogate keys, in particular), 2PL leads to better performance and no deadlocks, while “readers do not block writers” leads to lots of aborted transactions in a heavy OLTP workload. I verified that (I should still have the code around): lots of conflicts in PostgreSQL vs smooth concurrent execution with no retries in Sybase and SQL Server. I have since heard similar opinions from other SQL Server practitioners: they disable MVCC and rely only on good ol’ 2PL.
      - apavlo 3y ago
        See our 2014 paper on evaluating CC protocols on in-memory system with high contention / core counts: https://www.vldb.org/pvldb/vol8/p209-yu.pdf https://www.vldb.org/pvldb/vol8/p209-yu.pdf All the protocols regress to the same. This evaluation was only with stored procedures though. It would be worth doing a similar investigation with conversational DB protocols (e.g., JDBC, ODBC).
        sundar28 3y ago
        Thanks for the reminder.. this was a great paper, and I'm curious - why would conversational protocols indicate any difference? Also - curious if any such tests have used libraries such as seastar?
    - iamcreasy 3y ago
      Can you kindly share any reading list or anything similar for someone who wants to get into database research at doctoral level? I am a recent MS(Statistics + CS) graduate working as a Data Engineering, and I am looking for material to learn about the current research landscape and getting ready for grad school application.
- whoevercares 3y ago
  DeepDive from he listed become SnorkelAI, which become hot lately
pcthrowaway 3y ago
Uh... the link to macrobase redirects to pornhub. Not a good look for people browsing at work
- apavlo 3y ago
  Yikes! Thanks for the heads up. Peter left Stanford so I guess they took over the domain name :-(
  - pcthrowaway 3y ago
    Excellent, well done on the quick turnaround. I hope no one had to have a hard conversation with their boss just now. (for the curious, the link was previously *NSFW* http://macrobase http://macrobase dot io *NSFW*)
    - pbailis 3y ago
      Macrobase PI here – someone squatted on that io domain a long time ago while the project was active. Once we moved to macrobase.stanford.edu, a fan apparently took interest in our old domain. Thanks Andy for updating the link.
      - pcthrowaway 3y ago
        "A fan" hehe. I wonder if some genius at pornhub marketing thought the word had enough innuendo to be worth paying the squatter for redirect rights. It's not exactly "Macrohard"[ware] or its inverse (who we can thank for Github and VS Code)
whoevercares 3y ago
FWIW, many of his recent student went to Databricks now
eatonphil 3y ago
If you're interested in the idea of databases built from scratch since the time this post was written in 2017 (based on GitHub contributions info), here are a few: - Materialize: 2017 - DuckDB: 2018 - RedPanda: 2019 - TigerBeetle: 2020
- jchrisa 3y ago
  I am building a new immutable cryptographically verified database using IPLD data structures and prolly trees. This allows changes made anywhere to be transparently synced, and for operations to be commuted amongst untrusted peers, for instance allowing for shared index maintenance. https://use-fireproof.com/docs/architecture https://use-fireproof.com/docs/architecture It's also the easiest way to write React apps. Here are some ChatGPT expert builders that I've trained to use the CSS framework of your choice with Fireproof: https://use-fireproof.com/docs/chatgpt-quick-start/#react-expert-builders https://use-fireproof.com/docs/chatgpt-quick-start/#react-ex...
- zX41ZdbW 3y ago
  According to my estimation, a new database engine is born every week - mostly key-value and document databases. Only a small subset of them survive after one year. According to a guess by Stonebreaker, a DBMS takes around 7 years to become mature enough for general applications.
  - eatonphil 3y ago
    > According to my estimation, a new database engine is born every week Fair. I'm talking about databases with funding backing them (either by universities or otherwise).
- tlarkworthy 3y ago
  and DuckDB came out of academia too and is not based on Postgres either (highly relevant and notably absent in the authors list of academic DBs at the end of the article) https://duckdb.org/pdf/SIGMOD2019-demo-duckdb.pdf https://duckdb.org/pdf/SIGMOD2019-demo-duckdb.pdf EDIT: oh the article is old
esjeon 3y ago
For unsuspecting readers: this article talks about the feasibility of building a new practical DBMS. Database is the most critical piece of software for businesses, so it has already been thoroughly explored and researched. It's very difficult to find a better solution for existing problems. One should either invent a new paradigm or tackle unsolved problems to justify the cost of development. Technology-wise, writing a toy DBMS is nothing difficult. Even undergraduates can do it.
onetimeuse92304 3y ago
It may seem daunting, but I think many people make it more complex / difficult than it needs to be. I have rolled out two transactional databases of my own. In both cases I had to provide very specific properties and for some reason I could not find an existing product that would meet all requirements. For example, one of them was an embedded device that was very restricted in memory, all operations needed to run with hard bounds on time and memory and the storage for the data was a flash chip without wear levelling which required the database itself to manage writes to prolong the chip's life. The key is to notice how your database system is going to be different from others and what properties are not essential. Also, making general purpose DBMS tends to be much more complex vs making more niche solutions where you know a bit more about what the uses are going to be and what kinds of loads you can expect. Creating a custom engine for a given application can be very simple task because you can easily cross out requirements you don't care about and you only care that it works well for the loads that this particular application can generate. Also, it is unlikely you are going to beat fierce competition in general purpose "and a kitchen sink" database management system market, but much easier to find a niche that is underserved and create a usable, competitive product with relatively little effort. That's how SQLite started.
dang 3y ago
Discussed at the time: Building a Database System in Academia - https://news.ycombinator.com/item?id=13931752 https://news.ycombinator.com/item?id=13931752 - March 2017 (15 comments)
yingjunwu 3y ago
Proud to see my name (https://twitter.com/YingjunWu https://twitter.com/YingjunWu) mentioned in Andy's blog. I was Andy's visiting PhD at CMU and was the top 1 contributor to Peloton (https://github.com/cmu-db/peloton https://github.com/cmu-db/peloton). Today, building a database from scratch is extremely difficult, for several reasons: 1. it anyways takes a long time; 2. there are so many successful (open-source) databases; 3. hiring top engineers are so expensive. 4. you won't get enough attention unless your system is drastically better than existing ones. An interesting observation is that very few database was built since 2020 - almost all the newly built databases were developed on top of existing databases (PostgreSQL, ClickHouse, etc). I started building RisingWave (https://github.com/risingwavelabs/risingwave https://github.com/risingwavelabs/risingwave) in early 2021. The only reason we built the system from scratch was that none of the existing systems can address the problem we are solving - distributed SQL stream processing at cloud scale. We tried Flink but gave up, as it's too heavy and it's architecture was not designed for the cloud environment. If you want to build a database from scratch, or are simply interested in databases, we may talk.
- iudqnolq 3y ago
  You're obviously the expert here, but I was surprised that you found it notable very few databases have been released in the last three years. That seems like a very short timeframe. Per Wikipedia ClickHouse started as an experimental project in 2009 and was first released in 2016.

83 points by greghn 3y ago

This is an announcement from 2017 about "the next five years", which time period is now squarely in the past. Did the DBMS ever come into existence? (If so: link, please). If not: Why should we be interested in this announcement in 2023?

paddw 3y ago
This is Andy Pavlo, so he probably got sidetracked with https://ottertune.com/ https://ottertune.com/ Not sure what op's intention with this was
- apavlo 3y ago
  Actually, it was a combination of three things: 1. OtterTune Start-up (https://ottertune.com https://ottertune.com) 2. Biological Daughter (https://twitter.com/andy_pavlo/status/1187841279260004355 https://twitter.com/andy_pavlo/status/1187841279260004355) 3. Pandemic When the pandemic first started, I had a bunch of CMU students reach out to me saying that their summer internships were rescinded and that they were looking for a project to work on so that they wouldn't have a gap in their CV. I ended up taking on any student that could program C++ even if they hadn't taken my DB class before. It as my way of trying to help. But our research group grew to about 35 people. That was not sustainable and the code quality suffered greatly. We ended killing the project and now all our self-driving work is done in the context of Postgres (https://db.cs.cmu.edu/papers/2023/p27-lim.pdf https://db.cs.cmu.edu/papers/2023/p27-lim.pdf). I also now realize that building the DBMS engine first then building the query optimizer second is the wrong order. Our future project is going to start with the optimizer first.
  - stakhanov 3y ago
    Awesome, thanks for the quick response.
  - pcthrowaway 3y ago
    Oh hey, please fix the link to macrobase; it redirects to pornhub
    - brazzledazzle 3y ago
      Oh wow
  - erichocean 3y ago
    > Our future project is going to start with the optimizer first. What's your opinion of recent attempts like LingoDB, that move the query optimizer into a traditional compiler stack, in this case, MLIR?
    - apavlo 3y ago
      LingoDB is an interesting system. Jana has done great work with it. I like projects that take unorthodox approaches to old problems. The problem with (most) query optimizers is that they take a one shot approach at optimization. I think an optimizer should be built from the groundup to support adaptive query optimization. Something similar to Berkeley's Eddies project from 20 years ago.
      - zinclozenge 3y ago
        Do you know if there is anybody taking this approach? Alternatively, what would you consider to be the current state of the art when it comes to query optimizers?
      - erichocean 3y ago
        One way to do that might be to merge the query optimizer and executor together and then execute them simultaneously in a self-adjusting code framework.[0] The relevant optimizer variables would then be set initially (using whatever mechanism already exists e.g. heuristics, stats, etc.) and then while the query executes, you continually update those optimizer variables. The self-adjusting code property would cause the query to self-adjust as it ran, while still producing the same end result. I'm sure there are details I'm missing here, but I do believe the general approach could do implemented in LingoDB (or similar) as a compiler transformation, so the actually cost-to-develop this approach would remain tractable. [0] I suspect you'd need to model the whole thing as a streaming network so that you can update the network parts as you go, effectively re-wiring the streams while not invalidating earlier results. So SAC+logic to map from one stream architecture to another. JITs that support de-optimization have to do something similar (with a lot of careful upfront design), so that's at least plausible.
    - zinclozenge 3y ago
  - cmrdporcupine 3y ago
    I'm curious, wondering if you could explicate why you feel starting from the query optimization end is key? I have my (amateur) guesses, but would love to hear your expert opinion.
  - zX41ZdbW 3y ago
    I'm doing a similar thing - invite every student who is interested, without interviewing or skill tests: https://github.com/ClickHouse/ClickHouse/issues/42194 https://github.com/ClickHouse/ClickHouse/issues/42194 It works if you target for ~10% outcome if you have a good CI system with a decent test coverage and a ton of fuzzing.
  - zX41ZdbW 3y ago
    Moreover, making as many as possible people to learn database engineering and production C++ experience - is one of my goals with ClickHouse.
  - lifepillar 3y ago
    I see that MVCC is still your preferred way of doing CC, and what academic research is mostly focused. I am wondering whether that’s an advantage for in-memory databases specifically. I was once discussing MVCC vs 2PL with an experienced Sybase and SQL Server guy, and he claimed that, when transactions are implemented properly and the database is well-designed (no surrogate keys, in particular), 2PL leads to better performance and no deadlocks, while “readers do not block writers” leads to lots of aborted transactions in a heavy OLTP workload. I verified that (I should still have the code around): lots of conflicts in PostgreSQL vs smooth concurrent execution with no retries in Sybase and SQL Server. I have since heard similar opinions from other SQL Server practitioners: they disable MVCC and rely only on good ol’ 2PL.
    - apavlo 3y ago
      See our 2014 paper on evaluating CC protocols on in-memory system with high contention / core counts: https://www.vldb.org/pvldb/vol8/p209-yu.pdf https://www.vldb.org/pvldb/vol8/p209-yu.pdf All the protocols regress to the same. This evaluation was only with stored procedures though. It would be worth doing a similar investigation with conversational DB protocols (e.g., JDBC, ODBC).
      - sundar28 3y ago
        Thanks for the reminder.. this was a great paper, and I'm curious - why would conversational protocols indicate any difference? Also - curious if any such tests have used libraries such as seastar?
  - iamcreasy 3y ago
    Can you kindly share any reading list or anything similar for someone who wants to get into database research at doctoral level? I am a recent MS(Statistics + CS) graduate working as a Data Engineering, and I am looking for material to learn about the current research landscape and getting ready for grad school application.
whoevercares 3y ago
DeepDive from he listed become SnorkelAI, which become hot lately

pcthrowaway 3y ago

Uh... the link to macrobase redirects to pornhub. Not a good look for people browsing at work

apavlo 3y ago
Yikes! Thanks for the heads up. Peter left Stanford so I guess they took over the domain name :-(
- pcthrowaway 3y ago
  Excellent, well done on the quick turnaround. I hope no one had to have a hard conversation with their boss just now. (for the curious, the link was previously *NSFW* http://macrobase http://macrobase dot io *NSFW*)
  - pbailis 3y ago
    Macrobase PI here – someone squatted on that io domain a long time ago while the project was active. Once we moved to macrobase.stanford.edu, a fan apparently took interest in our old domain. Thanks Andy for updating the link.
    - pcthrowaway 3y ago
      "A fan" hehe. I wonder if some genius at pornhub marketing thought the word had enough innuendo to be worth paying the squatter for redirect rights. It's not exactly "Macrohard"[ware] or its inverse (who we can thank for Github and VS Code)

whoevercares 3y ago

FWIW, many of his recent student went to Databricks now

eatonphil 3y ago

If you're interested in the idea of databases built from scratch since the time this post was written in 2017 (based on GitHub contributions info), here are a few: - Materialize: 2017 - DuckDB: 2018 - RedPanda: 2019 - TigerBeetle: 2020

jchrisa 3y ago
I am building a new immutable cryptographically verified database using IPLD data structures and prolly trees. This allows changes made anywhere to be transparently synced, and for operations to be commuted amongst untrusted peers, for instance allowing for shared index maintenance. https://use-fireproof.com/docs/architecture https://use-fireproof.com/docs/architecture It's also the easiest way to write React apps. Here are some ChatGPT expert builders that I've trained to use the CSS framework of your choice with Fireproof: https://use-fireproof.com/docs/chatgpt-quick-start/#react-expert-builders https://use-fireproof.com/docs/chatgpt-quick-start/#react-ex...
zX41ZdbW 3y ago
According to my estimation, a new database engine is born every week - mostly key-value and document databases. Only a small subset of them survive after one year. According to a guess by Stonebreaker, a DBMS takes around 7 years to become mature enough for general applications.
- eatonphil 3y ago
  > According to my estimation, a new database engine is born every week Fair. I'm talking about databases with funding backing them (either by universities or otherwise).
tlarkworthy 3y ago
and DuckDB came out of academia too and is not based on Postgres either (highly relevant and notably absent in the authors list of academic DBs at the end of the article) https://duckdb.org/pdf/SIGMOD2019-demo-duckdb.pdf https://duckdb.org/pdf/SIGMOD2019-demo-duckdb.pdf EDIT: oh the article is old

esjeon 3y ago

For unsuspecting readers: this article talks about the feasibility of building a new practical DBMS. Database is the most critical piece of software for businesses, so it has already been thoroughly explored and researched. It's very difficult to find a better solution for existing problems. One should either invent a new paradigm or tackle unsolved problems to justify the cost of development. Technology-wise, writing a toy DBMS is nothing difficult. Even undergraduates can do it.

onetimeuse92304 3y ago

It may seem daunting, but I think many people make it more complex / difficult than it needs to be. I have rolled out two transactional databases of my own. In both cases I had to provide very specific properties and for some reason I could not find an existing product that would meet all requirements. For example, one of them was an embedded device that was very restricted in memory, all operations needed to run with hard bounds on time and memory and the storage for the data was a flash chip without wear levelling which required the database itself to manage writes to prolong the chip's life. The key is to notice how your database system is going to be different from others and what properties are not essential. Also, making general purpose DBMS tends to be much more complex vs making more niche solutions where you know a bit more about what the uses are going to be and what kinds of loads you can expect. Creating a custom engine for a given application can be very simple task because you can easily cross out requirements you don't care about and you only care that it works well for the loads that this particular application can generate. Also, it is unlikely you are going to beat fierce competition in general purpose "and a kitchen sink" database management system market, but much easier to find a niche that is underserved and create a usable, competitive product with relatively little effort. That's how SQLite started.

dang 3y ago

Discussed at the time: Building a Database System in Academia - https://news.ycombinator.com/item?id=13931752 https://news.ycombinator.com/item?id=13931752 - March 2017 (15 comments)

yingjunwu 3y ago

Proud to see my name (https://twitter.com/YingjunWu https://twitter.com/YingjunWu) mentioned in Andy's blog. I was Andy's visiting PhD at CMU and was the top 1 contributor to Peloton (https://github.com/cmu-db/peloton https://github.com/cmu-db/peloton). Today, building a database from scratch is extremely difficult, for several reasons: 1. it anyways takes a long time; 2. there are so many successful (open-source) databases; 3. hiring top engineers are so expensive. 4. you won't get enough attention unless your system is drastically better than existing ones. An interesting observation is that very few database was built since 2020 - almost all the newly built databases were developed on top of existing databases (PostgreSQL, ClickHouse, etc). I started building RisingWave (https://github.com/risingwavelabs/risingwave https://github.com/risingwavelabs/risingwave) in early 2021. The only reason we built the system from scratch was that none of the existing systems can address the problem we are solving - distributed SQL stream processing at cloud scale. We tried Flink but gave up, as it's too heavy and it's architecture was not designed for the cloud environment. If you want to build a database from scratch, or are simply interested in databases, we may talk.

iudqnolq 3y ago
You're obviously the expert here, but I was surprised that you found it notable very few databases have been released in the last three years. That seems like a very short timeframe. Per Wikipedia ClickHouse started as an experimental project in 2009 and was first released in 2016.