4 ms·

>For an efficient enough company, a project like this needs just one or two dedicated engineers. Maybe for a research project or a hobby project, but not for a

>For an efficient enough company, a project like this needs just one or two dedicated engineers. Maybe for a research project or a hobby project, but not for a real, high performance database to be used in a business-critical application. FTA: "Databases are a nightmare to write, from Atomicity, Consistency, Isolation, and Durability (ACID) requirements to sharding to fault recovery to administration - everything is hard beyond belief." >Engineers should be their own PMs. For small projects, sure (your "one or two dedicated engineers"). But once you start tackling projects that require larger teams, or even teams of teams, you need someone to track and prioritize the work remaining and the work in progress (as well as the corresponding budgets for personnel, services, and other resources). Similar to the way a sole proprietor can do their own accounting, but a multi-million dollar business probably should have an accountant. As an aside, I wonder if this might be a use case for a bitmap db engine like Featurebase (https://www.featurebase.com/ https://www.featurebase.com/).

delusional 3y ago
> what we’ve built is just a cursor streaming a binary file feed with a very limited set of functionality - but then again, it’s the exact functionality we need and we didn’t lose any features. The trick is that they didn't need a database that provides "Atomicity, Consistency, Isolation, and Durability (ACID)". By only implementing what they need they were able to keep the project small. It's like people are scared of doing anything without making it into some huge multi hundred developer effort. They've written a super simple append only document store. It's not rocket science. It's not a general purpose arbitrary SQL database.
- vannevar 3y ago
  If it's really true that they only needed a "cursor streaming binary file feed with a very limited set of functionality", then why claim in the headline they wrote their own database and begin the article with the observation that doing ACID databases is really hard? Maybe they really built only what they needed. If so, good on them, but ditch the clickbait title. On the other hand, a lot of people believe that they've built only what they need, only to discover in a week or a month or a year that they underestimated their own requirements. Then they gradually re-learn all the lessons that led to the development of real databases, to their eventual sorrow.
  - politician 3y ago
    OTOH, the Bitcask paper can be implemented in about a week for a write-ahead log that’ll give you pretty strong guarantees.
- mbrumlow 3y ago
  This. Everything we are teaching people to do is centered around solving the problems with already built services, preferably from one of the big 3 cloud vendors. We are churning out “qualified” engineers who don’t know how anything works and we pay them to configure a few AWS services.

delusional 3y ago

> what we’ve built is just a cursor streaming a binary file feed with a very limited set of functionality - but then again, it’s the exact functionality we need and we didn’t lose any features. The trick is that they didn't need a database that provides "Atomicity, Consistency, Isolation, and Durability (ACID)". By only implementing what they need they were able to keep the project small. It's like people are scared of doing anything without making it into some huge multi hundred developer effort. They've written a super simple append only document store. It's not rocket science. It's not a general purpose arbitrary SQL database.

vannevar 3y ago
If it's really true that they only needed a "cursor streaming binary file feed with a very limited set of functionality", then why claim in the headline they wrote their own database and begin the article with the observation that doing ACID databases is really hard? Maybe they really built only what they needed. If so, good on them, but ditch the clickbait title. On the other hand, a lot of people believe that they've built only what they need, only to discover in a week or a month or a year that they underestimated their own requirements. Then they gradually re-learn all the lessons that led to the development of real databases, to their eventual sorrow.
- politician 3y ago
  OTOH, the Bitcask paper can be implemented in about a week for a write-ahead log that’ll give you pretty strong guarantees.
mbrumlow 3y ago
This. Everything we are teaching people to do is centered around solving the problems with already built services, preferably from one of the big 3 cloud vendors. We are churning out “qualified” engineers who don’t know how anything works and we pay them to configure a few AWS services.