3 ms·

Of course, in production, it is really hard to find a use case that isn’t served ‘well enough’ by existing database technology. But: almost ever should you say

by xpe 3y ago

Of course, in production, it is really hard to find a use case that isn’t served ‘well enough’ by existing database technology. But: almost ever should you say never. What better way to learn indexing tradeoffs than to write your own database? What existed before Redis? What existed before LevelDB? What existed before Git? Sure, write a database if you want. But be humble about the effort involved and reliability you’ll get.

dekhn 3y ago
BerkeleyDB existed before Redis, LevelDB, and Git and was widely used as a database. SVN originally used BDB as its storage layer (but switched when people complained it was too hard to build BDB). https://en.wikipedia.org/wiki/Berkeley_DB https://en.wikipedia.org/wiki/Berkeley_DB contains people-decades of hard-won experience on how to build a simple database system. I would consider SQLite, itself 20+ years old now, the intellectual heir of BDB. It was as true then as it was today: very few people should write databases for production.
- srhtftw 3y ago
  BerkeleyDB also improved on things like dbm¹, ndbm, qdbm and cdb. > I would consider SQLite, itself 20+ years old now, the intellectual heir of BDB. Indeed. I've worked on projects that only use the btree part of SQLite. If all you need is a btree it's a good place to start. 1- https://en.wikipedia.org/wiki/DBM_(computing) https://en.wikipedia.org/wiki/DBM_(computing)
- xpe 3y ago
  Yes, thanks: the history is fun and instructive. My questions were rhetorical, BTW: emphasizing that progress happens when what exists now doesn't stop someone from making something different and maybe better. We're probably not disagreeing. To summarize: - practically, given real-world constraints and risk tolerances, yes, very few people should write databases for production. (Except villains. They should expend tremendous energy and effort writing bespoke artisanal databases, resulting in less time to be evil.) - educationally, there is a lot to be gained by writing your own, even if only a small part [1], such as something around indexing, a write-ahead-log, a query planner, or even a query parser [2] - if everyone thought e.g. 'writing a new database isn't worth it', collectively we'd be in a worse place FWIW, I have a soft place in my heart for all the great projects that come out of Berkeley. Note 1: Let me propose a game for a group of people. One person names some narrow aspect of a database, perhaps in the hopes that it is boring, trivial, or "solved". (Perhaps "autoincrementing indexes" for example.) Then the other people to talk about all the ways that the thing is actually quite hard, interesting, non-obvious, and maybe (?) even a good research area. Note 2: It is nice that SQL is declarative. AND there is room for improvement, starting with composability. See https://news.ycombinator.com/item?id=24730713 https://news.ycombinator.com/item?id=24730713

by xpe 3y ago

dekhn 3y ago

BerkeleyDB existed before Redis, LevelDB, and Git and was widely used as a database. SVN originally used BDB as its storage layer (but switched when people complained it was too hard to build BDB). https://en.wikipedia.org/wiki/Berkeley_DB https://en.wikipedia.org/wiki/Berkeley_DB contains people-decades of hard-won experience on how to build a simple database system. I would consider SQLite, itself 20+ years old now, the intellectual heir of BDB. It was as true then as it was today: very few people should write databases for production.

srhtftw 3y ago
BerkeleyDB also improved on things like dbm¹, ndbm, qdbm and cdb. > I would consider SQLite, itself 20+ years old now, the intellectual heir of BDB. Indeed. I've worked on projects that only use the btree part of SQLite. If all you need is a btree it's a good place to start. 1- https://en.wikipedia.org/wiki/DBM_(computing) https://en.wikipedia.org/wiki/DBM_(computing)
xpe 3y ago
Yes, thanks: the history is fun and instructive. My questions were rhetorical, BTW: emphasizing that progress happens when what exists now doesn't stop someone from making something different and maybe better. We're probably not disagreeing. To summarize: - practically, given real-world constraints and risk tolerances, yes, very few people should write databases for production. (Except villains. They should expend tremendous energy and effort writing bespoke artisanal databases, resulting in less time to be evil.) - educationally, there is a lot to be gained by writing your own, even if only a small part [1], such as something around indexing, a write-ahead-log, a query planner, or even a query parser [2] - if everyone thought e.g. 'writing a new database isn't worth it', collectively we'd be in a worse place FWIW, I have a soft place in my heart for all the great projects that come out of Berkeley. Note 1: Let me propose a game for a group of people. One person names some narrow aspect of a database, perhaps in the hopes that it is boring, trivial, or "solved". (Perhaps "autoincrementing indexes" for example.) Then the other people to talk about all the ways that the thing is actually quite hard, interesting, non-obvious, and maybe (?) even a good research area. Note 2: It is nice that SQL is declarative. AND there is room for improvement, starting with composability. See https://news.ycombinator.com/item?id=24730713 https://news.ycombinator.com/item?id=24730713