5 ms·

They're using litestream [1] to replicate a SQLite database, which streams additions to the SQLite WAL file to object storage and can replay it back. This is fa

They're using litestream [1] to replicate a SQLite database, which streams additions to the SQLite WAL file to object storage and can replay it back. This is fairly hands-off from SQLite's perspective. There's also the SQLite Session Extension [2] that is a built-in way to support generating and applying "patches". I'm curious how these tools will mature, it seems like a good match for microservices. [1]: https://github.com/benbjohnson/litestream https://github.com/benbjohnson/litestream [2]: https://www.sqlite.org/sessionintro.html https://www.sqlite.org/sessionintro.html

Xeoncross 5y ago
The litestream project was created by https://github.com/benbjohnson https://github.com/benbjohnson who wrote https://github.com/boltdb/bolt https://github.com/boltdb/bolt (a key value store) which has been instrumental (from my point of view) in the Go community as one of the original choices for an embedded database as it was not only fast, but had transactions with stable snapshots. It was used by https://github.com/blevesearch/bleve https://github.com/blevesearch/bleve, https://github.com/etcd-io/etcd https://github.com/etcd-io/etcd, and number of other projects. These days, https://github.com/dgraph-io/badger https://github.com/dgraph-io/badger is often favored because of it's improved write throughput.
- otterley 5y ago
  It should be noted that BoltDB has been retired as it suffered from a number of congenital defects. It was implicated in the major Roblox outage that happened a couple months ago, and Consul doesn't use it anymore. https://news.ycombinator.com/item?id=30015913 https://news.ycombinator.com/item?id=30015913 (This comment is not to suggest that the Litestream project is of poor quality or anything like that.)
  - benbjohnson 5y ago
    BoltDB author here. Just to clarify, the project was retired because of maintenance burden. CoreOS wanted to make some changes but I didn't have the bandwidth to test and maintain it all. Since this was before Go had good version management, we decided that they could fork the project as "bbolt" and users could move over as needed. They did a good job maintaining it so I eventually archived the original project.
- infogulch 5y ago
  Another one of Ben's projects is on the front page right now: Postgres wire compatible SQLite proxy - https://news.ycombinator.com/item?id=30875837 https://news.ycombinator.com/item?id=30875837
- benbjohnson 5y ago
  Bolt author here. Badger is a good database but it's mostly a tradeoff of write versus read throughput, specifically, range queries. Badger is an LSM so it doesn't perform as well (last time I checked) with iterating over ordered key/value pairs. LSMs have bloom filters to speed up point queries so those aren't as much of an issue.
- tapirl 5y ago
  It is strange that, although https://github.com/syndtr/goleveldb https://github.com/syndtr/goleveldb is used in the go-ethereum project for years, it is mentioned rarely (comparing to other go dbs).
  - Xeoncross 5y ago
    Yes, it has it's own trade-offs, but is certainly a top-performer for certain workloads: https://github.com/smallnest/kvbench https://github.com/smallnest/kvbench
silisili 5y ago
I watched an interview of Mr. Hipp, creator of SQLite, that I can't find now but was pretty interesting. Aside from being way different than I expected for some reason, in a good way - very down to earth and friendly, he was asked specifically about that, and more or less answered that his job was to write a solid DB, and replication can be done elsewhere. That's a pretty honest answer, and looks like someone took up the challenge.
- qbasic_forever 5y ago
  He was on an episode of the Changelog podcast in the fall and went into similar topics: https://changelog.com/news/RRAw/visit https://changelog.com/news/RRAw/visit It's a good listen.
- billywhizz 5y ago
  probably this one? https://corecursive.com/066-sqlite-with-richard-hipp/ https://corecursive.com/066-sqlite-with-richard-hipp/

Xeoncross 5y ago

The litestream project was created by https://github.com/benbjohnson https://github.com/benbjohnson who wrote https://github.com/boltdb/bolt https://github.com/boltdb/bolt (a key value store) which has been instrumental (from my point of view) in the Go community as one of the original choices for an embedded database as it was not only fast, but had transactions with stable snapshots. It was used by https://github.com/blevesearch/bleve https://github.com/blevesearch/bleve, https://github.com/etcd-io/etcd https://github.com/etcd-io/etcd, and number of other projects. These days, https://github.com/dgraph-io/badger https://github.com/dgraph-io/badger is often favored because of it's improved write throughput.

otterley 5y ago
It should be noted that BoltDB has been retired as it suffered from a number of congenital defects. It was implicated in the major Roblox outage that happened a couple months ago, and Consul doesn't use it anymore. https://news.ycombinator.com/item?id=30015913 https://news.ycombinator.com/item?id=30015913 (This comment is not to suggest that the Litestream project is of poor quality or anything like that.)
- benbjohnson 5y ago
  BoltDB author here. Just to clarify, the project was retired because of maintenance burden. CoreOS wanted to make some changes but I didn't have the bandwidth to test and maintain it all. Since this was before Go had good version management, we decided that they could fork the project as "bbolt" and users could move over as needed. They did a good job maintaining it so I eventually archived the original project.
infogulch 5y ago
Another one of Ben's projects is on the front page right now: Postgres wire compatible SQLite proxy - https://news.ycombinator.com/item?id=30875837 https://news.ycombinator.com/item?id=30875837
benbjohnson 5y ago
Bolt author here. Badger is a good database but it's mostly a tradeoff of write versus read throughput, specifically, range queries. Badger is an LSM so it doesn't perform as well (last time I checked) with iterating over ordered key/value pairs. LSMs have bloom filters to speed up point queries so those aren't as much of an issue.
tapirl 5y ago
It is strange that, although https://github.com/syndtr/goleveldb https://github.com/syndtr/goleveldb is used in the go-ethereum project for years, it is mentioned rarely (comparing to other go dbs).
- Xeoncross 5y ago
  Yes, it has it's own trade-offs, but is certainly a top-performer for certain workloads: https://github.com/smallnest/kvbench https://github.com/smallnest/kvbench

silisili 5y ago

I watched an interview of Mr. Hipp, creator of SQLite, that I can't find now but was pretty interesting. Aside from being way different than I expected for some reason, in a good way - very down to earth and friendly, he was asked specifically about that, and more or less answered that his job was to write a solid DB, and replication can be done elsewhere. That's a pretty honest answer, and looks like someone took up the challenge.

qbasic_forever 5y ago
He was on an episode of the Changelog podcast in the fall and went into similar topics: https://changelog.com/news/RRAw/visit https://changelog.com/news/RRAw/visit It's a good listen.
billywhizz 5y ago
probably this one? https://corecursive.com/066-sqlite-with-richard-hipp/ https://corecursive.com/066-sqlite-with-richard-hipp/