4 ms·

> But I'm just confused, why would an engineer want to join a company that is making these decisions? Why would the company want their users to know about these

> But I'm just confused, why would an engineer want to join a company that is making these decisions? Why would the company want their users to know about these decisions? This might actually be a really good filter for which types of engineers are good fit for their company. Because my read was very different than yours, as I looked at the decision lineage and thought to myself I could see myself making every one of those decisions. I of course can't say I would've with the same problems and constraints they had, just that I know I've considered similar solutions. The big difference might be, on the third iteration I tend to land on embedding raft directly into the backend, because lots of the problems I look at tend to benefit from a in-order log that is fully replicated to a quorum of servers. But based on the post and efficiency they're going for backing up the WAL on sqlite may make alot of sense. Actually litestream looks like it might have some fairly similar properties, so might be something else I'd like to add to my toolbox. Although I wonder what sort of replication lag there might be to replicas when using s3 as the stable store.

lmeyerov 5y ago
There's a world of a difference of someone doing that for a Google scale thing with big teams supporting it, or say a core part of making the best wireguard product that only they can do, vs a system that could have been postgres and now when the authors leave for their own sqlite-by-mail startup or are partying at SREConf for the 'our n+1th attempt to not do postgres' and an outage causes an unnecessary and ungoogleable shitshow for their colleagues. A good engineer can build all sorts of surprising things. A senior one knows why they should reserve that for only the special bits. I like reading the first two types of articles -- we frequently give tech talks ourselves -- but for making boring systems weird, I get flashbacks of consultantware/resumeware/NIH that took ages to weed out. I try to give talks on say GPUs for weirdly shaped problems, not on why we celebrated when we finally had time to rip out mongo from where it didn't belong :)
billywhizz 5y ago
the live replication feature litestream is implementing won't be using s3 - it will be direct node to node replication. hasn't been released yet and currently s3 is used for a cold backup that can be quickly restored to a fresh node as far as i know. https://github.com/benbjohnson/litestream/issues/8 https://github.com/benbjohnson/litestream/issues/8
benbjohnson 5y ago
Litestream author here. The default configuration is to replicate any new changes every second to S3. This is mainly a cost issue though. S3 doesn't charge for ingress bandwidth but it does charge $0.000005 per PUT request. If you have a non-stop, continuous stream of writes against your database then that's about $12/mo. In practice, most people don't have a continuous stream of writes. Michael Lynch, for example, wrote a post about how he pays $0.03/month for his Litestream replication[1]. I've worked with Raft a lot in my career. I wrote the original etcd Raft implementation (sorry!) [2] and even wrote a visualization explaining Raft[3]. It's a great tool when you need really high durability guarantees but I'm not convinced it gives you better availability guarantees than a single node that can quickly be recovered. [1]: https://mtlynch.io/litestream/ https://mtlynch.io/litestream/ [2]: https://github.com/goraft/raft https://github.com/goraft/raft [3]: http://thesecretlivesofdata.com/raft/ http://thesecretlivesofdata.com/raft/

lmeyerov 5y ago

There's a world of a difference of someone doing that for a Google scale thing with big teams supporting it, or say a core part of making the best wireguard product that only they can do, vs a system that could have been postgres and now when the authors leave for their own sqlite-by-mail startup or are partying at SREConf for the 'our n+1th attempt to not do postgres' and an outage causes an unnecessary and ungoogleable shitshow for their colleagues. A good engineer can build all sorts of surprising things. A senior one knows why they should reserve that for only the special bits. I like reading the first two types of articles -- we frequently give tech talks ourselves -- but for making boring systems weird, I get flashbacks of consultantware/resumeware/NIH that took ages to weed out. I try to give talks on say GPUs for weirdly shaped problems, not on why we celebrated when we finally had time to rip out mongo from where it didn't belong :)

billywhizz 5y ago

the live replication feature litestream is implementing won't be using s3 - it will be direct node to node replication. hasn't been released yet and currently s3 is used for a cold backup that can be quickly restored to a fresh node as far as i know. https://github.com/benbjohnson/litestream/issues/8 https://github.com/benbjohnson/litestream/issues/8

benbjohnson 5y ago

Litestream author here. The default configuration is to replicate any new changes every second to S3. This is mainly a cost issue though. S3 doesn't charge for ingress bandwidth but it does charge $0.000005 per PUT request. If you have a non-stop, continuous stream of writes against your database then that's about $12/mo. In practice, most people don't have a continuous stream of writes. Michael Lynch, for example, wrote a post about how he pays $0.03/month for his Litestream replication[1]. I've worked with Raft a lot in my career. I wrote the original etcd Raft implementation (sorry!) [2] and even wrote a visualization explaining Raft[3]. It's a great tool when you need really high durability guarantees but I'm not convinced it gives you better availability guarantees than a single node that can quickly be recovered. [1]: https://mtlynch.io/litestream/ https://mtlynch.io/litestream/ [2]: https://github.com/goraft/raft https://github.com/goraft/raft [3]: http://thesecretlivesofdata.com/raft/ http://thesecretlivesofdata.com/raft/