2 ms·

It's interesting comment, phamilton. I agree with antirez and I agree with you. full discloser - I am ex-googler. I believe that "horizontal scale" movement st

It's interesting comment, phamilton. I agree with antirez and I agree with you. full discloser - I am ex-googler. I believe that "horizontal scale" movement staryted from Google. You could see it in their GFS and Mapreduce papers from early 2000s. And by that time they were completely right, of course. Geniuses of Jeff Dean and Sanjay Gwattemat put Google year ahead compared to other state of the art for more than a decade. There was not even one system developed in Google that is not horizontally scalable (i am omitting acquisitions here). We used to joke in 2009 that the most expensive server we have is our perforce server. Nowdays it's internally developed source control system that is backed by Bigtable. So of course, antirez is right, of course! If you need infinite scale - you must go horizontally. But the reality is that most companies and most use-cases do not need terrabytes of data. I would say that today the comfort zone for Dragonfly is upto 512GB per instance (1). So dragonfly solves the issue for... I would say 99% percent of the use-cases. Only the last percentile would need horizontal scale, and probably their business is already big enough, so that they can affort a high-quality eng team to work with horizontal clusters. (1) We need to improve some things (mainly around serialization format of rdb) to reach another magnitude of 4TB. Nobody wants to wait for days to load a 4TB snapshot.

phamilton 4y ago
512GB is probably good enough for 99%. 4TB is probably good enough for 99.9%. Context for all this: I lead tech at Remind, where we support 30M MAU with around 1 engineer per million MAU. Years ago we focused heavily on horizontally scaling, including our datastores. We stored a few hundred GB in clustered redis, dozens of TB in dynamoDB, a half dozen postgres clusters, etc. The past year we've reversed course and doubled down on AWS Aurora. Every time we've moved data into Aurora our stability has improved, our devs can move faster and our costs go down. We've got an order of magnitude headroom in Aurora and frankly our code is far from well optimized. There's so much to be gained from simplicity.

phamilton 4y ago

512GB is probably good enough for 99%. 4TB is probably good enough for 99.9%. Context for all this: I lead tech at Remind, where we support 30M MAU with around 1 engineer per million MAU. Years ago we focused heavily on horizontally scaling, including our datastores. We stored a few hundred GB in clustered redis, dozens of TB in dynamoDB, a half dozen postgres clusters, etc. The past year we've reversed course and doubled down on AWS Aurora. Every time we've moved data into Aurora our stability has improved, our devs can move faster and our costs go down. We've got an order of magnitude headroom in Aurora and frankly our code is far from well optimized. There's so much to be gained from simplicity.