3 ms·
I don't believe it's religion - it's experience. Google, like Amazon, initially started building scale-out systems by sacrificing strong consistency and transa
by dgacmu 9y ago
I don't believe it's religion - it's experience. Google, like Amazon, initially started building scale-out systems by sacrificing strong consistency and transactions. Amazon did it with Dynamo, and Google did it with BigTable. Over time - and with very substantial engineering investment - Google has started walking that back, first with Megastore, and now with Spanner.
What we're seeing happen is a reinvention of a lot of classical transactional systems from a "massive scale-first" perspective instead of a "local transactions on spinning disk fast" perspective. The eventually and causally-consistent systems have something to add, but I don't think it's wise to discount Google's years of engineering experience in this as religion. Rather, it's reflective of starting to operate at a number of engineers scale at which it has become worthwhile investing very dedicated engineering and research effort into supporting a transactional model, at scale, so that thousands of other engineers can be more productive.
There's a chance this experience doesn't generalize -- recall that Google has massive amounts of dedicated inter-datacenter bandwidth with good management and redundancy, the ability to hire dedicated teams of database & distributed systems experts, etc. But it probably generalizes to the other giant companies in the field -- Microsoft, Amazon, Facebook, and Baidu have similarly huge numbers of "non-distributed-systems-expert developers" who need to get stuff done, correctly, and the ability to invest heavily in the infrastructure to make them more productive.
Another way to summarize it is: At some point, you're going to have to fix problems at the application and algorithm level, and not just hope that the underlying storage system makes everything magic. It's easier when those problems are performance problems than when they're correctness/consistency problems.
(Disclaimer: This is my view as a distributed systems professor, not a one-day-per-week Google employee, but I do have that second hat around.)