3 ms·

This article is very resourceful (I'll be returning to it again and again) but no no no, its opinion is fundamentally flawed. Ignoring the fact they obviously

This article is very resourceful (I'll be returning to it again and again) but no no no, its opinion is fundamentally flawed. Ignoring the fact they obviously have a strongly-consistent product to sell, which therefore informs their view, the flaw is that the article is the cliche "I have a hammer therefore everything is a nail". Honestly, if you go back to the beginning of the article, their first quote sums it up "better to have programmers deal with performance problems ... as bottlenecks arise" admits to the fact that this is the inevitable outcome of strongly consistent systems! If physics is against it, you're asking for trouble. I've given tech talks around the world on this, just don't pick a fight against physics or else YOU WILL LOSE FAITH in any/all database systems you use: http://gun.js.org/distributed/matters.html http://gun.js.org/distributed/matters.html . That is what these strongly consistent database systems run off of, faith (religion) that the database is their god that will solve all their problems. You only need to take a look at http://jepsen.io/ http://jepsen.io/ to be proven that basically every single system out there (except for like RethinkDB and one or two others) fail miserably at being strongly consistent even when they are suppose to be. But hey, they need an excuse to get people to buy into Cloud Spanner - and nothing sounds better than "Hey we're Google, let us manage your data for you, because you aren't smart enough to deal with consistency."

dgacmu 9y ago
I don't believe it's religion - it's experience. Google, like Amazon, initially started building scale-out systems by sacrificing strong consistency and transactions. Amazon did it with Dynamo, and Google did it with BigTable. Over time - and with very substantial engineering investment - Google has started walking that back, first with Megastore, and now with Spanner. What we're seeing happen is a reinvention of a lot of classical transactional systems from a "massive scale-first" perspective instead of a "local transactions on spinning disk fast" perspective. The eventually and causally-consistent systems have something to add, but I don't think it's wise to discount Google's years of engineering experience in this as religion. Rather, it's reflective of starting to operate at a number of engineers scale at which it has become worthwhile investing very dedicated engineering and research effort into supporting a transactional model, at scale, so that thousands of other engineers can be more productive. There's a chance this experience doesn't generalize -- recall that Google has massive amounts of dedicated inter-datacenter bandwidth with good management and redundancy, the ability to hire dedicated teams of database & distributed systems experts, etc. But it probably generalizes to the other giant companies in the field -- Microsoft, Amazon, Facebook, and Baidu have similarly huge numbers of "non-distributed-systems-expert developers" who need to get stuff done, correctly, and the ability to invest heavily in the infrastructure to make them more productive. Another way to summarize it is: At some point, you're going to have to fix problems at the application and algorithm level, and not just hope that the underlying storage system makes everything magic. It's easier when those problems are performance problems than when they're correctness/consistency problems. (Disclaimer: This is my view as a distributed systems professor, not a one-day-per-week Google employee, but I do have that second hat around.)
elvinyung 9y ago
> You only need to take a look at http://jepsen.io/ http://jepsen.io/ to be proven that basically every single system out there (except for like RethinkDB and one or two others) fail miserably at being strongly consistent even when they are suppose to be. What a terrible assertion. Most of the systems that Jepsen tests are AP systems. Basically all of the systems that are supposed to be CP (e.g. etcd [1], Zookeeper [2], CockroachDB [3]) have been shown to be more-or-less fine at what they are supposed to do. [1] https://aphyr.com/posts/316-call-me-maybe-etcd-and-consul https://aphyr.com/posts/316-call-me-maybe-etcd-and-consul [2] https://aphyr.com/posts/291-call-me-maybe-zookeeper https://aphyr.com/posts/291-call-me-maybe-zookeeper [3] http://jepsen.io/analyses/cockroachdb-beta-20160829 http://jepsen.io/analyses/cockroachdb-beta-20160829

dgacmu 9y ago

I don't believe it's religion - it's experience. Google, like Amazon, initially started building scale-out systems by sacrificing strong consistency and transactions. Amazon did it with Dynamo, and Google did it with BigTable. Over time - and with very substantial engineering investment - Google has started walking that back, first with Megastore, and now with Spanner. What we're seeing happen is a reinvention of a lot of classical transactional systems from a "massive scale-first" perspective instead of a "local transactions on spinning disk fast" perspective. The eventually and causally-consistent systems have something to add, but I don't think it's wise to discount Google's years of engineering experience in this as religion. Rather, it's reflective of starting to operate at a number of engineers scale at which it has become worthwhile investing very dedicated engineering and research effort into supporting a transactional model, at scale, so that thousands of other engineers can be more productive. There's a chance this experience doesn't generalize -- recall that Google has massive amounts of dedicated inter-datacenter bandwidth with good management and redundancy, the ability to hire dedicated teams of database & distributed systems experts, etc. But it probably generalizes to the other giant companies in the field -- Microsoft, Amazon, Facebook, and Baidu have similarly huge numbers of "non-distributed-systems-expert developers" who need to get stuff done, correctly, and the ability to invest heavily in the infrastructure to make them more productive. Another way to summarize it is: At some point, you're going to have to fix problems at the application and algorithm level, and not just hope that the underlying storage system makes everything magic. It's easier when those problems are performance problems than when they're correctness/consistency problems. (Disclaimer: This is my view as a distributed systems professor, not a one-day-per-week Google employee, but I do have that second hat around.)

elvinyung 9y ago

> You only need to take a look at http://jepsen.io/ http://jepsen.io/ to be proven that basically every single system out there (except for like RethinkDB and one or two others) fail miserably at being strongly consistent even when they are suppose to be. What a terrible assertion. Most of the systems that Jepsen tests are AP systems. Basically all of the systems that are supposed to be CP (e.g. etcd [1], Zookeeper [2], CockroachDB [3]) have been shown to be more-or-less fine at what they are supposed to do. [1] https://aphyr.com/posts/316-call-me-maybe-etcd-and-consul https://aphyr.com/posts/316-call-me-maybe-etcd-and-consul [2] https://aphyr.com/posts/291-call-me-maybe-zookeeper https://aphyr.com/posts/291-call-me-maybe-zookeeper [3] http://jepsen.io/analyses/cockroachdb-beta-20160829 http://jepsen.io/analyses/cockroachdb-beta-20160829