6 ms·
Fallacies of Distributed Computing
- mindcrime 16y agoAs popular as "cloud" solutions are becoming, I think it's more important than ever to consider these fallacies, especially fallacy number 1: The network is reliable.
- ry0ohki 16y agoOr the even older "Don't put all of your eggs in one basket"
- liuliu 16y agoCan you make the "network is homogeneous" assumption given that all servers are at the same datacenter? It seems to me that more or less is true if you make the latency resolution at millisecond level.
- jerf 16y agoIf A, B, C, and D are all on gigabit on the same switch, if A and B are using their full gigabit to talk to each other, C could get significantly different bandwidths if it tries to talk to A vs. D. Is that homogeneous? Depends on how you define it but "no" is probably the more useful answer. (There's a couple of other possible outcomes, like the switch prioritizes C, but then the A<->B link sees variable bandwidth over time, another type of inhomogeneity. One way or another something is going to give.)
- tybris 16y agoNope. Even if it was, consider how much traffic the instances around yours are producing and how that is affecting you.
- pquerna 16y agoThere are more general issues at AWS this morning, but the continuing issues all center around EBS. I think this is just reinforcing to me to never use a networked filesystem. It is the wrong abstraction for storing data across multiple physical units. Too often you end up hanging the whole box waiting for an IO operation to finish. NFS fell out of favor for most Linux things a long time ago for these reasons. IBM's GPFS... I've seen it fail spectacularly. But their marketing materials said it is awesome! And now Amazon's EBS, essentially a networked block store, is failing. Don't trust a vendor about networked storage. Its a lie, just like the cake.
- cageface 16y agoNFS on a well designed and maintained LAN is ok, but on any network less reliable I agree it's the wrong abstraction. You can only get away with hiding the network in those rare situations where the network can be made that robust.
- pquerna 16y agoThat is the point though, the #1 Fallacies of Distributed Computing: The network is reliable. There is no such thing as a well designed and maintained LAN. All networks are unreliable.
- cageface 16y agoI worked for the last ten years at a major animation studio where we pushed huge I/O over NFS every day. This was a lot easier to do with NFS semantics than it would have been with a less "fallacious" model, but we only got away with it because we had complete control over the whole network.
- oinksoft 16y agoFrom somebody who only tends to do single-system administration...what do you use in place of NFS? Local storage and sync it? Something else?
- cagenut 16y agoEBS isn't a shared filesystem, its a block device mountable by only one host (instance) at a time. Much more analogous to fibrechannel or iscsi than to nfs.
- ssmoot 16y agoThat's an odd thing to say. Internal networks can easily be more reliable than applications given that higher end equipment has a lot more money poured into it at a development level, and anything above the bottom end of the market typically has multiple layers of redundancy options available. Wether it's VRRP, stacking, bonding, IPMP, iSCSI channels etc. EBS is unreliable, granted, but blaming the network seems faulty. For one, it's nothing like NFS. For two, NFS can be adapted to highly-available systems fairly easily. These days it's a pretty simple system. Even with just a single Array, put two FreeBSD "heads" on it, MPIO connectivity, and CARP between the boxes. Monitor the CARP interface to import/export the file-systems and there you go. If you can tolerate a minute or two of downtime when a host fails you have reliable, automatic redundancy. EBS, FCoE, iSCSI, et all are entirely different beasts. But again, you can make the hosts redundant pretty easily. With iSCSI you can give the clients multiple paths to the devices increasing reliability and available bandwidth without LACP. Consolidate storage needs to save money. It's win/win/win. Networked storage, wether it's block devices or shared file-systems, whatever the technology, aren't going away anytime soon. Matter of fact, they're becoming more critical to operations day by day. And these days, they're one of the easiest things to setup, and ensure availability on. That EBS is the weak-link at Amazon is a bit counter-intuitive based on my own experience. I wonder what the root problems actually are...
- champion 16y agoWhy do people think being in the "cloud" for hosting changes any of these rules? EC2 is a rentable compute utility, it doesn't provide for automatic failure or redundancy. Also people keep talking about an EC2 outage but it is really 1 region and from the status updates, maybe only a single Availability Zone that is affected. That said, maybe I'm just less upset because my site is in a US-East Zone that doesn't appear to be affected.
- jbigelow 16y ago"EC2 is a rentable compute utility, it doesn't provide for automatic failure or redundancy." Nobody wants to admit that "cloud" became a great way to market (and mark up) traditional web hosting services that we've had since the early 90s. It still boggles my mind that Microsoft's Azure, which overall I'm a fan of, makes me pick an instance size (and corresponding price) for my app; the original premise of the cloud was to take these kinds decisions out of the mix. What happens when my micro-instance pegs out? It won't automatically bump up to the next higher level or spin up a new instance for me. This isn't "cloud" this is marketing.
- tybris 16y agoThe difference is in the word "utility". In the 90s, hosting was a fixed quantity for a fixed price. In the 00/10s, hosting is whatever you need whenever you need it and pay for whatever you use. It's like going from chopped wood to an electric grid, a radical transformation of technology and business models.
- mtogo 16y agoA radical transformation, really? To me it seems more like a small step, just changing the payment model a little bit and making it a bit more convenient to provision more instances.
- singlow 16y agoActually, EC2 and AWS in general does give a lot of options for automatic failure and redundancy. I have a highly available application running in EC2 and I have multiple instances ready for failover and scripts triggered by monitors to move into another zone if the whole zone goes bonkers. This would be much more expensive and complicated if I needed to have dedicated redundant hardware in two geographically separated datacenters.
- jzila 16y agoI'm not sure whether the OP is implying that Amazon is guilty of these fallacies or whether AWS's users are. Having worked at Amazon, I can attest that Amazon does take these issues into account, and many more. That said, the purpose of the services that failed isn't to guard against network partitioning at the data center cluster level. The services that were designed with cluster network partitions in mind are still up (S3, EBS, SQS, etc). If you are renting instances in a specific data center, and you wish to have cross-cluster redundancy, it is your responsibility to build that into your distributed system. This is ultimately a question of whether you need to commit to an SLA higher than that of the service on which you depend. (It's more complicated than that, but that should suffice for simplicity's sake). There are companies that built services on top of EC2 to mitigate these risks (e.g. Elastra), and there are competing solutions that provide application-level hosting that should guard against such partitions (e.g. Windows Azure).
- mindcrime 16y agoI'm not sure whether the OP is implying that Amazon is guilty of these fallacies or whether AWS's users are. I had AWS customers in mind, more than Amazon.com themselves. If you are renting instances in a specific data center, and you wish to have cross-cluster redundancy, it is your responsibility to build that into your distributed system. Exactly. It's easy, though, to build something on top of AWS (or Rackspace or whatever) and then stand back, marvel at it's magnificence; and forget that an outage - in a network segment or service that you don't control - can totally wreck your masterpiece.
- jzila 16y agoFair enough. As I mentioned in my earlier post, there are available solutions that mitigate the risks of these fallacies for AWS's customers. When you build on top of a networked service, you are taking a hard dependency on its SLA, and if you need something with greater reliability, it is your responsibility to provide the necessary redundancy.
- fraserharris 16y agoOpen question: does Google App Engine have automatic redundancy across multiple datacenters?
- tybris 16y agoProbably, but it's not like they're immune to failure. http://www.readwriteweb.com/cloud/2010/02/google-app-engine-is-down---ba.php http://www.readwriteweb.com/cloud/2010/02/google-app-engine-... http://techcrunch.com/2009/07/02/google-app-engine-broken-for-4-hours-and-counting/ http://techcrunch.com/2009/07/02/google-app-engine-broken-fo...
- brown9-2 16y agoI don't believe that AppEngine offers the choice of what region/zone your app runs in, so they would have to.
- alecco 16y agoThis compounds with other fallacies in computing. Like picking designs that behave very well on average but show very nasty behavior once in a while. In particular, beware of people talking about big O notation without mentioning worst case behavior.
- pjscott 16y agoWait, what? Big O notation is all about asymptotic worst-case behavior. Anybody using it in another way is abusing the notation, although it may sometimes be useful to talk that way. For example, a naive quicksort takes O(n^2) time. On random data, its expected runtime is O(n lg n). With ideal partitioning, it can be brought down to guaranteed O(n lg n) on arbitrary data, though this will usually make it slower in practice. But someone who says that quicksort is "usually O(n lg n)" is misusing the notation, and hopefully will quickly point this out by mentioning worst-case time. Is this roughly what you had in mind?
- billswift 16y agoBig O can be used to represent worst-case behavior, but it can be, and is used also for average-case. And no, it is not a misuse of the notation - it is legitimately used for both (also for best-case, but I have never heard of anybody actually bothering with that). Just be sure to understand which is being talked about.
- pjscott 16y agoOkay, this is a subtle point, but you're right. Big-O notation describes an asymptotic upper bound on any function. If that function is average run time, then that's perfectly kosher. Actually establishing what this means, though, is a bit trickier, and something people often mess up.
- arethuza 16y agoI don't think it's a subtle point at all - Big-O is not inherently anything to do with worst case time complexity of algorithms, it is a general relationship between functions.
- jsdalton 16y agoWe use AWS for the following reasons: * Flexible pricing, with fair costs * Instant, flexible provisioning * Awesome APIs and third party libraries (e.g. boto) * Extensive documentation/support in the wild (i.e. you can get just about any question answered via Google or Server Fault) * Solid uptime track record with SLA to back it up Did I expect that all server outages would magically disappear because we were now in the "cloud"? Not in the least bit. Did others sign up for AWS thinking that? Sounds incredibly foolish to me if they did. But this sounds suspiciously like a straw man to me...
- mindcrime 16y agoDid others sign up for AWS thinking that? Sounds incredibly foolish to me if they did. But this sounds suspiciously like a straw man to me... There's no straw-man, because nothing is being debated. That said, I do have a hunch that some non-trivial number of Amazon AWS customers do sign-up, build significant parts of their business on AWS, and not fully consider the implications of this configuration in terms of reliability, uptime, etc. Building in the kind of redundancy and failover mechanisms to handle outages like this isn't easy, but it is important; and awareness of the kinds of things pointed out by the "Eight Fallacies" is part of that.
- albedoa 16y agoThat it is not being debated is the reason it's a strawman. The parent is suggesting that the list of fallacies is a response to an argument that nobody is making. I'm not agreeing or disagreeing, but that seems to be what he meant. If I came up to you and said, "You know, it's a fallacy for you to have assumed that your car wouldn't require any maintenance," you'd probably respond with something like, "Uh, yeah."
- mindcrime 16y agoThat it is not being debated is the reason it's a strawman. That's not how I've understood the use of the term "strawman," FWIW. I've always felt that a strawman can only exist inside the context of an argument... that is, that a strawman is a technique used in rhetoric/debate, where you setup a false argument for your opponent and then tear it down. That seems to jibe with what Wikipedia says: "A straw man is a component of an argument and is an informal fallacy based on misrepresentation of an opponent's position.[1] To "attack a straw man" is to create the illusion of having refuted a proposition by substituting it with a superficially similar yet unequivalent proposition (the "straw man"), and refuting it, without ever having actually refuted the original position." In this case, there was no debate, no opponent, etc., ergo, no strawman. Maybe there's another usage of the term, that I'm not acquainted with?
- MatthewB 16y agoI think the way the "cloud" was explained for years before it became mainstream was misleading. People would explain that if there were any failures within the cloud, data would magically shift to another part of the cloud with zero downtime. Obviously this isn't true, even Google has suffered outages in their cloud. That's not to say the solution is to use managed data centers, but we will get better and better at improving the reliability of the cloud. The cloud is not fail-proof.