4 ms·

That's an odd thing to say. Internal networks can easily be more reliable than applications given that higher end equipment has a lot more money poured into it

by ssmoot 16y ago

That's an odd thing to say. Internal networks can easily be more reliable than applications given that higher end equipment has a lot more money poured into it at a development level, and anything above the bottom end of the market typically has multiple layers of redundancy options available. Wether it's VRRP, stacking, bonding, IPMP, iSCSI channels etc. EBS is unreliable, granted, but blaming the network seems faulty. For one, it's nothing like NFS. For two, NFS can be adapted to highly-available systems fairly easily. These days it's a pretty simple system. Even with just a single Array, put two FreeBSD "heads" on it, MPIO connectivity, and CARP between the boxes. Monitor the CARP interface to import/export the file-systems and there you go. If you can tolerate a minute or two of downtime when a host fails you have reliable, automatic redundancy. EBS, FCoE, iSCSI, et all are entirely different beasts. But again, you can make the hosts redundant pretty easily. With iSCSI you can give the clients multiple paths to the devices increasing reliability and available bandwidth without LACP. Consolidate storage needs to save money. It's win/win/win. Networked storage, wether it's block devices or shared file-systems, whatever the technology, aren't going away anytime soon. Matter of fact, they're becoming more critical to operations day by day. And these days, they're one of the easiest things to setup, and ensure availability on. That EBS is the weak-link at Amazon is a bit counter-intuitive based on my own experience. I wonder what the root problems actually are...

jerf 16y ago
Let me summarize the argument against using network filesystems. Elsewhere on the HN homepage as I write this [1] there is a link to the Wikipedia article on the Fallacies of Distributed Computing [2], which are: 1. The network is reliable. 2. Latency is zero. 3. Bandwidth is infinite. 4. The network is secure. 5. Topology doesn't change. 6. There is one administrator. 7. Transport cost is zero. 8. The network is homogeneous. Technically, every one of them is a myth of local disk storage as well, inasmuch as nothing is every 100% reliable in the mathematical sense. In reality, unlike network programming where the moment you step out into the big world you are slammed in the face by the mythical nature of these claims, they are true enough for when you are working with local disk that you can write significant programs as if they are true, excepting perhaps latency (busy disk may mean you get scheduled much later). The basic argument is the practical differences are so significant that where the file interface is most effective when it just pretends most of the myths are true because adding the complexity of being able to deal with violations isn't worth it (and most programmers using local disk would get it wrong even if you offered it; how many programs gracefully handle running out of disk space, let alone more bizarre failure cases?), the network must give you the tools to handle those issues, and correspondingly must have a more complicated API. In particular, every time you so much as touch the network, you need a code path to handle the possibility that the network communication utterly failed, just disappeared into oblivion. You just don't code that way when you deal with files; the file may be gone, the disk may run out of space, but you don't sit there in the middle of a fast read loop and handle failure on every read, such as a topology change, i.e., "the disk drive is no longer present". That's not because it's not possible, because you can certainly yank out a USB key mid-read, but you tend not to handle it in most userland code. In fact the OS won't even let you handle many failures, as it will either abstract away the retries or simply lock your userspace hard. So the fundamental idea is that filesystem abstractions built for relatively reliable communication simply isn't correct for network communication. Incidentally, the same argument can pretty much be made against remote RPC by simply replacing "disk access" with "function call". You do not have to guard every function call you make with a check for absolute failure. (Not the function returning failure or the function throwing an exception, but the function that may have been there two minutes ago unexpectedly being entirely gone.) RPC may look cute in the source code but it's the wrong layer of abstraction. [1]: http://news.ycombinator.com/item?id=2470865 http://news.ycombinator.com/item?id=2470865 [2]: http://en.wikipedia.org/wiki/Fallacies_of_Distributed_Computing http://en.wikipedia.org/wiki/Fallacies_of_Distributed_Comput...
- ssmoot 16y ago
  That's actually this post (the fallacies link). :-) And you make a good argument about handling read errors. In practice that's not what high availability is about though, and it's certainly not what we're talking about with AWS, iSCSI or even an HA-NFS solution. Availability in those terms is properly designed, redundant infrastructure. There are many cases I'm sure (say a Stock-Exchange) where HA-NFS would not be appropriate. In your average web-app however, request failures while transitioning to failover are (in my experience) just part of the bargain. In that context, waiting 60 seconds for a stale NFS handle to pick a new connection up on the VRRPed failover node is acceptable, and like you said, you make assumptions and don't worry about the edge-case here. There are clearly situations where NFS wouldn't work well, but those are definitely the exception to the rule. There are not a lot of reasons/scenarios where iSCSI couldn't or shouldn't replace local storage for virtualization needs however. In my opinion. iSCSI is simple and extremely reliable. Perhaps even more reliable than local disks, especially if the primary concern is availability. Frankly, I don't know that these rules are really something that applies at the level of a polished iSCSI or NFS stack. They certainly apply to the internals of those systems. While you or I aren't going to worry about wether a buffered-file-reader can fetch the next 8K block, that thought and engineering effort definitely has been put into these protocols and implementations at a lower level than you or I will typically work with.

by ssmoot 16y ago

jerf 16y ago

Let me summarize the argument against using network filesystems. Elsewhere on the HN homepage as I write this [1] there is a link to the Wikipedia article on the Fallacies of Distributed Computing [2], which are: 1. The network is reliable. 2. Latency is zero. 3. Bandwidth is infinite. 4. The network is secure. 5. Topology doesn't change. 6. There is one administrator. 7. Transport cost is zero. 8. The network is homogeneous. Technically, every one of them is a myth of local disk storage as well, inasmuch as nothing is every 100% reliable in the mathematical sense. In reality, unlike network programming where the moment you step out into the big world you are slammed in the face by the mythical nature of these claims, they are true enough for when you are working with local disk that you can write significant programs as if they are true, excepting perhaps latency (busy disk may mean you get scheduled much later). The basic argument is the practical differences are so significant that where the file interface is most effective when it just pretends most of the myths are true because adding the complexity of being able to deal with violations isn't worth it (and most programmers using local disk would get it wrong even if you offered it; how many programs gracefully handle running out of disk space, let alone more bizarre failure cases?), the network must give you the tools to handle those issues, and correspondingly must have a more complicated API. In particular, every time you so much as touch the network, you need a code path to handle the possibility that the network communication utterly failed, just disappeared into oblivion. You just don't code that way when you deal with files; the file may be gone, the disk may run out of space, but you don't sit there in the middle of a fast read loop and handle failure on every read, such as a topology change, i.e., "the disk drive is no longer present". That's not because it's not possible, because you can certainly yank out a USB key mid-read, but you tend not to handle it in most userland code. In fact the OS won't even let you handle many failures, as it will either abstract away the retries or simply lock your userspace hard. So the fundamental idea is that filesystem abstractions built for relatively reliable communication simply isn't correct for network communication. Incidentally, the same argument can pretty much be made against remote RPC by simply replacing "disk access" with "function call". You do not have to guard every function call you make with a check for absolute failure. (Not the function returning failure or the function throwing an exception, but the function that may have been there two minutes ago unexpectedly being entirely gone.) RPC may look cute in the source code but it's the wrong layer of abstraction. [1]: http://news.ycombinator.com/item?id=2470865 http://news.ycombinator.com/item?id=2470865 [2]: http://en.wikipedia.org/wiki/Fallacies_of_Distributed_Computing http://en.wikipedia.org/wiki/Fallacies_of_Distributed_Comput...

ssmoot 16y ago
That's actually this post (the fallacies link). :-) And you make a good argument about handling read errors. In practice that's not what high availability is about though, and it's certainly not what we're talking about with AWS, iSCSI or even an HA-NFS solution. Availability in those terms is properly designed, redundant infrastructure. There are many cases I'm sure (say a Stock-Exchange) where HA-NFS would not be appropriate. In your average web-app however, request failures while transitioning to failover are (in my experience) just part of the bargain. In that context, waiting 60 seconds for a stale NFS handle to pick a new connection up on the VRRPed failover node is acceptable, and like you said, you make assumptions and don't worry about the edge-case here. There are clearly situations where NFS wouldn't work well, but those are definitely the exception to the rule. There are not a lot of reasons/scenarios where iSCSI couldn't or shouldn't replace local storage for virtualization needs however. In my opinion. iSCSI is simple and extremely reliable. Perhaps even more reliable than local disks, especially if the primary concern is availability. Frankly, I don't know that these rules are really something that applies at the level of a polished iSCSI or NFS stack. They certainly apply to the internals of those systems. While you or I aren't going to worry about wether a buffered-file-reader can fetch the next 8K block, that thought and engineering effort definitely has been put into these protocols and implementations at a lower level than you or I will typically work with.