4 ms·

There are more general issues at AWS this morning, but the continuing issues all center around EBS. I think this is just reinforcing to me to never use a netwo

by pquerna 15y ago

There are more general issues at AWS this morning, but the continuing issues all center around EBS. I think this is just reinforcing to me to never use a networked filesystem. It is the wrong abstraction for storing data across multiple physical units. Too often you end up hanging the whole box waiting for an IO operation to finish. NFS fell out of favor for most Linux things a long time ago for these reasons. IBM's GPFS... I've seen it fail spectacularly. But their marketing materials said it is awesome! And now Amazon's EBS, essentially a networked block store, is failing. Don't trust a vendor about networked storage. Its a lie, just like the cake.

cageface 15y ago
NFS on a well designed and maintained LAN is ok, but on any network less reliable I agree it's the wrong abstraction. You can only get away with hiding the network in those rare situations where the network can be made that robust.
- pquerna 15y ago
  That is the point though, the #1 Fallacies of Distributed Computing: The network is reliable. There is no such thing as a well designed and maintained LAN. All networks are unreliable.
  - cageface 15y ago
    I worked for the last ten years at a major animation studio where we pushed huge I/O over NFS every day. This was a lot easier to do with NFS semantics than it would have been with a less "fallacious" model, but we only got away with it because we had complete control over the whole network.
oinksoft 15y ago
From somebody who only tends to do single-system administration...what do you use in place of NFS? Local storage and sync it? Something else?
cagenut 15y ago
EBS isn't a shared filesystem, its a block device mountable by only one host (instance) at a time. Much more analogous to fibrechannel or iscsi than to nfs.
ssmoot 15y ago
That's an odd thing to say. Internal networks can easily be more reliable than applications given that higher end equipment has a lot more money poured into it at a development level, and anything above the bottom end of the market typically has multiple layers of redundancy options available. Wether it's VRRP, stacking, bonding, IPMP, iSCSI channels etc. EBS is unreliable, granted, but blaming the network seems faulty. For one, it's nothing like NFS. For two, NFS can be adapted to highly-available systems fairly easily. These days it's a pretty simple system. Even with just a single Array, put two FreeBSD "heads" on it, MPIO connectivity, and CARP between the boxes. Monitor the CARP interface to import/export the file-systems and there you go. If you can tolerate a minute or two of downtime when a host fails you have reliable, automatic redundancy. EBS, FCoE, iSCSI, et all are entirely different beasts. But again, you can make the hosts redundant pretty easily. With iSCSI you can give the clients multiple paths to the devices increasing reliability and available bandwidth without LACP. Consolidate storage needs to save money. It's win/win/win. Networked storage, wether it's block devices or shared file-systems, whatever the technology, aren't going away anytime soon. Matter of fact, they're becoming more critical to operations day by day. And these days, they're one of the easiest things to setup, and ensure availability on. That EBS is the weak-link at Amazon is a bit counter-intuitive based on my own experience. I wonder what the root problems actually are...
- jerf 15y ago
  Let me summarize the argument against using network filesystems. Elsewhere on the HN homepage as I write this [1] there is a link to the Wikipedia article on the Fallacies of Distributed Computing [2], which are: 1. The network is reliable. 2. Latency is zero. 3. Bandwidth is infinite. 4. The network is secure. 5. Topology doesn't change. 6. There is one administrator. 7. Transport cost is zero. 8. The network is homogeneous. Technically, every one of them is a myth of local disk storage as well, inasmuch as nothing is every 100% reliable in the mathematical sense. In reality, unlike network programming where the moment you step out into the big world you are slammed in the face by the mythical nature of these claims, they are true enough for when you are working with local disk that you can write significant programs as if they are true, excepting perhaps latency (busy disk may mean you get scheduled much later). The basic argument is the practical differences are so significant that where the file interface is most effective when it just pretends most of the myths are true because adding the complexity of being able to deal with violations isn't worth it (and most programmers using local disk would get it wrong even if you offered it; how many programs gracefully handle running out of disk space, let alone more bizarre failure cases?), the network must give you the tools to handle those issues, and correspondingly must have a more complicated API. In particular, every time you so much as touch the network, you need a code path to handle the possibility that the network communication utterly failed, just disappeared into oblivion. You just don't code that way when you deal with files; the file may be gone, the disk may run out of space, but you don't sit there in the middle of a fast read loop and handle failure on every read, such as a topology change, i.e., "the disk drive is no longer present". That's not because it's not possible, because you can certainly yank out a USB key mid-read, but you tend not to handle it in most userland code. In fact the OS won't even let you handle many failures, as it will either abstract away the retries or simply lock your userspace hard. So the fundamental idea is that filesystem abstractions built for relatively reliable communication simply isn't correct for network communication. Incidentally, the same argument can pretty much be made against remote RPC by simply replacing "disk access" with "function call". You do not have to guard every function call you make with a check for absolute failure. (Not the function returning failure or the function throwing an exception, but the function that may have been there two minutes ago unexpectedly being entirely gone.) RPC may look cute in the source code but it's the wrong layer of abstraction. [1]: http://news.ycombinator.com/item?id=2470865 http://news.ycombinator.com/item?id=2470865 [2]: http://en.wikipedia.org/wiki/Fallacies_of_Distributed_Computing http://en.wikipedia.org/wiki/Fallacies_of_Distributed_Comput...
tybris 15y ago
Not sure if I agree. Networked filesystems require a strong consistency model, which forces you to give up some reliability, as we can plainly see today. That doesn't mean to say that strong consistency models are bad. They're easier and cheaper to work with. If your system is not so mission-critical or self-sufficient that spending the time, effort and money to develop a fault-tolerant, eventually consistent system (e.g., see Netflix) is worth doing, then relying on a network filesystem isn't such a bad option. Of course, when you do, you should realize that the reliability issues tend to be a lot more fundamental than which provider you use, the number of customers they have, or what buzzword people associate it with them.

by pquerna 15y ago

cageface 15y ago

NFS on a well designed and maintained LAN is ok, but on any network less reliable I agree it's the wrong abstraction. You can only get away with hiding the network in those rare situations where the network can be made that robust.

pquerna 15y ago
That is the point though, the #1 Fallacies of Distributed Computing: The network is reliable. There is no such thing as a well designed and maintained LAN. All networks are unreliable.
- cageface 15y ago
  I worked for the last ten years at a major animation studio where we pushed huge I/O over NFS every day. This was a lot easier to do with NFS semantics than it would have been with a less "fallacious" model, but we only got away with it because we had complete control over the whole network.

oinksoft 15y ago

From somebody who only tends to do single-system administration...what do you use in place of NFS? Local storage and sync it? Something else?

cagenut 15y ago

EBS isn't a shared filesystem, its a block device mountable by only one host (instance) at a time. Much more analogous to fibrechannel or iscsi than to nfs.

ssmoot 15y ago

That's an odd thing to say. Internal networks can easily be more reliable than applications given that higher end equipment has a lot more money poured into it at a development level, and anything above the bottom end of the market typically has multiple layers of redundancy options available. Wether it's VRRP, stacking, bonding, IPMP, iSCSI channels etc. EBS is unreliable, granted, but blaming the network seems faulty. For one, it's nothing like NFS. For two, NFS can be adapted to highly-available systems fairly easily. These days it's a pretty simple system. Even with just a single Array, put two FreeBSD "heads" on it, MPIO connectivity, and CARP between the boxes. Monitor the CARP interface to import/export the file-systems and there you go. If you can tolerate a minute or two of downtime when a host fails you have reliable, automatic redundancy. EBS, FCoE, iSCSI, et all are entirely different beasts. But again, you can make the hosts redundant pretty easily. With iSCSI you can give the clients multiple paths to the devices increasing reliability and available bandwidth without LACP. Consolidate storage needs to save money. It's win/win/win. Networked storage, wether it's block devices or shared file-systems, whatever the technology, aren't going away anytime soon. Matter of fact, they're becoming more critical to operations day by day. And these days, they're one of the easiest things to setup, and ensure availability on. That EBS is the weak-link at Amazon is a bit counter-intuitive based on my own experience. I wonder what the root problems actually are...

jerf 15y ago
Let me summarize the argument against using network filesystems. Elsewhere on the HN homepage as I write this [1] there is a link to the Wikipedia article on the Fallacies of Distributed Computing [2], which are: 1. The network is reliable. 2. Latency is zero. 3. Bandwidth is infinite. 4. The network is secure. 5. Topology doesn't change. 6. There is one administrator. 7. Transport cost is zero. 8. The network is homogeneous. Technically, every one of them is a myth of local disk storage as well, inasmuch as nothing is every 100% reliable in the mathematical sense. In reality, unlike network programming where the moment you step out into the big world you are slammed in the face by the mythical nature of these claims, they are true enough for when you are working with local disk that you can write significant programs as if they are true, excepting perhaps latency (busy disk may mean you get scheduled much later). The basic argument is the practical differences are so significant that where the file interface is most effective when it just pretends most of the myths are true because adding the complexity of being able to deal with violations isn't worth it (and most programmers using local disk would get it wrong even if you offered it; how many programs gracefully handle running out of disk space, let alone more bizarre failure cases?), the network must give you the tools to handle those issues, and correspondingly must have a more complicated API. In particular, every time you so much as touch the network, you need a code path to handle the possibility that the network communication utterly failed, just disappeared into oblivion. You just don't code that way when you deal with files; the file may be gone, the disk may run out of space, but you don't sit there in the middle of a fast read loop and handle failure on every read, such as a topology change, i.e., "the disk drive is no longer present". That's not because it's not possible, because you can certainly yank out a USB key mid-read, but you tend not to handle it in most userland code. In fact the OS won't even let you handle many failures, as it will either abstract away the retries or simply lock your userspace hard. So the fundamental idea is that filesystem abstractions built for relatively reliable communication simply isn't correct for network communication. Incidentally, the same argument can pretty much be made against remote RPC by simply replacing "disk access" with "function call". You do not have to guard every function call you make with a check for absolute failure. (Not the function returning failure or the function throwing an exception, but the function that may have been there two minutes ago unexpectedly being entirely gone.) RPC may look cute in the source code but it's the wrong layer of abstraction. [1]: http://news.ycombinator.com/item?id=2470865 http://news.ycombinator.com/item?id=2470865 [2]: http://en.wikipedia.org/wiki/Fallacies_of_Distributed_Computing http://en.wikipedia.org/wiki/Fallacies_of_Distributed_Comput...

tybris 15y ago

Not sure if I agree. Networked filesystems require a strong consistency model, which forces you to give up some reliability, as we can plainly see today. That doesn't mean to say that strong consistency models are bad. They're easier and cheaper to work with. If your system is not so mission-critical or self-sufficient that spending the time, effort and money to develop a fault-tolerant, eventually consistent system (e.g., see Netflix) is worth doing, then relying on a network filesystem isn't such a bad option. Of course, when you do, you should realize that the reliability issues tend to be a lot more fundamental than which provider you use, the number of customers they have, or what buzzword people associate it with them.