4 ms·

I love this quote: > I worry a little bit about hiding the places where the networking happens, just like I worry about ORM hiding the SQL. Because you can’t i

by mrkurt 6y ago

I love this quote: > I worry a little bit about hiding the places where the networking happens, just like I worry about ORM hiding the SQL. Because you can’t ignore either networking or SQL. The irony (or not irony, who even knows?), though, is that I always worried a little bit about EBS for similar reasons. It's gotten really, really good but it was a terrifying abstraction back in the day.

viraptor 6y ago
> it was a terrifying abstraction back in the day. How come? NAS, SAN, iSCSI, etc. existed and were commonly used way before AWS or EBS. And EBS really is a dynamically routed network share in the end.
- donavanm 6y ago
  You get a lot more visibility and control with iSCSI, ATA over eth, infiniband, etc. The downside is you need to use and understand that visibility and control. EBS is surfaced to you as a simple locally attached block dev. The network performance and challenges are not just managed for you, they’re largely opaque. As the user you can’t observe the (equivalent to) fiber channel topology, SCSI commands, discarded frames, etc. and that’s the power of EBS as well, users don’t have to deal with anything besides a locally attached virtual disk.
  - p_l 6y ago
    So much this. There's also the part where EBS SLA and durability might surprise you, like the great "internet disappeared" around 10 years ago.
    - donavanm 6y ago
      The last point is a good one around absolute vs subjective experience. If you’re looking at 1M disks 99.999% availability is great. But if your entire company runs on 10 instances your day will be utterly ruined when any one of those root volumes goes away. To that sad admin it feels like they’re sold 1/100,000 and got 1/10. And their impact is closer to “business critically failing” than “active automated remediation workflow incremented.”
- acdha 6y ago
  > NAS, SAN, iSCSI, etc. existed and were commonly used way before AWS or EBS And were notorious sources of performance and reliability issues, especially iSCSI. Anyone with operational experience with those was not unreasonably questioning the risk of being in the same boat with less control, and EBS had some highly-visible failures which validated that concern. It took a number of smooth years after the big 2011 outage to get people to be more confident in it.
  - tatersolid 6y ago
    In my 20+ years of stoarage and ops experience, iSCSI was way more reliable, tunable, and monitor-able than anything in the FiberChannel world. “Just use TCP/IP because everything else does” is a strategy that works, and it’s the reason FC is (mostly) dead. Though nothing beat the braided differential parallel SCSI cables for cable porn supremacy. Shared-disk clusters looked like they were damn serious alien-military-grade stuff back then.
    - acdha 6y ago
      Maybe you were lucky - I had the misfortune of supporting some iSCSI devices which were slower & less stable, and lots of places had to have the “just because it uses the same cable doesn’t mean you should share a port” talk with their network people – but I was mostly comparing it to direct-attached storage which has fewer parts to fail and, critically, doesn’t have the same correlated failures which you get with shared anything (and, of course, the trade-off or forcing you to do redundancy at a higher level). Most of the problems I remember in the last decade fell into that last category: over-commitment and reserved capacity, failures showing that redundancy wasn’t as good as the vendor promised, etc. which are quite reasonably concerns for EBS, too.
      - tatersolid 6y ago
        I dealt mostly with LeftHand and EqualLogic gear when iSCSI was getting rolling. Both were far more reliable than the various FC gear I dealt with around the same time. You could pull any node and things kept chugging. Rolling software updates were non-events. Simplicity has its virtues I suppose. Running 4 or 8 gigabit cables to each node on dedicated switch ports likely also helped.

by mrkurt 6y ago

viraptor 6y ago

> it was a terrifying abstraction back in the day. How come? NAS, SAN, iSCSI, etc. existed and were commonly used way before AWS or EBS. And EBS really is a dynamically routed network share in the end.

donavanm 6y ago
You get a lot more visibility and control with iSCSI, ATA over eth, infiniband, etc. The downside is you need to use and understand that visibility and control. EBS is surfaced to you as a simple locally attached block dev. The network performance and challenges are not just managed for you, they’re largely opaque. As the user you can’t observe the (equivalent to) fiber channel topology, SCSI commands, discarded frames, etc. and that’s the power of EBS as well, users don’t have to deal with anything besides a locally attached virtual disk.
- p_l 6y ago
  So much this. There's also the part where EBS SLA and durability might surprise you, like the great "internet disappeared" around 10 years ago.
  - donavanm 6y ago
    The last point is a good one around absolute vs subjective experience. If you’re looking at 1M disks 99.999% availability is great. But if your entire company runs on 10 instances your day will be utterly ruined when any one of those root volumes goes away. To that sad admin it feels like they’re sold 1/100,000 and got 1/10. And their impact is closer to “business critically failing” than “active automated remediation workflow incremented.”
acdha 6y ago
> NAS, SAN, iSCSI, etc. existed and were commonly used way before AWS or EBS And were notorious sources of performance and reliability issues, especially iSCSI. Anyone with operational experience with those was not unreasonably questioning the risk of being in the same boat with less control, and EBS had some highly-visible failures which validated that concern. It took a number of smooth years after the big 2011 outage to get people to be more confident in it.
- tatersolid 6y ago
  In my 20+ years of stoarage and ops experience, iSCSI was way more reliable, tunable, and monitor-able than anything in the FiberChannel world. “Just use TCP/IP because everything else does” is a strategy that works, and it’s the reason FC is (mostly) dead. Though nothing beat the braided differential parallel SCSI cables for cable porn supremacy. Shared-disk clusters looked like they were damn serious alien-military-grade stuff back then.
  - acdha 6y ago
    Maybe you were lucky - I had the misfortune of supporting some iSCSI devices which were slower & less stable, and lots of places had to have the “just because it uses the same cable doesn’t mean you should share a port” talk with their network people – but I was mostly comparing it to direct-attached storage which has fewer parts to fail and, critically, doesn’t have the same correlated failures which you get with shared anything (and, of course, the trade-off or forcing you to do redundancy at a higher level). Most of the problems I remember in the last decade fell into that last category: over-commitment and reserved capacity, failures showing that redundancy wasn’t as good as the vendor promised, etc. which are quite reasonably concerns for EBS, too.
    - tatersolid 6y ago
      I dealt mostly with LeftHand and EqualLogic gear when iSCSI was getting rolling. Both were far more reliable than the various FC gear I dealt with around the same time. You could pull any node and things kept chugging. Rolling software updates were non-events. Simplicity has its virtues I suppose. Running 4 or 8 gigabit cables to each node on dedicated switch ports likely also helped.