6 ms·

Assuming the problem is indeed with EBS, I would say this should be a warning sign to anyone considering going with a PaaS provider, which Amazon is quickly bec

by olegp 16y ago

Assuming the problem is indeed with EBS, I would say this should be a warning sign to anyone considering going with a PaaS provider, which Amazon is quickly becoming, instead of an IaaS provider like Slicehost or Linode. The increased complexity of their offering makes it more likely that things will break, leaving you locked in. I did a 15 minute talk on the subject, which you can check out here: http://iforum.com.ua/video-2011-tech-podsechin http://iforum.com.ua/video-2011-tech-podsechin EDIT: here are the slides if you can't bother watching the video http://bit.ly/eqDNei http://bit.ly/eqDNei

ghshephard 16y ago
Every time someone makes the claim that downtime should be a warning sign about going with a PaaS provider (or, indeed, an IaaS provider, or in some cases, people even make this claim about going with someone else's Data Center) - I always respond: "And why do you believe that you would do any better?" Every environment I've been involved in as an operations professional for the last 15 years has experienced downtime regardless of how much we invested in staging, testing, change control procedures, redundancy and ITIL methodology. While it is true that there are some environments in the world that experience no, or close to no downtime (The NYSE, google.com, the Space Shuttle) - it's usually not worth the investments those organizations make in ensuring 100% uptime. I'm more interested in seeing what the monitoring methodology is, what the response protocols are, and what the reported downtime has been over the last year has been, than I am about getting too concerned about the occasional outage.
- bad_user 16y ago
  Lets hope they don't lose any of my data.
- olegp 16y ago
  The problem with Amazon is that despite touting an open API, their infrastructure internals and practices are a trade secret, so the likes of Eucalyptus are having to play catchup. In other words, I cannot replicate their infrastructure in my own data center, even I had the money to pay them. I suspect this is the main reason that Heroku didn't move off Amazon, and not the fact that Amazon was providing them great value for money. There is definitely value in platforms, but those platforms should be built on completely open standards with no vendor lock in or influence, as is the case with OpenStack for example: http://www.theregister.co.uk/2011/02/10/rackspace_buys_openstack_partner/ http://www.theregister.co.uk/2011/02/10/rackspace_buys_opens... If you didn't watch my talk, I should point out that I'm working on Akshell (http://www.akshell.com http://www.akshell.com), which is itself a platform provider, so I am in fact agreeing with you.
  - pzb 16y ago
    Compared to PaaS provides, Amazon is easy to migrate off of because they are simply giving you virtual hardware. The basics of their setup are well known: Xen and Linux. EBS is some sort of block-based network storage. True, we do not know the backend storage setup, but it does not really matter. The Xen VM simply sees a disk device. NetApp, EMC, Dell, HP and many others all have products that offer similar functionality, including snapshots. The only part that is potentially hard to migrate off is the security groups, and then only if you are using named groups. If you are just using IP based rules, most any firewall would work.
  - dillona 16y ago
    That's where I think we have to give credit to Rackspace for open sourcing all (or much) of their tech behind Rackspace Cloud
- billswift 16y ago
  Maybe that's true for an individual, but from a broader (societal/economic) view this is a bigger problem, because it affects many more people.
- mechanical_fish 16y ago
  Yep. The thing to remember is that even if you could do better -- and it is certainly possible -- your customers might not pay you for it. It will cost you, a lot, to build a significantly better system than AWS. Reliability is about redundancy, and redundancy is synonymous with paying for things that sit idle for years at a time, and paying for the hardware and personnel to run disaster drills over and over, and customers hate that. Reliability is also about avoiding excessive complexity and limiting the rate at which you change things, and customers hate that too. If they wanted highly reliable, slow-changing established tech they'd be using land-line phones, not Quora or Reddit.
- lobster_johnson 16y ago
  I agree with your sentiments, but this: > I always respond: "And why do you believe that you would do any better?" …is an apples-to-oranges comparison that implies that the thing that Amazon is currently doing is what you would be doing in an Amazon-less scenario. It's not. Amazon hosts thousands of customers and needs to service all of them in the same infrastructure, which coincidentally is also shared between every customer by virtue of being virtualized. It's a complex structure in which every resource -- CPU, RAM, disk, network -- is shared, which means that even a minor EBS problem could potentially have a butterfly-effect propagating through multiple customer's hosts. There are limitations imposed by the structure; for example, the only way to access large amounts of block storage with local-like performance is through EBS. But EBS has been shown to be high-latency and flaky, and Reddit's chronic problems with EBS is a good example why it one probably should stay away from it entirely. Network latency on EC2 is also pretty horrendous compared to classic non-virtual setups; I have set up HAProxy clusters where the connection setup time to backend hosts would be measured in tens of milliseconds. Now, if you hosted your own stuff, you would be the only customer, and you would not be sharing resources with anyone else, and you could design your hardware exactly to your specifications (fast local disks with low latency, fast networking with low latency and so on). The difference in complexity is significant. There are tons of challenges and costs involved in hosting your own stuff and maintaining uptime, but the complexity equation is different.
- deleted 16y ago
  [deleted]
justincormack 16y ago
I dont see EBS as a very PaaS type service. It is a virtualised network storage volume with snapshotting, just what you might expect from an IaaS provider. Clearly it is one of the more complex parts of the Amazom infrastructure though (other providers dont provide anything similar). If you dont use EBS many applications would have to implement a lot of its functionality themselves, and I suspect this would lead to some data loss instead, when drives fail or instances fail leaving drives unavailable. You can avoid EBS if you use local drives and s3, it is just difficult to run a database like this if it was not built that way.

by olegp 16y ago

ghshephard 16y ago

Every time someone makes the claim that downtime should be a warning sign about going with a PaaS provider (or, indeed, an IaaS provider, or in some cases, people even make this claim about going with someone else's Data Center) - I always respond: "And why do you believe that you would do any better?" Every environment I've been involved in as an operations professional for the last 15 years has experienced downtime regardless of how much we invested in staging, testing, change control procedures, redundancy and ITIL methodology. While it is true that there are some environments in the world that experience no, or close to no downtime (The NYSE, google.com, the Space Shuttle) - it's usually not worth the investments those organizations make in ensuring 100% uptime. I'm more interested in seeing what the monitoring methodology is, what the response protocols are, and what the reported downtime has been over the last year has been, than I am about getting too concerned about the occasional outage.

bad_user 16y ago
Lets hope they don't lose any of my data.
olegp 16y ago
The problem with Amazon is that despite touting an open API, their infrastructure internals and practices are a trade secret, so the likes of Eucalyptus are having to play catchup. In other words, I cannot replicate their infrastructure in my own data center, even I had the money to pay them. I suspect this is the main reason that Heroku didn't move off Amazon, and not the fact that Amazon was providing them great value for money. There is definitely value in platforms, but those platforms should be built on completely open standards with no vendor lock in or influence, as is the case with OpenStack for example: http://www.theregister.co.uk/2011/02/10/rackspace_buys_openstack_partner/ http://www.theregister.co.uk/2011/02/10/rackspace_buys_opens... If you didn't watch my talk, I should point out that I'm working on Akshell (http://www.akshell.com http://www.akshell.com), which is itself a platform provider, so I am in fact agreeing with you.
- pzb 16y ago
  Compared to PaaS provides, Amazon is easy to migrate off of because they are simply giving you virtual hardware. The basics of their setup are well known: Xen and Linux. EBS is some sort of block-based network storage. True, we do not know the backend storage setup, but it does not really matter. The Xen VM simply sees a disk device. NetApp, EMC, Dell, HP and many others all have products that offer similar functionality, including snapshots. The only part that is potentially hard to migrate off is the security groups, and then only if you are using named groups. If you are just using IP based rules, most any firewall would work.
- dillona 16y ago
  That's where I think we have to give credit to Rackspace for open sourcing all (or much) of their tech behind Rackspace Cloud
billswift 16y ago
Maybe that's true for an individual, but from a broader (societal/economic) view this is a bigger problem, because it affects many more people.
mechanical_fish 16y ago
Yep. The thing to remember is that even if you could do better -- and it is certainly possible -- your customers might not pay you for it. It will cost you, a lot, to build a significantly better system than AWS. Reliability is about redundancy, and redundancy is synonymous with paying for things that sit idle for years at a time, and paying for the hardware and personnel to run disaster drills over and over, and customers hate that. Reliability is also about avoiding excessive complexity and limiting the rate at which you change things, and customers hate that too. If they wanted highly reliable, slow-changing established tech they'd be using land-line phones, not Quora or Reddit.
lobster_johnson 16y ago
I agree with your sentiments, but this: > I always respond: "And why do you believe that you would do any better?" …is an apples-to-oranges comparison that implies that the thing that Amazon is currently doing is what you would be doing in an Amazon-less scenario. It's not. Amazon hosts thousands of customers and needs to service all of them in the same infrastructure, which coincidentally is also shared between every customer by virtue of being virtualized. It's a complex structure in which every resource -- CPU, RAM, disk, network -- is shared, which means that even a minor EBS problem could potentially have a butterfly-effect propagating through multiple customer's hosts. There are limitations imposed by the structure; for example, the only way to access large amounts of block storage with local-like performance is through EBS. But EBS has been shown to be high-latency and flaky, and Reddit's chronic problems with EBS is a good example why it one probably should stay away from it entirely. Network latency on EC2 is also pretty horrendous compared to classic non-virtual setups; I have set up HAProxy clusters where the connection setup time to backend hosts would be measured in tens of milliseconds. Now, if you hosted your own stuff, you would be the only customer, and you would not be sharing resources with anyone else, and you could design your hardware exactly to your specifications (fast local disks with low latency, fast networking with low latency and so on). The difference in complexity is significant. There are tons of challenges and costs involved in hosting your own stuff and maintaining uptime, but the complexity equation is different.
deleted 16y ago
[deleted]

justincormack 16y ago

I dont see EBS as a very PaaS type service. It is a virtualised network storage volume with snapshotting, just what you might expect from an IaaS provider. Clearly it is one of the more complex parts of the Amazom infrastructure though (other providers dont provide anything similar). If you dont use EBS many applications would have to implement a lot of its functionality themselves, and I suspect this would lead to some data loss instead, when drives fail or instances fail leaving drives unavailable. You can avoid EBS if you use local drives and s3, it is just difficult to run a database like this if it was not built that way.