6 ms·

Magical Block Store: Why EBS Can't Work

124 points by lindvall 15y ago

prodigal_erik 15y ago
He didn't touch on Joyent's 2+ day partial outage a couple months ago: http://news.ycombinator.com/item?id=2269329 http://news.ycombinator.com/item?id=2269329
- jamie 15y ago
  Don't forget about this: http://www.datacenterknowledge.com/archives/2008/01/15/joyent-backup-services-down-for-three-days/ http://www.datacenterknowledge.com/archives/2008/01/15/joyen...
  - jamie 15y ago
    I think both of these links illustrate that errors happen, mistakes happen, software has bugs, and murphy's law always strikes. The question is, when it strikes, do you have enough control to fix the problem? If you've outsourced the solution, does the provider have enough control/knowledge to fix the problem? These things will get much worse before they get better, and it's best to think of all these abstractions as being a double edge sword.
spullara 15y ago
Also worth noting is that Amazon isn't forcing you to use EBS. They also have tons of fast local storage available to RAID as you wish.
- smhinsey 15y ago
  Do you mean RAIDing local instance storage? I haven't heard of this approach so far, do you have any links?
  - blantonl 15y ago
    Depending on the instance that you have provisioned, you have anywhere from 1 to 4 ephemeral (local) disks available to your instance. They are typically available as /dev/sd[bcde] In centOS, implementing a RAID-0 block device across 2 ephemeral disks that is present on an m1.large instance can be done via the following: mdadm --create /dev/md0 --metadata=1.1 --level=0 --quiet --run -c 256 -n 2 /dev/sdb /dev/sdc You'll then need to format the block device with your fs of choice. Then mount it from there.
    - count 15y ago
      Just remember the super important keyword: ephemeral WHEN your EC2 node disappears (and it will), you will lose everything on that RAID. That's not a bad thing if you know it'll happen and plan for it, but do be aware of it.
- deleted 15y ago
  [deleted]
- lindvall 15y ago
  I strongly believe one of the most positive aspects of EC2 was that it demonstrated a beautiful philosophy that a node and their disks should not be relied upon to always be around and pushed it into the mainstream. Even for people who didn't use EC2 the existence of the platform caused more people to rethink their architectures to try to rely less on Important Nodes. EBS is a step back from that philosophy and it's a point worth noting. One of the great things this post does is enumerates some of the underlying reasons why relying on EBS will inevitably lead to more failures and in ways that are harder and harder to diagnose.
  - leoc 15y ago
    > EBS is a step back from that philosophy and it's a point worth noting. Amazon doesn't use EBS itself, right? Isn't EBS something that AWS allowed its customers to nag it into against (what it considers) its better judgement?
    - lindvall 15y ago
      Yep. And this may be one of those cases where they would have been better off ignoring their customer requests for the good of their reputation and their customers uptime.
- blantonl 15y ago
  I agree. We run all of our MySQL and Mongo slave servers with local RAID-0 ephemeral storage. One dies? So what, we remove it from the pool and provision another. Only our master and our slave backup server runs on EBS. We aren't as write oriented so we can live with some of the limitations of EBS, but we've even considered moving our master MySQL and Mongo servers to ephemeral storage and just relying on our slave back database server to run on EBS (for which we take freeze/snapshots of often). That server rarely ever falls more behind in relay updates.
  - epi0Bauqu 15y ago
    Do you run xlarge to get the extra ephemeral disks?
- jinushaun 15y ago
  Except Amazon does... Their free micro instance only supports EBS. For some of us who just want to dip our toes into AWS without too much money or commitment, EBS is our only choice.
  - spullara 15y ago
    Then you really don't need to worry about fault tolerance or uptime.
blantonl 15y ago
I am an active user of EBS on a highly trafficked Web properly, and came from a long and tedious background in enterprise software. I really think that one paragraph in his blog post summed everything up quite nicely. It could not ring more true: My opinion is that the only reason the big enterprise storage vendors have gotten away with network block storage for the last decade is that they can afford to over-engineer the hell out of them and have the luxury of running enterprise workloads, which is a code phrase for “consolidated idle workloads.” When the going gets tough in enterprise storage systems, you do capacity planning and make sure your hot apps are on dedicated spindles, controllers, and network ports.
spudlyo 15y ago
Trying to use a tool like iostat against a shared, network provided block device to figure out what your level of service your database is getting from the filesystem below it is an exercise in frustration that will get you nowhere. This may be true under Solaris. Since 2.5 Linux has had /proc/diskstats and an iostat that shows the average i/o request latency (await) for a disk, network or otherwise. For EBS it's 40ms or less on a good day. On a bad day it's 500ms or more if your i/o requests get serviced at all.
SoftwareMaven 15y ago
Many things in software are impossible magic, until they are not. His argument boils down to "it is a hard problem that nobody has solved yet." That doesn't mean nobody will ever solve it. Regardless, I do agree that building your application today like it is a solved problem is the wrong way to do it.
- blantonl 15y ago
  Regardless, I do agree that building your application today like it is a solved problem is the wrong way to do it. That presumption assumes that the application is being used as the right tool to resolve the problem. And it also assumes that "the problem" is a finite and solvable item.
  - sigil 15y ago
    > And it also assumes that "the problem" is a finite and solvable item. Yes. To make this a bit more concrete, if "the problem" is making distributed storage look and behave exactly like local storage, the CAP Theorem has something to say about its solvability.
    - jamesaguilar 15y ago
      Depends. Local storage is also not perfectly available. If the network is reliable, you can probably get availability high enough that the system feels "close enough" to how local storage feels. Today's networks aren't that reliable, but someday there may be enough redundancy and bandwidth for this to happen.
      - sigil 15y ago
        > Local storage is also not perfectly available. Technically true, although you don't have to contend with the consistency or partitioning factors in the local disk case -- there's only one copy of the state. This means you can focus on making the availability factor as close to 1.0 as possible. This may not be the case when you're forced to balance all three CAP factors. I sometimes wonder if a follow on result to CAP will be a "practical" (physical or information theoretic) limit like C x A x P <= 1-h for some constant h, and we'll just have to come to terms with that as computer scientists, as physics had to with dx x dp >= h. This is of course wildly unsubstantiated pessimism. Also, I would gladly entertain any argument demolishing the "local disks are not subject to CAP" claim I made above by talking about read / write caches as separate copies of the local disk state.
cagenut 15y ago
Its really fascinating to watch amazon re-learn/re-implement the lessons IBM baked into mainframes decades ago. Once you get out of shared-nothing/web-scripting land you realize that I/O is much more important and difficult than cpu. What amazon calls EBS IBM has been calling "DASD" forever. I wonder if there are any crossover lessons that they haven't taken advantage of because there just aren't any old ibm'ers working at amazon.
- blantonl 15y ago
  IBM's implementation of DASD on the mainframe was always implemented under the assumption that it was a secondary storage medium for data. Meaning, it wasn't accessed often, and it wasn't implemented for top performance. Think of a bridge between high performance disk and tape.
deleted 15y ago
[deleted]
johnb 15y ago
It's funny how disk abstractions get you every time. We used to store and process all of our uploads from our rails app on a GFS partition. GFS behaved like a normal disk most of the time, but we started having trouble processing concurrent uploads and couldn't replicate in dev. It turned out so GFS could work at all, it had different locking than regular disks. Every time you created a new file it had to lock the containing folder. We solved it by splitting our upload folder in 1000 sequential buckets and wrote each upload to the next folder along... but it took us a long time to stop assuming it was a regular disk.
- sciurus 15y ago
  FWIW, this behavior is explained early on in the documentation for GFS2.
  - johnb 15y ago
    As we were using EngineYard for hosting at the time, everything was set up for us and we never thought to look it up. We now pay a lot more attention to underlying stack. Just because you've outsourced hosting (either cloud or managed physical servers), you really need to know every component yourself.
edw 15y ago
This awesome entry perfectly captures why I have always hated NFS. I can deal with the possibility that if a machine's hard drive dies, my system is going to have a very hard time continuing to operate in a normal manner, but then NFS comes along, and you realize that all sorts of I/O operations that previously employed a piece of equipment that failed once every two and a half years now depend on a working network with a working NFS server on that network, and the combination of that network and that server are orders of magnitude less reliable. And now you have situations on a regular basis where you type "ls" and you shell hangs and not even "kill -9" is going to save you. And you go back to using FTP or some other abstraction that does not apply 40,000 hour MTBF thinking to equipment that disappears for coffee breaks daily.
- chubot 15y ago
  A thousand times yes. My first thought when hearing about the EBS outage was "wow that seems more fragile than NFS, no wonder it failed spectacularly." NFS presents you with this nice familiar file system interface, and then random sys admins and programmers start creating a tangled mess of dependencies by dropping stuff there, without regard to what happens when it fails. Like the EBS outage, the failures tend to surprise people. The great quote by Leslie Lamport: "A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable." And this was an excellent and honest article about faulty programming abstractions. It's basically bashing you over the head with the "Fallacies of distributed computing". Don't silently turn local operations into remote operations. They're not the same thing and have to be treated differently at all levels. Even Werner Vogels wrote a diatribe against "transparency", which is the same issue by another name: http://scholar.google.com/scholar?cluster=7009698499164949721&hl=en&as_sdt=0,5 http://scholar.google.com/scholar?cluster=700969849916494972... So I wonder what he thinks of this architectural choice. You have to give up something when communicating over the network. Vogels seems to have chosen consistency rather than availability in his designs. This paper was a turning point in his research. Its candor surprised me. The file system interface does not let you relax consistency, so by default you have chosen availability. As the Joyent guys honestly remarked, this often has to be learned the hard way.
- agazso 15y ago
alecco 15y ago
Amazon Six Sigma "Blackbelts", meet Mr. Black Swan. Edit: my point is you can't hide unexpected/unknown events on statistical models; we should know better, coming from CS.
CPlatypus 15y ago
He only gets it half right. A filesystem interface instead of a block interface is the right choice IMO. Private storage instead of distributed storage is the wrong choice for capacity, performance, and (most importantly) availability reasons. They didn't go with a ZFS-based solution because it was the best fit to requirements. They went with it because they had ZFS experts and advocates on staff. As Schopenhauer said, every man mistakes the limits of his own vision for the limits of the world, and these are people who've failed to Get It when it comes to distributed storage ever since they tried and failed to make ZFS distributed (leading to the enlistment of the Lustre crew who have also largely failed at the same task). If they can't solve a problem they're arrogant enough to believe nobody can, so they position DAS and SAN as the only possible alternatives. Disclaimers: I'm the project lead for CloudFS, which is IMO exactly the kind of distributed storage people should be using for this sort of thing. I've also had some fairly public disputes with Bryan "Jackass" Cantrill, formerly of Sun and now of Joyent, about ZFS FUD.
- Nrsolis 15y ago
  Do you have any opinions of Ceph? They seem to be dong a similar thing.
- ssmoot 15y ago
  ZFS is just the FS. But you know that already. The SAN solutions they migrated to are not ZFS based. Unless I'm mis-remembering (I read this a couple days ago) they were only using ZFS to slice LUNs. Point is, you're taking pot-shots at ZFS when the main thrust appears to be: "It was hard to make iSCSI reliable. Once we did, by buying expensive storage-vendor backed solutions, we found it wasn't financially compelling." They're a hosting provider. If it takes a replicated SAN pair (which is the wrong way to go about it BTW, though admittedly it's the way the storage vendors and their "appliance" mentality would have it done) to service just a pair of VM hosts (they're still using Zones right?) then it just didn't make sense money-wise for them. If they planned capacity to provide great performance, they weren't making enough money on the services for what they were selling them for. That's not an "iSCSI is unreliable" problem. It's not a "networked storage is broken" problem. It's not a "networked storage is slow" problem. It's not even a "ZFS didn't work out" problem. If you go out and spend major bucks on NetApp, not only are you going to have to deal with all the black-box-appliance BS, but it's going to cost a lot of money. A LOT. And DAS is going to end up cheaper to deploy, maintain, and your margins are going to be a lot higher. DAS is the right choice for a hosting provider who wants to maximize their profits in a competitive space. It's not the best choice for performance, availability or flexibility for clients though. So you have to ask yourself what kind of budget you have to work with, and what goals are important to you? BTW, there's _budget_, and then there's NetApp/EMC budget. Just because you need/want more than DAS can give you doesn't mean you need to tie your boat to an insane Enterprise grade budget.
lobster_johnson 15y ago
> It’s commonly believed that EBS is built on DRBD with a dose of S3-derived replication logic. Actually, it was discovered some time ago (http://openfoo.org/blog/amazon_ec2_underlying_architecture.html http://openfoo.org/blog/amazon_ec2_underlying_architecture.h...) that EBS probably used Red Hat's open-source GNDB: http://sourceware.org/cluster/gnbd/ http://sourceware.org/cluster/gnbd/

124 points by lindvall 15y ago

He didn't touch on Joyent's 2+ day partial outage a couple months ago: http://news.ycombinator.com/item?id=2269329 http://news.ycombinator.com/item?id=2269329

jamie 15y ago
Don't forget about this: http://www.datacenterknowledge.com/archives/2008/01/15/joyent-backup-services-down-for-three-days/ http://www.datacenterknowledge.com/archives/2008/01/15/joyen...
- jamie 15y ago
  I think both of these links illustrate that errors happen, mistakes happen, software has bugs, and murphy's law always strikes. The question is, when it strikes, do you have enough control to fix the problem? If you've outsourced the solution, does the provider have enough control/knowledge to fix the problem? These things will get much worse before they get better, and it's best to think of all these abstractions as being a double edge sword.

spullara 15y ago

Also worth noting is that Amazon isn't forcing you to use EBS. They also have tons of fast local storage available to RAID as you wish.

smhinsey 15y ago
Do you mean RAIDing local instance storage? I haven't heard of this approach so far, do you have any links?
- blantonl 15y ago
  Depending on the instance that you have provisioned, you have anywhere from 1 to 4 ephemeral (local) disks available to your instance. They are typically available as /dev/sd[bcde] In centOS, implementing a RAID-0 block device across 2 ephemeral disks that is present on an m1.large instance can be done via the following: mdadm --create /dev/md0 --metadata=1.1 --level=0 --quiet --run -c 256 -n 2 /dev/sdb /dev/sdc You'll then need to format the block device with your fs of choice. Then mount it from there.
  - count 15y ago
    Just remember the super important keyword: ephemeral WHEN your EC2 node disappears (and it will), you will lose everything on that RAID. That's not a bad thing if you know it'll happen and plan for it, but do be aware of it.
deleted 15y ago
[deleted]
lindvall 15y ago
I strongly believe one of the most positive aspects of EC2 was that it demonstrated a beautiful philosophy that a node and their disks should not be relied upon to always be around and pushed it into the mainstream. Even for people who didn't use EC2 the existence of the platform caused more people to rethink their architectures to try to rely less on Important Nodes. EBS is a step back from that philosophy and it's a point worth noting. One of the great things this post does is enumerates some of the underlying reasons why relying on EBS will inevitably lead to more failures and in ways that are harder and harder to diagnose.
- leoc 15y ago
  > EBS is a step back from that philosophy and it's a point worth noting. Amazon doesn't use EBS itself, right? Isn't EBS something that AWS allowed its customers to nag it into against (what it considers) its better judgement?
  - lindvall 15y ago
    Yep. And this may be one of those cases where they would have been better off ignoring their customer requests for the good of their reputation and their customers uptime.
blantonl 15y ago
I agree. We run all of our MySQL and Mongo slave servers with local RAID-0 ephemeral storage. One dies? So what, we remove it from the pool and provision another. Only our master and our slave backup server runs on EBS. We aren't as write oriented so we can live with some of the limitations of EBS, but we've even considered moving our master MySQL and Mongo servers to ephemeral storage and just relying on our slave back database server to run on EBS (for which we take freeze/snapshots of often). That server rarely ever falls more behind in relay updates.
- epi0Bauqu 15y ago
  Do you run xlarge to get the extra ephemeral disks?
jinushaun 15y ago
Except Amazon does... Their free micro instance only supports EBS. For some of us who just want to dip our toes into AWS without too much money or commitment, EBS is our only choice.
- spullara 15y ago
  Then you really don't need to worry about fault tolerance or uptime.

blantonl 15y ago

I am an active user of EBS on a highly trafficked Web properly, and came from a long and tedious background in enterprise software. I really think that one paragraph in his blog post summed everything up quite nicely. It could not ring more true: My opinion is that the only reason the big enterprise storage vendors have gotten away with network block storage for the last decade is that they can afford to over-engineer the hell out of them and have the luxury of running enterprise workloads, which is a code phrase for “consolidated idle workloads.” When the going gets tough in enterprise storage systems, you do capacity planning and make sure your hot apps are on dedicated spindles, controllers, and network ports.

spudlyo 15y ago

Trying to use a tool like iostat against a shared, network provided block device to figure out what your level of service your database is getting from the filesystem below it is an exercise in frustration that will get you nowhere. This may be true under Solaris. Since 2.5 Linux has had /proc/diskstats and an iostat that shows the average i/o request latency (await) for a disk, network or otherwise. For EBS it's 40ms or less on a good day. On a bad day it's 500ms or more if your i/o requests get serviced at all.

SoftwareMaven 15y ago

Many things in software are impossible magic, until they are not. His argument boils down to "it is a hard problem that nobody has solved yet." That doesn't mean nobody will ever solve it. Regardless, I do agree that building your application today like it is a solved problem is the wrong way to do it.

blantonl 15y ago
Regardless, I do agree that building your application today like it is a solved problem is the wrong way to do it. That presumption assumes that the application is being used as the right tool to resolve the problem. And it also assumes that "the problem" is a finite and solvable item.
- sigil 15y ago
  > And it also assumes that "the problem" is a finite and solvable item. Yes. To make this a bit more concrete, if "the problem" is making distributed storage look and behave exactly like local storage, the CAP Theorem has something to say about its solvability.
  - jamesaguilar 15y ago
    Depends. Local storage is also not perfectly available. If the network is reliable, you can probably get availability high enough that the system feels "close enough" to how local storage feels. Today's networks aren't that reliable, but someday there may be enough redundancy and bandwidth for this to happen.
    - sigil 15y ago
      > Local storage is also not perfectly available. Technically true, although you don't have to contend with the consistency or partitioning factors in the local disk case -- there's only one copy of the state. This means you can focus on making the availability factor as close to 1.0 as possible. This may not be the case when you're forced to balance all three CAP factors. I sometimes wonder if a follow on result to CAP will be a "practical" (physical or information theoretic) limit like C x A x P <= 1-h for some constant h, and we'll just have to come to terms with that as computer scientists, as physics had to with dx x dp >= h. This is of course wildly unsubstantiated pessimism. Also, I would gladly entertain any argument demolishing the "local disks are not subject to CAP" claim I made above by talking about read / write caches as separate copies of the local disk state.

cagenut 15y ago

Its really fascinating to watch amazon re-learn/re-implement the lessons IBM baked into mainframes decades ago. Once you get out of shared-nothing/web-scripting land you realize that I/O is much more important and difficult than cpu. What amazon calls EBS IBM has been calling "DASD" forever. I wonder if there are any crossover lessons that they haven't taken advantage of because there just aren't any old ibm'ers working at amazon.

blantonl 15y ago
IBM's implementation of DASD on the mainframe was always implemented under the assumption that it was a secondary storage medium for data. Meaning, it wasn't accessed often, and it wasn't implemented for top performance. Think of a bridge between high performance disk and tape.

deleted 15y ago

[deleted]

johnb 15y ago

It's funny how disk abstractions get you every time. We used to store and process all of our uploads from our rails app on a GFS partition. GFS behaved like a normal disk most of the time, but we started having trouble processing concurrent uploads and couldn't replicate in dev. It turned out so GFS could work at all, it had different locking than regular disks. Every time you created a new file it had to lock the containing folder. We solved it by splitting our upload folder in 1000 sequential buckets and wrote each upload to the next folder along... but it took us a long time to stop assuming it was a regular disk.

sciurus 15y ago
FWIW, this behavior is explained early on in the documentation for GFS2.
- johnb 15y ago
  As we were using EngineYard for hosting at the time, everything was set up for us and we never thought to look it up. We now pay a lot more attention to underlying stack. Just because you've outsourced hosting (either cloud or managed physical servers), you really need to know every component yourself.

edw 15y ago

This awesome entry perfectly captures why I have always hated NFS. I can deal with the possibility that if a machine's hard drive dies, my system is going to have a very hard time continuing to operate in a normal manner, but then NFS comes along, and you realize that all sorts of I/O operations that previously employed a piece of equipment that failed once every two and a half years now depend on a working network with a working NFS server on that network, and the combination of that network and that server are orders of magnitude less reliable. And now you have situations on a regular basis where you type "ls" and you shell hangs and not even "kill -9" is going to save you. And you go back to using FTP or some other abstraction that does not apply 40,000 hour MTBF thinking to equipment that disappears for coffee breaks daily.

chubot 15y ago
A thousand times yes. My first thought when hearing about the EBS outage was "wow that seems more fragile than NFS, no wonder it failed spectacularly." NFS presents you with this nice familiar file system interface, and then random sys admins and programmers start creating a tangled mess of dependencies by dropping stuff there, without regard to what happens when it fails. Like the EBS outage, the failures tend to surprise people. The great quote by Leslie Lamport: "A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable." And this was an excellent and honest article about faulty programming abstractions. It's basically bashing you over the head with the "Fallacies of distributed computing". Don't silently turn local operations into remote operations. They're not the same thing and have to be treated differently at all levels. Even Werner Vogels wrote a diatribe against "transparency", which is the same issue by another name: http://scholar.google.com/scholar?cluster=7009698499164949721&hl=en&as_sdt=0,5 http://scholar.google.com/scholar?cluster=700969849916494972... So I wonder what he thinks of this architectural choice. You have to give up something when communicating over the network. Vogels seems to have chosen consistency rather than availability in his designs. This paper was a turning point in his research. Its candor surprised me. The file system interface does not let you relax consistency, so by default you have chosen availability. As the Joyent guys honestly remarked, this often has to be learned the hard way.
agazso 15y ago

alecco 15y ago

Amazon Six Sigma "Blackbelts", meet Mr. Black Swan. Edit: my point is you can't hide unexpected/unknown events on statistical models; we should know better, coming from CS.

CPlatypus 15y ago

He only gets it half right. A filesystem interface instead of a block interface is the right choice IMO. Private storage instead of distributed storage is the wrong choice for capacity, performance, and (most importantly) availability reasons. They didn't go with a ZFS-based solution because it was the best fit to requirements. They went with it because they had ZFS experts and advocates on staff. As Schopenhauer said, every man mistakes the limits of his own vision for the limits of the world, and these are people who've failed to Get It when it comes to distributed storage ever since they tried and failed to make ZFS distributed (leading to the enlistment of the Lustre crew who have also largely failed at the same task). If they can't solve a problem they're arrogant enough to believe nobody can, so they position DAS and SAN as the only possible alternatives. Disclaimers: I'm the project lead for CloudFS, which is IMO exactly the kind of distributed storage people should be using for this sort of thing. I've also had some fairly public disputes with Bryan "Jackass" Cantrill, formerly of Sun and now of Joyent, about ZFS FUD.

Nrsolis 15y ago
Do you have any opinions of Ceph? They seem to be dong a similar thing.
ssmoot 15y ago
ZFS is just the FS. But you know that already. The SAN solutions they migrated to are not ZFS based. Unless I'm mis-remembering (I read this a couple days ago) they were only using ZFS to slice LUNs. Point is, you're taking pot-shots at ZFS when the main thrust appears to be: "It was hard to make iSCSI reliable. Once we did, by buying expensive storage-vendor backed solutions, we found it wasn't financially compelling." They're a hosting provider. If it takes a replicated SAN pair (which is the wrong way to go about it BTW, though admittedly it's the way the storage vendors and their "appliance" mentality would have it done) to service just a pair of VM hosts (they're still using Zones right?) then it just didn't make sense money-wise for them. If they planned capacity to provide great performance, they weren't making enough money on the services for what they were selling them for. That's not an "iSCSI is unreliable" problem. It's not a "networked storage is broken" problem. It's not a "networked storage is slow" problem. It's not even a "ZFS didn't work out" problem. If you go out and spend major bucks on NetApp, not only are you going to have to deal with all the black-box-appliance BS, but it's going to cost a lot of money. A LOT. And DAS is going to end up cheaper to deploy, maintain, and your margins are going to be a lot higher. DAS is the right choice for a hosting provider who wants to maximize their profits in a competitive space. It's not the best choice for performance, availability or flexibility for clients though. So you have to ask yourself what kind of budget you have to work with, and what goals are important to you? BTW, there's _budget_, and then there's NetApp/EMC budget. Just because you need/want more than DAS can give you doesn't mean you need to tie your boat to an insane Enterprise grade budget.

lobster_johnson 15y ago

> It’s commonly believed that EBS is built on DRBD with a dose of S3-derived replication logic. Actually, it was discovered some time ago (http://openfoo.org/blog/amazon_ec2_underlying_architecture.html http://openfoo.org/blog/amazon_ec2_underlying_architecture.h...) that EBS probably used Red Hat's open-source GNDB: http://sourceware.org/cluster/gnbd/ http://sourceware.org/cluster/gnbd/