6 ms·

Amazon EBS (Elastic Block Store) - persistent storage

90 points by tortilla 18y ago

maximilian 18y ago
This isn't new, but was in beta for a while right? Is this now out of beta for use by us layfolk?
- cperciva 18y ago
  I think Amazon was calling it a private "alpha" rather than a "beta"; but aside from that, yes on both points.
geuis 18y ago
been waiting for this. Woo boy, EC2 just got shiny again!
- maximilian 18y ago
  I've actually been racking my brain trying to come up with excuses to try it out and use it for something. I haven't come up with too much yet, but it seems pretty cool.
dabeeeenster 18y ago
This is the final piece of the puzzle, following on from persistent IP addresses, in terms of using EC2 as a web server platform. Time to get tinkering! A shame Amazon dont have a data centre in Europe for European firms, in terms of reducing latency...
- ovi256 18y ago
  They do have an S3 datacenter in Europe. Source: aws.amazon.com/s3 Could be mighty useful as a CDN.
  - falsestprophet 18y ago
    It is useful as a media server, but a CDN is a lot more. "CDN nodes are deployed in multiple locations, often over multiple backbones. These nodes cooperate with each other to satisfy requests for content by end users, transparently moving content to optimize the delivery process. Optimization can take the form of reducing bandwidth costs, improving end-user performance, or increasing global availability of content. The number of nodes and servers making up a CDN varies, depending on the architecture, some reaching thousands of nodes with tens of thousands of servers. Requests for content are algorithmically directed to nodes that are optimal in some way. When optimizing for performance, locations that are best for serving content to the user may be chosen. This may be measured by choosing locations that are the fewest hops or fewest number of network seconds away from the requesting client, so as to optimize delivery across local networks. When optimizing for cost, locations that are least expensive may be chosen instead. Often these two goals tend to align, as servers that are close to the end user sometimes have an advantage in serving costs, perhaps because they are located within the same network as the end user. However the value of a CDN is often demonstrated when these two goals do not align i.e. when the best performing servers and network route is located in the furthest geographic distance."
- streety 18y ago
  There might be an opportunity here. The facilities of flexiscale and amazon EC2/EBS are now comparable. A service that sat on top of the two and offered geographical redundancy could be a real winner.
EastSmith 18y ago
"I/O requests cost $0.10 per million" :) I wonder what is the amount MySql generates? Any data? Edit, as I found this (http://blog.rightscale.com/2008/08/20/amazon-ebs-explained/ http://blog.rightscale.com/2008/08/20/amazon-ebs-explained/): "As a point of reference, our main database server is pretty busy and chugs along at an average of 17 transactions per second, which should total to around $4.40 per month. But our monitoring servers, prior to some recent optimizations, hammered the disks as fast as they would go at over 1000 random writes per second sustained 24×7. That would end up costing over $250 per month! As far as I can tell, for most situations the EBS transaction costs will be in the noise, but you can make it expensive if you’re not careful."
- mseebach 18y ago
  > But our monitoring servers, prior to some recent optimizations, hammered the disks as fast as they would go at over 1000 random writes per second sustained 24×7. I don't know what that monitoring thing does, but if you make it hammer your non-persistent local storage, and then sync logs to S3, you should be fine. As I understand this service, it's only for transactional stuff that absolutely need to be there after a reboot (e.g. transactional DB), if the plug is pulled accidentially.
patrickg-zill 18y ago
"The durability of your volume depends both on the size of your volume and the percentage of the data that has changed since your last snapshot. As an example, volumes that operate with 20 GB or less of modified data since their most recent Amazon EBS snapshot can expect an annual failure rate (AFR) of between 0.1% - 0.5%, where failure refers to a complete loss of the volume. This compares with commodity hard disks that will typically fail with an AFR of around 4%, making EBS volumes 10 times more reliable than typical commodity disk drives." Sorry, this is unacceptable.
- river_styx 18y ago
  I don't see how this is different from the risks associated with any other storage service. Just be sure to have a proper backup strategy/maintenance plan.
- redorb 18y ago
  yeah I was thinking you can stack these things and almost get 99.99999% (with around 3-4 of them) of course that makes the cost more, but not 4x.
  - newt0311 18y ago
    Stacking would be the wrong way to go about it. With ZFS, you could still get 99.99999...% but with only a slight percentage increase in necessary storage space.
    - evgen 18y ago
      You will also pay for increased i/o transaction costs. TANSTAAFL.
    - rcoder 18y ago
      I don't think you read closely enough: they define the failure rate as describing the likelihood of complete loss of the volume, not minor data corruption. Unless you're mirroring your data across multiple drives, there is no way ZFS can magically recreate a volume that simply ceases to exist. Think of it this way: how would ZFS help you in your own server room if a non-mirrored disk caught fire and melted? Answer: it wouldn't. You'd go to backups, like you would with any other filesystem.
goodkarma 18y ago
I'd really like to use AWS for future Rails deployments.. Has anyone gotten a Rails app deployed using EC2/EBS/S3?
- kehunt 18y ago
  I've run http://Dibs.net http://Dibs.net on EC2 for over a year, with images served up on S3. Several servers have uptimes since I booted them in June '07. I have nothing but good things to say about my experience with AWS. If you can, I highly recommend going to the AWS Start Up Tour (http://aws.typepad.com/aws/2008/08/2008-aws-start.html http://aws.typepad.com/aws/2008/08/2008-aws-start.html) It's a good place to meet people and last year Jeff Bezos and John Doerr showed up to the one in Palo Alto. Edit: Forgot to mention - I run PostgreSQL on instance storage with log shipping and backups to S3, and a couple of MySQL DBs for Cacti and geolocation lookups. EBS will help a lot in this area, which is kind of a pain right now. Can't wait to start using it.
  - goodkarma 18y ago
    I've tried various gems and plugins and kept having issues, eventually giving up on AWS (that was several months ago). Can you offer any tips or suggestions for a successful rails deployment?
khangtoh 18y ago
I officially announce ;) this acronym for EC2+EBS so we can just say ECBS whenever we talked about EC2 with EBS.
- trezor 18y ago
  so we can just say ECBS This might reflect what I think of the whole cloud-hype, but I read that as "Easy BS". Maybe not the name you want coined for something which is supposed to be good, not shit ;)
  - LogicHoleFlaw 18y ago
    Easy Bulletproof Serving
johnrob 18y ago
This could be leveraged into a much cheaper (albeit less reliable) version of simple queue service. You can have multiple EC2 instances feeding off of the same disk used as a central repository. In general, EBS is making me think creatively about how to better architect distributed systems. SQS was both expensive and somewhat limited in scope. But now that I get disks in common, there's no restriction as to how I might partition work among EC2s.
- rapind 18y ago
  SQS got a lot cheaper, I think around Feb this year. Are you sure you aren't going by the old prices? I find it very affordable now. $0.01 per 10,000 requests $0.100 per GB in $0.170 per GB out (less with higher volume). I can't remember exactly, but I think it was only a per request cost before of around $0.10 per 1,000 or something. Also the in/out costs don't apply when they're coming from EC2, which would be the case in your scenario I believe. So from EC2 if you're doing an EC2 based distributed system, you're looking at one dollar per million requests (so for processing a message in a distributed system typically one dollar per 500,000 messages pushed then pulled). Pretty cheap by my standards. Much cheaper than it was at initial launch.

90 points by tortilla 18y ago

maximilian 18y ago

This isn't new, but was in beta for a while right? Is this now out of beta for use by us layfolk?

cperciva 18y ago
I think Amazon was calling it a private "alpha" rather than a "beta"; but aside from that, yes on both points.

geuis 18y ago

been waiting for this. Woo boy, EC2 just got shiny again!

maximilian 18y ago
I've actually been racking my brain trying to come up with excuses to try it out and use it for something. I haven't come up with too much yet, but it seems pretty cool.

dabeeeenster 18y ago

This is the final piece of the puzzle, following on from persistent IP addresses, in terms of using EC2 as a web server platform. Time to get tinkering! A shame Amazon dont have a data centre in Europe for European firms, in terms of reducing latency...

ovi256 18y ago
They do have an S3 datacenter in Europe. Source: aws.amazon.com/s3 Could be mighty useful as a CDN.
- falsestprophet 18y ago
  It is useful as a media server, but a CDN is a lot more. "CDN nodes are deployed in multiple locations, often over multiple backbones. These nodes cooperate with each other to satisfy requests for content by end users, transparently moving content to optimize the delivery process. Optimization can take the form of reducing bandwidth costs, improving end-user performance, or increasing global availability of content. The number of nodes and servers making up a CDN varies, depending on the architecture, some reaching thousands of nodes with tens of thousands of servers. Requests for content are algorithmically directed to nodes that are optimal in some way. When optimizing for performance, locations that are best for serving content to the user may be chosen. This may be measured by choosing locations that are the fewest hops or fewest number of network seconds away from the requesting client, so as to optimize delivery across local networks. When optimizing for cost, locations that are least expensive may be chosen instead. Often these two goals tend to align, as servers that are close to the end user sometimes have an advantage in serving costs, perhaps because they are located within the same network as the end user. However the value of a CDN is often demonstrated when these two goals do not align i.e. when the best performing servers and network route is located in the furthest geographic distance."
streety 18y ago
There might be an opportunity here. The facilities of flexiscale and amazon EC2/EBS are now comparable. A service that sat on top of the two and offered geographical redundancy could be a real winner.

EastSmith 18y ago

"I/O requests cost $0.10 per million" :) I wonder what is the amount MySql generates? Any data? Edit, as I found this (http://blog.rightscale.com/2008/08/20/amazon-ebs-explained/ http://blog.rightscale.com/2008/08/20/amazon-ebs-explained/): "As a point of reference, our main database server is pretty busy and chugs along at an average of 17 transactions per second, which should total to around $4.40 per month. But our monitoring servers, prior to some recent optimizations, hammered the disks as fast as they would go at over 1000 random writes per second sustained 24×7. That would end up costing over $250 per month! As far as I can tell, for most situations the EBS transaction costs will be in the noise, but you can make it expensive if you’re not careful."

mseebach 18y ago
> But our monitoring servers, prior to some recent optimizations, hammered the disks as fast as they would go at over 1000 random writes per second sustained 24×7. I don't know what that monitoring thing does, but if you make it hammer your non-persistent local storage, and then sync logs to S3, you should be fine. As I understand this service, it's only for transactional stuff that absolutely need to be there after a reboot (e.g. transactional DB), if the plug is pulled accidentially.

patrickg-zill 18y ago

"The durability of your volume depends both on the size of your volume and the percentage of the data that has changed since your last snapshot. As an example, volumes that operate with 20 GB or less of modified data since their most recent Amazon EBS snapshot can expect an annual failure rate (AFR) of between 0.1% - 0.5%, where failure refers to a complete loss of the volume. This compares with commodity hard disks that will typically fail with an AFR of around 4%, making EBS volumes 10 times more reliable than typical commodity disk drives." Sorry, this is unacceptable.

river_styx 18y ago
I don't see how this is different from the risks associated with any other storage service. Just be sure to have a proper backup strategy/maintenance plan.
redorb 18y ago
yeah I was thinking you can stack these things and almost get 99.99999% (with around 3-4 of them) of course that makes the cost more, but not 4x.
- newt0311 18y ago
  Stacking would be the wrong way to go about it. With ZFS, you could still get 99.99999...% but with only a slight percentage increase in necessary storage space.
  - evgen 18y ago
    You will also pay for increased i/o transaction costs. TANSTAAFL.
  - rcoder 18y ago
    I don't think you read closely enough: they define the failure rate as describing the likelihood of complete loss of the volume, not minor data corruption. Unless you're mirroring your data across multiple drives, there is no way ZFS can magically recreate a volume that simply ceases to exist. Think of it this way: how would ZFS help you in your own server room if a non-mirrored disk caught fire and melted? Answer: it wouldn't. You'd go to backups, like you would with any other filesystem.

goodkarma 18y ago

I'd really like to use AWS for future Rails deployments.. Has anyone gotten a Rails app deployed using EC2/EBS/S3?

kehunt 18y ago
I've run http://Dibs.net http://Dibs.net on EC2 for over a year, with images served up on S3. Several servers have uptimes since I booted them in June '07. I have nothing but good things to say about my experience with AWS. If you can, I highly recommend going to the AWS Start Up Tour (http://aws.typepad.com/aws/2008/08/2008-aws-start.html http://aws.typepad.com/aws/2008/08/2008-aws-start.html) It's a good place to meet people and last year Jeff Bezos and John Doerr showed up to the one in Palo Alto. Edit: Forgot to mention - I run PostgreSQL on instance storage with log shipping and backups to S3, and a couple of MySQL DBs for Cacti and geolocation lookups. EBS will help a lot in this area, which is kind of a pain right now. Can't wait to start using it.
- goodkarma 18y ago
  I've tried various gems and plugins and kept having issues, eventually giving up on AWS (that was several months ago). Can you offer any tips or suggestions for a successful rails deployment?

khangtoh 18y ago

I officially announce ;) this acronym for EC2+EBS so we can just say ECBS whenever we talked about EC2 with EBS.

trezor 18y ago
so we can just say ECBS This might reflect what I think of the whole cloud-hype, but I read that as "Easy BS". Maybe not the name you want coined for something which is supposed to be good, not shit ;)
- LogicHoleFlaw 18y ago
  Easy Bulletproof Serving

johnrob 18y ago

This could be leveraged into a much cheaper (albeit less reliable) version of simple queue service. You can have multiple EC2 instances feeding off of the same disk used as a central repository. In general, EBS is making me think creatively about how to better architect distributed systems. SQS was both expensive and somewhat limited in scope. But now that I get disks in common, there's no restriction as to how I might partition work among EC2s.

rapind 18y ago
SQS got a lot cheaper, I think around Feb this year. Are you sure you aren't going by the old prices? I find it very affordable now. $0.01 per 10,000 requests $0.100 per GB in $0.170 per GB out (less with higher volume). I can't remember exactly, but I think it was only a per request cost before of around $0.10 per 1,000 or something. Also the in/out costs don't apply when they're coming from EC2, which would be the case in your scenario I believe. So from EC2 if you're doing an EC2 based distributed system, you're looking at one dollar per million requests (so for processing a message in a distributed system typically one dollar per 500,000 messages pushed then pulled). Pretty cheap by my standards. Much cheaper than it was at initial launch.