6 ms·

Great writeup. I'd love to hear other people's experience with regards to workarounds when / if EBS goes down (switching over to RDS for a short time, etc.). T

Great writeup. I'd love to hear other people's experience with regards to workarounds when / if EBS goes down (switching over to RDS for a short time, etc.). The comment about moving to local storage was interesting. Isn't the local storage on EC2 instances extremely limited (like 10-20GB?)

ktsmith 16y ago
Assuming "local storage" is synonymous with "instance storage" it's 160GB to 1690GB. http://aws.amazon.com/ec2/instance-types/ http://aws.amazon.com/ec2/instance-types/
- rgrieselhuber 16y ago
  That makes more sense, thanks.
- gregburek 16y ago
  So, I assume instances reddit uses have instance storage root volumes instead of EBS root vols. I've always assumed the persistence of ebs AMIs was a plus without a downside. Why would you opt for instance-storage AMIs instead of ebs root volume EC2 AMIs?
  - ktsmith 16y ago
    Being that EBS booting only became an option in December 2009 I would not be surprised if Reddit had not migrated their instances to that boot/storage method. They acknowledged in the last two years they hadn't even had time to move one of their databases from a single EBS volume to striped EBS volumes.
    - jedberg 16y ago
      We're currently in the process of replacing every one of our hosts with new OS versions. As we do this we are in fact going to the EBS based instances. Those instances actually show the same problems, but they aren't too bad, because once you boot them, you don't need the root vol that much (that's what the instance storage is for).
      - ktsmith 16y ago
        Thanks for the follow up jedberg, I was just guessing based on what has been publicly stated in the various blog posts over the last year or two. I used the same process for my own s3 -> ebs boot volume migration. That took a few weeks and I didn't have that many instances to migrate in the first place. Given the large number of instances reddit uses and the surprisingly small staff and one would reasonably expect that the migration would not be done.
      - gregburek 16y ago
        Thanks for the extra info! I'm doing a lot of work now with python on EC2 and the reddit write ups + presentations have been a huge help. Thanks again.
        jedberg 16y ago
        > python on EC2 #1 tip: Don't use threading. Python threading + EC2 will not work well. Instead rely on the OS doing the task switching and run multiple copies. If you want more info, I did a talk at Pycon about this and other things: redd.it/b5jyy
        gregburek 16y ago
        Duly noted. I started with this talk and have been using it as a guide to scaling edge cases with python as well as AWS. I thought raid10 was overkill before I started digging into the postgres/EBS mess, but now it seem almost routine enough that Amazon should have it as a configuration option.
      - WALoeIII 16y ago
        Did you get stuck in the Fedora 8 trap as well? It was the 'starter' in 2008/2009 and it took us two years to get off it.
        jedberg 16y ago
        Ubuntu 8.10 for us.

ktsmith 16y ago

Assuming "local storage" is synonymous with "instance storage" it's 160GB to 1690GB. http://aws.amazon.com/ec2/instance-types/ http://aws.amazon.com/ec2/instance-types/

rgrieselhuber 16y ago
That makes more sense, thanks.
gregburek 16y ago
So, I assume instances reddit uses have instance storage root volumes instead of EBS root vols. I've always assumed the persistence of ebs AMIs was a plus without a downside. Why would you opt for instance-storage AMIs instead of ebs root volume EC2 AMIs?
- ktsmith 16y ago
  Being that EBS booting only became an option in December 2009 I would not be surprised if Reddit had not migrated their instances to that boot/storage method. They acknowledged in the last two years they hadn't even had time to move one of their databases from a single EBS volume to striped EBS volumes.
  - jedberg 16y ago
    We're currently in the process of replacing every one of our hosts with new OS versions. As we do this we are in fact going to the EBS based instances. Those instances actually show the same problems, but they aren't too bad, because once you boot them, you don't need the root vol that much (that's what the instance storage is for).
    - ktsmith 16y ago
      Thanks for the follow up jedberg, I was just guessing based on what has been publicly stated in the various blog posts over the last year or two. I used the same process for my own s3 -> ebs boot volume migration. That took a few weeks and I didn't have that many instances to migrate in the first place. Given the large number of instances reddit uses and the surprisingly small staff and one would reasonably expect that the migration would not be done.
    - gregburek 16y ago
      Thanks for the extra info! I'm doing a lot of work now with python on EC2 and the reddit write ups + presentations have been a huge help. Thanks again.
      - jedberg 16y ago
        > python on EC2 #1 tip: Don't use threading. Python threading + EC2 will not work well. Instead rely on the OS doing the task switching and run multiple copies. If you want more info, I did a talk at Pycon about this and other things: redd.it/b5jyy
        gregburek 16y ago
        Duly noted. I started with this talk and have been using it as a guide to scaling edge cases with python as well as AWS. I thought raid10 was overkill before I started digging into the postgres/EBS mess, but now it seem almost routine enough that Amazon should have it as a configuration option.
    - WALoeIII 16y ago
      Did you get stuck in the Fedora 8 trap as well? It was the 'starter' in 2008/2009 and it took us two years to get off it.
      - jedberg 16y ago
        Ubuntu 8.10 for us.