3 ms·

Something seems to be off here. Their frontpage claims 45k running applications. There's no way to do that with only 22 instances. One of these figures must be

by moe 17y ago

Something seems to be off here. Their frontpage claims 45k running applications. There's no way to do that with only 22 instances. One of these figures must be wrong.

nethergoat 17y ago
You're right, that's not all they have. 22 is the number of m2.2xlarge nodes they run. There's no mention of how many other nodes they have running. That being said, given how many toy, no-traffic apps are likely running (for free!) on the platform, I think we can safely assume there's a very high degree of multi-tenancy.
anotherjesse 17y ago
m2.2xlarge are High-Memory Double Extra Large instances. 34.2 GB of memory 13 EC2 Compute Units (4 virtual cores with 3.25 EC2 Compute Units each) 850 GB of instance storage 64-bit platform I/O Performance: High Which means each application has 16MB of ram. Does chroot (or whatever jail mechanism they use) allow shared memory for standard libraries?
jamesheroku 17y ago
This article contains quite a bit of inaccurate information. Despite the size of 22 double-XL instances, they were a small portion of our overall footprint in EC2; it takes well over that kind of capacity to run the platform. Those instance types all happened to be in one availability zone for a variety of reasons. Our platform overall does not live in a single zone. Losing machines is not a problem for us (we cycle them constantly, in fact). Normally losing even that many machines would not even be noticed by our customers; this was an unusual case in which several factors cascaded into a larger problem. To be clear, this downtime (45 minutes or so, with full normal state by 90 minutes) was, unfortunately, our fault - not Amazon's. EC2 instances vaporizing is an expected part of using the service. We've made a couple of operational changes that will prevent these issues in the future, and we sincerely apologize to any customers who were affected.
mark_l_watson 17y ago
I think that most of the 45K instances are small experiments that only get a few HTTP requests per day. I would guess that in this case an instance that does not get a request for a while that it gets 'swapped out.' I have no firm data, but a customer last year put a very low volume web app on Heroku, and I guess this is exactly what we saw: the first request after a quiesant period took many seconds - then subsequent requests were quick. If I am right about this, customers with high volume web apps would not notice this.

by moe 17y ago

Something seems to be off here. Their frontpage claims 45k running applications. There's no way to do that with only 22 instances. One of these figures must be wrong.

nethergoat 17y ago

You're right, that's not all they have. 22 is the number of m2.2xlarge nodes they run. There's no mention of how many other nodes they have running. That being said, given how many toy, no-traffic apps are likely running (for free!) on the platform, I think we can safely assume there's a very high degree of multi-tenancy.

anotherjesse 17y ago

m2.2xlarge are High-Memory Double Extra Large instances. 34.2 GB of memory 13 EC2 Compute Units (4 virtual cores with 3.25 EC2 Compute Units each) 850 GB of instance storage 64-bit platform I/O Performance: High Which means each application has 16MB of ram. Does chroot (or whatever jail mechanism they use) allow shared memory for standard libraries?

jamesheroku 17y ago

This article contains quite a bit of inaccurate information. Despite the size of 22 double-XL instances, they were a small portion of our overall footprint in EC2; it takes well over that kind of capacity to run the platform. Those instance types all happened to be in one availability zone for a variety of reasons. Our platform overall does not live in a single zone. Losing machines is not a problem for us (we cycle them constantly, in fact). Normally losing even that many machines would not even be noticed by our customers; this was an unusual case in which several factors cascaded into a larger problem. To be clear, this downtime (45 minutes or so, with full normal state by 90 minutes) was, unfortunately, our fault - not Amazon's. EC2 instances vaporizing is an expected part of using the service. We've made a couple of operational changes that will prevent these issues in the future, and we sincerely apologize to any customers who were affected.

mark_l_watson 17y ago

I think that most of the 45K instances are small experiments that only get a few HTTP requests per day. I would guess that in this case an instance that does not get a request for a while that it gets 'swapped out.' I have no firm data, but a customer last year put a very low volume web app on Heroku, and I guess this is exactly what we saw: the first request after a quiesant period took many seconds - then subsequent requests were quick. If I am right about this, customers with high volume web apps would not notice this.