4 ms·

I don't understand what they did the last days if this is another power outage. In their posting about the outage on the 20th they said: "At this point all we

by jrnkntl 16y ago

I don't understand what they did the last days if this is another power outage. In their posting about the outage on the 20th they said: "At this point all we know is a severe lightning storm in the area caused a power outage and redundant UPS systems failed." http://status.linode.com/2010/11/possible-power-outage-in-fremont.html http://status.linode.com/2010/11/possible-power-outage-in-fr... Redundant UPS systems failed? And now it fails again? What kind of data center are they running in Fremont?

davidw 16y ago
Wow, a "severe lightning storm" in the bay area? My friend mentioned that on facebook too. Not an every-day occurrence.
- gojomo 16y ago
  In SF proper, we saw the 'lightning', but not the 'severe'. It could have been worse in Fremont, I suppose, or that might just be a bit of puffery being used to explain the outage.
  - updog 16y ago
    Nope, I was in Fremont, and saw maybe 1 flash all night. Heard no thunder.
mrkurt 16y ago
Linode doesn't run datacenters, they colo with other providers (Hurricane Electric in Fremont). HE doesn't have the best reputation for resilient datacenter services. However, redundant power is a very complex problem and prone to failure if you can't afford to do it right... which you can't if you're selling colocation for as cheap as HE does. I really, really wish Linode would launch a sort of premium offering in better datacenters.
- jrnkntl 16y ago
  I wasn't explicitly referring to Linode but to the people running the datacenter (which is, in this case, HE). I wouldn't so much wish for a premium offering, just better datacenters in general for Linode's racks.
- davidw 16y ago
  Incidentally, in some ways it seems kind of odd to run data centers in places like Fremont, where both land and people are expensive. Google and Facebook, for instance, have demonstrated that it's quite possible to build datacenters in rural Oregon, where land is a lot cheaper, and presumably labor costs are as well, even though my guess is that in someplace like Prineville, most of their people are imports...
  - mrkurt 16y ago
    Power is expensive as well (it's usually the big expense), but there are a couple of reasons you might opt for more expensive areas for datacenters, particularly if you're selling hosting. 1) Many companies want facilities that are physically accessible to them, it's very difficult to sell to these companies if they'll have to travel 900 miles to do any work and don't trust the provider to do the work for them, or simply don't want to pay for it. Given the target market for hosting, Fremont's a pretty good option. 2) Depending on the service, a datacenter is only as useful as the transit providers you can access. We do all our hosting at a Chicago facility where we can get a cross connect to any bandwidth provider you can think of. Our services are also targeted at end users, so we're better off being centrally located and using providers with extensive peering agreements (less latency, woo!). If we had serious computation requirements rather than delivery requirements, one of those spiff Icelandic datacenters might be a good choice. I'm unsure of what Google and others stick in their rural datacenters, but I doubt it's anything that slows down your experience.
    - amock 16y ago
      Power is not the big expense and neither is land. See http://perspectives.mvdirona.com/2010/10/31/DatacenterNetworksAreInMyWay.aspx http://perspectives.mvdirona.com/2010/10/31/DatacenterNetwor... for a breakdown of expenses.
dholowiski 16y ago
I worked in a datacenter (serving many companies you've heard of) during a catastrophic power failure that lasted almost 24 hours. It's kind of like a plane crash - it's never one thing failing that causes the problem (that's what redundancy is for) - it's that perfect chain of events, multiple 'once in a lifetime' failures that causes it. For example, power outage occurs at the same time the UPS batteries are being changed. Bypass fails and Diesel generators fail to kick in. Circuit breakers blow everywhere making it extremely difficult to get the generators back on line. This all happens in the middle of the night in a winter storm (or 'lightning storm) which causes a further delay in response time. Been there, done that. Edit: Also, bureaucracy, lack of documentation, and a manager CF added several hours to the outage. Sometimes you just have to STFU and let the geeks fix the problem.
- jmcnevin 16y ago
  What you've described here would make an AWESOME movie. Granted, I'm not sure how one would work "Datacenter" into the title and still sell many tickets.
  - dholowiski 16y ago
    What makes it even more exciting is that some of the companies were a bank and a hospital. Lesson learned - a bank gets far more anxious than a hospital when their servers go down. I'm pretty sure there were Zombies roaming the aisles of servers where the power was out, that might help the box office sales.
  - rryyan 16y ago
    Cory Doctorow (of Boing Boing fame) wrote a sci-fi novelette about a datacenter during a crisis: http://craphound.com/overclocked/Cory_Doctorow_-_Overclocked_-_When_Sysadmins_Ruled_the_Earth.html http://craphound.com/overclocked/Cory_Doctorow_-_Overclocked... I personally didn't find this piece particularly great, but there's an entry in the genre for you.
- d2viant 16y ago
  Except, this isn't a once in a lifetime occurrence. As of today it's twice in a lifetime. That's indicates to me they didn't just hit that perfect chain of events, it tells me they have problems in their architecture or implementation that need to be addressed and tested before the next crisis occurs. I worked in a datacenter as well. The whole point is that you test those scenarios when it's not an emergency so you can be confident everything will operate smoothly.
deleted 16y ago
[deleted]
nenolod 16y ago
you're expecting competence from HE? i colo'd at HE until last year when they ran ~400V through my racks which had 110V circuits, several PDUs and servers were damaged. that was the cherry on top of the sundae though, there had been 2 power outages at that point and 2 other outages after that point.

by jrnkntl 16y ago

davidw 16y ago

Wow, a "severe lightning storm" in the bay area? My friend mentioned that on facebook too. Not an every-day occurrence.

gojomo 16y ago
In SF proper, we saw the 'lightning', but not the 'severe'. It could have been worse in Fremont, I suppose, or that might just be a bit of puffery being used to explain the outage.
- updog 16y ago
  Nope, I was in Fremont, and saw maybe 1 flash all night. Heard no thunder.

mrkurt 16y ago

Linode doesn't run datacenters, they colo with other providers (Hurricane Electric in Fremont). HE doesn't have the best reputation for resilient datacenter services. However, redundant power is a very complex problem and prone to failure if you can't afford to do it right... which you can't if you're selling colocation for as cheap as HE does. I really, really wish Linode would launch a sort of premium offering in better datacenters.

jrnkntl 16y ago
I wasn't explicitly referring to Linode but to the people running the datacenter (which is, in this case, HE). I wouldn't so much wish for a premium offering, just better datacenters in general for Linode's racks.
davidw 16y ago
Incidentally, in some ways it seems kind of odd to run data centers in places like Fremont, where both land and people are expensive. Google and Facebook, for instance, have demonstrated that it's quite possible to build datacenters in rural Oregon, where land is a lot cheaper, and presumably labor costs are as well, even though my guess is that in someplace like Prineville, most of their people are imports...
- mrkurt 16y ago
  Power is expensive as well (it's usually the big expense), but there are a couple of reasons you might opt for more expensive areas for datacenters, particularly if you're selling hosting. 1) Many companies want facilities that are physically accessible to them, it's very difficult to sell to these companies if they'll have to travel 900 miles to do any work and don't trust the provider to do the work for them, or simply don't want to pay for it. Given the target market for hosting, Fremont's a pretty good option. 2) Depending on the service, a datacenter is only as useful as the transit providers you can access. We do all our hosting at a Chicago facility where we can get a cross connect to any bandwidth provider you can think of. Our services are also targeted at end users, so we're better off being centrally located and using providers with extensive peering agreements (less latency, woo!). If we had serious computation requirements rather than delivery requirements, one of those spiff Icelandic datacenters might be a good choice. I'm unsure of what Google and others stick in their rural datacenters, but I doubt it's anything that slows down your experience.
  - amock 16y ago
    Power is not the big expense and neither is land. See http://perspectives.mvdirona.com/2010/10/31/DatacenterNetworksAreInMyWay.aspx http://perspectives.mvdirona.com/2010/10/31/DatacenterNetwor... for a breakdown of expenses.

dholowiski 16y ago

I worked in a datacenter (serving many companies you've heard of) during a catastrophic power failure that lasted almost 24 hours. It's kind of like a plane crash - it's never one thing failing that causes the problem (that's what redundancy is for) - it's that perfect chain of events, multiple 'once in a lifetime' failures that causes it. For example, power outage occurs at the same time the UPS batteries are being changed. Bypass fails and Diesel generators fail to kick in. Circuit breakers blow everywhere making it extremely difficult to get the generators back on line. This all happens in the middle of the night in a winter storm (or 'lightning storm) which causes a further delay in response time. Been there, done that. Edit: Also, bureaucracy, lack of documentation, and a manager CF added several hours to the outage. Sometimes you just have to STFU and let the geeks fix the problem.

jmcnevin 16y ago
What you've described here would make an AWESOME movie. Granted, I'm not sure how one would work "Datacenter" into the title and still sell many tickets.
- dholowiski 16y ago
  What makes it even more exciting is that some of the companies were a bank and a hospital. Lesson learned - a bank gets far more anxious than a hospital when their servers go down. I'm pretty sure there were Zombies roaming the aisles of servers where the power was out, that might help the box office sales.
- rryyan 16y ago
  Cory Doctorow (of Boing Boing fame) wrote a sci-fi novelette about a datacenter during a crisis: http://craphound.com/overclocked/Cory_Doctorow_-_Overclocked_-_When_Sysadmins_Ruled_the_Earth.html http://craphound.com/overclocked/Cory_Doctorow_-_Overclocked... I personally didn't find this piece particularly great, but there's an entry in the genre for you.
d2viant 16y ago
Except, this isn't a once in a lifetime occurrence. As of today it's twice in a lifetime. That's indicates to me they didn't just hit that perfect chain of events, it tells me they have problems in their architecture or implementation that need to be addressed and tested before the next crisis occurs. I worked in a datacenter as well. The whole point is that you test those scenarios when it's not an emergency so you can be confident everything will operate smoothly.

deleted 16y ago

[deleted]

nenolod 16y ago

you're expecting competence from HE? i colo'd at HE until last year when they ran ~400V through my racks which had 110V circuits, several PDUs and servers were damaged. that was the cherry on top of the sundae though, there had been 2 power outages at that point and 2 other outages after that point.