4 ms·

> I’d like to know how much money really was saved by not allowing the UPSes to be cut off. I don't think this was a cost-cutting measure. Instead, (automatic)

> I’d like to know how much money really was saved by not allowing the UPSes to be cut off. I don't think this was a cost-cutting measure. Instead, (automatic) cut-off devices are also systems that can fail. And if there is such a centralized device, it can nullify the hard work of 4 redundant power supplies if it "decides" to malfunction one day. So, this was a SPOF-cutting measure. I still think that having 4 separate switches for 4 power supplies is not so wrong. Firefighters anyway wrote that the electrical room had a door at 400C and observed 1-meter arcs, so they had to ask the electrical company to cut the power from their side for the sake of safety anyway.

tssva 5y ago
I disagree with your idea that not having an automated power disconnect system is acceptable but for the sake of argument lets say it is. The standard for manual emergency power cutoff switches is to be located where they are easily accessible in an emergency which means outside of the electrical room. Most large data centers have manual switches in multiple locations. They should at a minimum have had cutoff switches located in their NOC.
- smarx007 5y ago
  Right! Sorry, I didn't write it clearly but I agree that there should be automatic disconnect, but think it's OK to have 4 separate disconnects to maintain the supply redundancy. Though now it can get really expensive.
the_arun 5y ago
Building datacenters and is hard. So this is an opportunity for cloud providers. 1. Imagine if Amazon starts building datacenters for others as a service 2. People starts leasing or renting entire DCs built by Amazon, Google etc. Maintenance is done by these cloud providers. Do we have these already?
- traceroute66 5y ago
  > 1. Imagine if Amazon starts building datacenters for others as a service Just as an FYI, those fancy datacentres you see on Amazon/Google/Microsoft marketing videos are really the exception rather than the rule. You will find the majority of cloud servers in exactly the same third-party datacentres that everyone else uses. Why ? Because only the US and perhaps one or two other countries in the world has the spare land for a cloud operator to dump a massive datacentre campus on. Most other countries don't. Or if they do, its either protected land (greenbelt etc.) or its uninhabitable (e.g. Australian Outback).
  - rjvs 5y ago
    The observation in the first half of your comment seems sound but your explanation doesn't make sense. There's rather a lot of land in the world outside of the US that could have a datacenter built on it. There's also a lot of Australia that's uninhabited but not uninhabitable.
    - traceroute66 5y ago
      I oversimplified a bit, by "land" I also meant associated infrastructure, which includes for example access to electricity. In most countries, building out new high-voltage infrastructure to where it does not exist is both financially expensive and technically painful (planning permission etc). Same for pulling fibre runs to the middle of nowhere. No doubt other things like local laws, tax rebates and whatnot also come into play as well. You may seek to argue that there are a small number of third-party datacentre sites where the cloud operator is the sole tenant of the building. But this again is not the same thing as the cloud operator building their own. They get the option to up sticks and leave at the end of their contract. They also don't have any responsibility over facilities management etc. At most sites, the cloud operator simply has whole or part of a floor (or floors in larger buildings), the rest of the building is occupied by third-party customers.
- donavanm 5y ago
  Your order of operations is inverted. The existing data center providers have been doing this for a few decades. Equinox, Dell, and friends are the ones who do the build out, management, etc. If youre big enough theyll also build to suite, though at that size you probably have a lot of in house expertise as well. AWS, Azure, Google Cloud, etc are the alternative to dealing with dedicated physical infrastructure and financing. Things like AWS Outpost are there to bridge the gap for companies/workloads that dont fit existing public cloud offerings.
mardifoufs 5y ago
The report says that the cut-off device was not present due to an economic/financial decision made by the company . It's weird to see that in a post incident report from a fire department, but I guess they thought it was worth mentioning.

tssva 5y ago

I disagree with your idea that not having an automated power disconnect system is acceptable but for the sake of argument lets say it is. The standard for manual emergency power cutoff switches is to be located where they are easily accessible in an emergency which means outside of the electrical room. Most large data centers have manual switches in multiple locations. They should at a minimum have had cutoff switches located in their NOC.

smarx007 5y ago
Right! Sorry, I didn't write it clearly but I agree that there should be automatic disconnect, but think it's OK to have 4 separate disconnects to maintain the supply redundancy. Though now it can get really expensive.

the_arun 5y ago

Building datacenters and is hard. So this is an opportunity for cloud providers. 1. Imagine if Amazon starts building datacenters for others as a service 2. People starts leasing or renting entire DCs built by Amazon, Google etc. Maintenance is done by these cloud providers. Do we have these already?

traceroute66 5y ago
> 1. Imagine if Amazon starts building datacenters for others as a service Just as an FYI, those fancy datacentres you see on Amazon/Google/Microsoft marketing videos are really the exception rather than the rule. You will find the majority of cloud servers in exactly the same third-party datacentres that everyone else uses. Why ? Because only the US and perhaps one or two other countries in the world has the spare land for a cloud operator to dump a massive datacentre campus on. Most other countries don't. Or if they do, its either protected land (greenbelt etc.) or its uninhabitable (e.g. Australian Outback).
- rjvs 5y ago
  The observation in the first half of your comment seems sound but your explanation doesn't make sense. There's rather a lot of land in the world outside of the US that could have a datacenter built on it. There's also a lot of Australia that's uninhabited but not uninhabitable.
  - traceroute66 5y ago
    I oversimplified a bit, by "land" I also meant associated infrastructure, which includes for example access to electricity. In most countries, building out new high-voltage infrastructure to where it does not exist is both financially expensive and technically painful (planning permission etc). Same for pulling fibre runs to the middle of nowhere. No doubt other things like local laws, tax rebates and whatnot also come into play as well. You may seek to argue that there are a small number of third-party datacentre sites where the cloud operator is the sole tenant of the building. But this again is not the same thing as the cloud operator building their own. They get the option to up sticks and leave at the end of their contract. They also don't have any responsibility over facilities management etc. At most sites, the cloud operator simply has whole or part of a floor (or floors in larger buildings), the rest of the building is occupied by third-party customers.
donavanm 5y ago
Your order of operations is inverted. The existing data center providers have been doing this for a few decades. Equinox, Dell, and friends are the ones who do the build out, management, etc. If youre big enough theyll also build to suite, though at that size you probably have a lot of in house expertise as well. AWS, Azure, Google Cloud, etc are the alternative to dealing with dedicated physical infrastructure and financing. Things like AWS Outpost are there to bridge the gap for companies/workloads that dont fit existing public cloud offerings.

mardifoufs 5y ago

The report says that the cut-off device was not present due to an economic/financial decision made by the company . It's weird to see that in a post incident report from a fire department, but I guess they thought it was worth mentioning.