9 ms·

I've been in South Africa for the last month, which has had an energy crisis since around 2007 and somehow they manage to keep mobile networks up during load sh

by doix 4y ago

I've been in South Africa for the last month, which has had an energy crisis since around 2007 and somehow they manage to keep mobile networks up during load shedding. There are 2:30 hour (sometimes 4:30 hour) blackouts daily and yet everything seems to keep chugging along. Mobile towers are probably have some of the highest electricity:benefit ratios out there, it would be silly to not prioritize them.

tapland 4y ago
In Sweden we haven’t separated their power supply from anything else, so rolling blackouts will kill them as well. And the blackouts are probably coming sooner than a fix can be in place
- dijit 4y ago
  I actually had a problem with Swedens approach a few years ago. The Swedish power grid is a single grid and the idea is: all eggs in one basket, but pay close attention to the basket. Unfortunately when it comes to datacenter designations, two independent power feeds is required for tier 2 and above. Excluding all datacenters in Sweden. At least some portions of Canada are also excluded because they have rules that say that only hospitals are allowed to have two independent power feeds. So Ubisofts Montreal Datacenter is in the same building as a hospital...
  - gerikson 4y ago
    How does that work in for example the US? The Texas grid is separate from the rest of the continent, no?
    - lostapathy 4y ago
      You can connect to separate energy providers - such that if one provider or their transmission infrastructure goes down, you're still connected to and online with the other provider.
    - zinekeller 4y ago
      The "two separate flow" refers not the national/regional grids but instead to the specific substations, the idea being that if a substation was tripped (due to chance or for maintenance) you can still get electricity from the alternate route.
  - Gud 4y ago
    What do you mean by independent power feeds? If you mean fed by two different substations it’s not a problem in Sweden at all. Most(every station I’ve ever worked in) medium/high voltage substations in Sweden are fed by at least two separate lines. Same as any other country I’ve built substations.
    - dijit 4y ago
      That’s interesting, we visited many datacenters across Skåne and none of them said anything higher than T2 is possible due to the single power grid. Does Bahnhof and Tele2 know about your stations?
      - Gud 4y ago
        Just curious what you mean by independent power feed. If you mean fed by two separate power plants, you are right, but I don’t see how that would be possible in any country. My next work assignment literally starting in a week is building a high voltage substation for a data center(customer is well known on HN) so that’s why I’m curious
- gerikson 4y ago
  Sweden is not on the same power network as most of the rest of Europe. It's a factor in the recent price spike - Swedish companies are selling surplus power to Europe at the prices these are prepared to pay.
  - yobbo 4y ago
    "Surplus power" is not accurate. It's more accurate to say that the transfer cables are mostly at full load because the European demand is so large. In effect, this means European prices in Scandinavia.
- yobbo 4y ago
  There are different generations/eras of "towers". The ones dating from NMT and the cold war have their own little shrapnel-protected bunkers and most likely UPS:s. The latest ones which hang on every roof in cities might be turned off, but it just means phones scan until they find 2G/3G signals.
  - kalleboo 4y ago
    I remember in the several hour long power outage in Southern Sweden in 2003 the GSM network was still up
pram 4y ago
The towers have battery backups usually.
PragmaticPulp 4y ago
The less reliable the grid, the more backup power is planned into infrastructure like cell towers. In Europe the grid has historically been quite reliable, so backup options are likely more limited at these sites.
- Semaphor 4y ago
  My wife is from SA. There, every supermarket and even smaller stores have generators because blackouts are frequent. A few years ago, we had a bigger 6h or so power outage, all the stores had to throw tons of products away because there was literally no redundant power supply at all. It was my 2nd power outage in Germany in over 30 years.
- jefftk 4y ago
  Yes: this is the same as when an inch of snow shuts down a warm city and people from cold cities say "we handle worse storms several times a month". People engineer for likely conditions, and those aren't the same everywhere.
  - horsawlarway 4y ago
    Yup. And in most cases it simply doesn't make sense to spend the capital on the preparations. Ex: Atlanta gets roughly 2 inches of snow a year (usually one or two minor snowfalls) and it's far cheaper to simply shut down for a day when it happens than it is to have the infrastructure in place to handle staying open. Every 10 years or so we'll get a big snow, and be down for a week (ex: 2014) and people will freak out - but on the whole, still much cheaper to have that downtime than to be exorbitantly over-prepared for situations that are rare and not disastrous. I suspect this will be the case for the mobile outage here - on the whole, the circumstances causing the outage (lack of power) is likely not recurring, and the costs of weathering this outage with existing infra is much more favorable than being over-prepared for blackout conditions which are rare.
    - ethbr0 4y ago
      As someone who lived there for awhile, was going to mention Atlanta too. And I'd assume similar latitude cities like Birmingham, Charlotte, and Dallas? Cities operate with limited budgets. It does not make financial sense to prepare for some relatively rare events, versus allocating that money to better prepare for likely events. If Atlanta had a fleet of snowplows, they'd be unused 75% of winters and used for only a couple days in 20%. Bad investment.
crote 4y ago
To be fair, if you experience blackouts that often your infrastructure is probably prepared for it. Most of Europe basically never has blackouts, so why spend money on backup batteries when they will be deprecated before they'll even be used once?
- beached_whale 4y ago
  Canada/US learnt that lesson back in 2003 with the Northeast Blackout. Generators on cell towers where not fueled and some data centre’s caught fire when the cooling went down but the systems didn’t. Lessons where learned. Cell infra is strategic and should not be allowed to go down, everything uses it and people will die if it goes down.
  - sbierwagen 4y ago
    I'm in a Seattle suburb and we had a two-day blackout a few years back. Cell reception worked fine for 12 hours but then went to 1 bar. Presumably the closest cell site ran out of gas on the backup generator and my phone switched to a site outside the blackout zone. What is wild was that this was entirely localized! A single substation affecting maybe 20,000 people. How many cell sites were affected, a dozen? And still T-Mobile couldn't manage refueling. In a country-wide crisis we would be totally boned.
  - thanatos519 4y ago
    Indeed! In Toronto, the cooling for 151 Front Street West was grid-powered electric pumps pulling cold water from deep in Lake Ontario... so despite all the fuel they had ready to run the systems, they couldn't use it and had to shut the building down.
    - beached_whale 4y ago
      i heard stories of cabinet fires that day on front.
  - 29083011397778 4y ago
    > Cell infra is strategic and should not be allowed to go down, everything uses it and people will die if it goes down. > Lessons were learned. That's why our emergency services use Rogers as the main, and Telus as a backup - but only after the multi-day Rogers outage that took out payment processors and emergency services this year. Lessons will have been learned instead of parroted when we have a crown corporation as the main, and a private corporation as the backup. We need cell service to be reliable and rock-solid, even if it's not always cost-competitive to be so. Same reason hospitals, fire, and police services are public: turning a profit is less important than availability.
ryanjshaw 4y ago
It's unfortunately not that simple. I'm South African. We have load shedding "stages", where each stage sheds 1GW from the grid. We were just at stage 6 recently, and this means there are days with 1x2.5hr outages and 2x4.5hr outages, with maybe 2-3hrs between these stages to recharge batteries. In my experience the cellphone towers do not always have sufficient time to recharge their batteries. I live on an estate with a cellphone tower owned by the largest network operator, and this is still an issue. SMS-based MFA becomes a problem when this happens. Interestingly, I have never once had a fibre outage through years of load shedding. (I have battery backup for my fibre CPE.) I can easily get 200mbps to a local DC while people relying on 4G or 5G are completely cut off.
bombcar 4y ago
If everything is tied to the "same grid" you may not be able to shut off only specific classes of customers. If the grid is designed for unreliability, you probably have the ability to order load sheds of various levels, so you can cut power to houses without cutting streetlights and cell towers. Alternatively, you could have it built into smart meters.

by doix 4y ago

tapland 4y ago

In Sweden we haven’t separated their power supply from anything else, so rolling blackouts will kill them as well. And the blackouts are probably coming sooner than a fix can be in place

dijit 4y ago
I actually had a problem with Swedens approach a few years ago. The Swedish power grid is a single grid and the idea is: all eggs in one basket, but pay close attention to the basket. Unfortunately when it comes to datacenter designations, two independent power feeds is required for tier 2 and above. Excluding all datacenters in Sweden. At least some portions of Canada are also excluded because they have rules that say that only hospitals are allowed to have two independent power feeds. So Ubisofts Montreal Datacenter is in the same building as a hospital...
- gerikson 4y ago
  How does that work in for example the US? The Texas grid is separate from the rest of the continent, no?
  - lostapathy 4y ago
    You can connect to separate energy providers - such that if one provider or their transmission infrastructure goes down, you're still connected to and online with the other provider.
  - zinekeller 4y ago
    The "two separate flow" refers not the national/regional grids but instead to the specific substations, the idea being that if a substation was tripped (due to chance or for maintenance) you can still get electricity from the alternate route.
- Gud 4y ago
  What do you mean by independent power feeds? If you mean fed by two different substations it’s not a problem in Sweden at all. Most(every station I’ve ever worked in) medium/high voltage substations in Sweden are fed by at least two separate lines. Same as any other country I’ve built substations.
  - dijit 4y ago
    That’s interesting, we visited many datacenters across Skåne and none of them said anything higher than T2 is possible due to the single power grid. Does Bahnhof and Tele2 know about your stations?
    - Gud 4y ago
      Just curious what you mean by independent power feed. If you mean fed by two separate power plants, you are right, but I don’t see how that would be possible in any country. My next work assignment literally starting in a week is building a high voltage substation for a data center(customer is well known on HN) so that’s why I’m curious
gerikson 4y ago
Sweden is not on the same power network as most of the rest of Europe. It's a factor in the recent price spike - Swedish companies are selling surplus power to Europe at the prices these are prepared to pay.
- yobbo 4y ago
  "Surplus power" is not accurate. It's more accurate to say that the transfer cables are mostly at full load because the European demand is so large. In effect, this means European prices in Scandinavia.
yobbo 4y ago
There are different generations/eras of "towers". The ones dating from NMT and the cold war have their own little shrapnel-protected bunkers and most likely UPS:s. The latest ones which hang on every roof in cities might be turned off, but it just means phones scan until they find 2G/3G signals.
- kalleboo 4y ago
  I remember in the several hour long power outage in Southern Sweden in 2003 the GSM network was still up

pram 4y ago

The towers have battery backups usually.

PragmaticPulp 4y ago

The less reliable the grid, the more backup power is planned into infrastructure like cell towers. In Europe the grid has historically been quite reliable, so backup options are likely more limited at these sites.

Semaphor 4y ago
My wife is from SA. There, every supermarket and even smaller stores have generators because blackouts are frequent. A few years ago, we had a bigger 6h or so power outage, all the stores had to throw tons of products away because there was literally no redundant power supply at all. It was my 2nd power outage in Germany in over 30 years.
jefftk 4y ago
Yes: this is the same as when an inch of snow shuts down a warm city and people from cold cities say "we handle worse storms several times a month". People engineer for likely conditions, and those aren't the same everywhere.
- horsawlarway 4y ago
  Yup. And in most cases it simply doesn't make sense to spend the capital on the preparations. Ex: Atlanta gets roughly 2 inches of snow a year (usually one or two minor snowfalls) and it's far cheaper to simply shut down for a day when it happens than it is to have the infrastructure in place to handle staying open. Every 10 years or so we'll get a big snow, and be down for a week (ex: 2014) and people will freak out - but on the whole, still much cheaper to have that downtime than to be exorbitantly over-prepared for situations that are rare and not disastrous. I suspect this will be the case for the mobile outage here - on the whole, the circumstances causing the outage (lack of power) is likely not recurring, and the costs of weathering this outage with existing infra is much more favorable than being over-prepared for blackout conditions which are rare.
  - ethbr0 4y ago
    As someone who lived there for awhile, was going to mention Atlanta too. And I'd assume similar latitude cities like Birmingham, Charlotte, and Dallas? Cities operate with limited budgets. It does not make financial sense to prepare for some relatively rare events, versus allocating that money to better prepare for likely events. If Atlanta had a fleet of snowplows, they'd be unused 75% of winters and used for only a couple days in 20%. Bad investment.

crote 4y ago

To be fair, if you experience blackouts that often your infrastructure is probably prepared for it. Most of Europe basically never has blackouts, so why spend money on backup batteries when they will be deprecated before they'll even be used once?

beached_whale 4y ago
Canada/US learnt that lesson back in 2003 with the Northeast Blackout. Generators on cell towers where not fueled and some data centre’s caught fire when the cooling went down but the systems didn’t. Lessons where learned. Cell infra is strategic and should not be allowed to go down, everything uses it and people will die if it goes down.
- sbierwagen 4y ago
  I'm in a Seattle suburb and we had a two-day blackout a few years back. Cell reception worked fine for 12 hours but then went to 1 bar. Presumably the closest cell site ran out of gas on the backup generator and my phone switched to a site outside the blackout zone. What is wild was that this was entirely localized! A single substation affecting maybe 20,000 people. How many cell sites were affected, a dozen? And still T-Mobile couldn't manage refueling. In a country-wide crisis we would be totally boned.
- thanatos519 4y ago
  Indeed! In Toronto, the cooling for 151 Front Street West was grid-powered electric pumps pulling cold water from deep in Lake Ontario... so despite all the fuel they had ready to run the systems, they couldn't use it and had to shut the building down.
  - beached_whale 4y ago
    i heard stories of cabinet fires that day on front.
- 29083011397778 4y ago
  > Cell infra is strategic and should not be allowed to go down, everything uses it and people will die if it goes down. > Lessons were learned. That's why our emergency services use Rogers as the main, and Telus as a backup - but only after the multi-day Rogers outage that took out payment processors and emergency services this year. Lessons will have been learned instead of parroted when we have a crown corporation as the main, and a private corporation as the backup. We need cell service to be reliable and rock-solid, even if it's not always cost-competitive to be so. Same reason hospitals, fire, and police services are public: turning a profit is less important than availability.

ryanjshaw 4y ago

It's unfortunately not that simple. I'm South African. We have load shedding "stages", where each stage sheds 1GW from the grid. We were just at stage 6 recently, and this means there are days with 1x2.5hr outages and 2x4.5hr outages, with maybe 2-3hrs between these stages to recharge batteries. In my experience the cellphone towers do not always have sufficient time to recharge their batteries. I live on an estate with a cellphone tower owned by the largest network operator, and this is still an issue. SMS-based MFA becomes a problem when this happens. Interestingly, I have never once had a fibre outage through years of load shedding. (I have battery backup for my fibre CPE.) I can easily get 200mbps to a local DC while people relying on 4G or 5G are completely cut off.

bombcar 4y ago

If everything is tied to the "same grid" you may not be able to shut off only specific classes of customers. If the grid is designed for unreliability, you probably have the ability to order load sheds of various levels, so you can cut power to houses without cutting streetlights and cell towers. Alternatively, you could have it built into smart meters.