7 ms·

Hurricane Milton's Impact: 70 Florida Cities See 50%+ Drop in Internet Traffic

91 points by emot 2y ago

1970-01-01 2y ago
There are several DCs in the Tampa area. How are they? https://www.datacentermap.com/usa/florida/tampa/ https://www.datacentermap.com/usa/florida/tampa/
- teqsun 2y ago
  Aren't datacenters usually pretty hardened buildings? I would think they're okay but disconnected.
  - hotspot_one 2y ago
    need to get the electricity from somewhere. Need to feed the people who work there.
  - phil21 2y ago
    Honestly not really. Some of the older facilities took great care on site selection and some hardened building features, but modern sites are more based on power grid availability and are simply standard amazon distribution warehouse style buildings. The redundancy is in having your data and infrastructure in multiple buildings and geographic locations. This is due to a myriad of reasons but they all boil down to cost and practicality. The scale of modern facilities typically eclipses anything built 20 years ago by a huge margin which further limits site selection. The days of AT&T long lines are long gone. Simply driving a standard SUV through the right wall or two in most facilities would be enough to cripple them for a long while, much less flooding or a direct hit by hurricane force winds. Of course there are exceptions.
    - quesera 2y ago
      > redundancy is in having your data and infrastructure in multiple buildings aka "RAIDC" -- redundant array of inexpensive data centers. :)
- rconti 2y ago
  Wow. I didn't expect to see so many so close to the water. They might be in high-rises though.
  - dylan604 2y ago
    You could be in the basement of a high-rise though, so that's not quite as helpful. My dad was in commercial construction, and I got to visit several sites. One in particular had a basement with concrete islands dotting the floor. They were support for the computer racks to be raised so if the sprinklers were to open the racks would be above the flooding water. They however would not protect from 12'-15' storm surges
    - mlyle 2y ago
      Getting all the equipment 24 inches off the floor saves you from a pretty big fraction of events. You can often mostly protect against fairly severe events with sump pumps and flood barriers, but some water almost always gets in. If nothing important is on the ground you'll often still be OK.
      - dylan604 2y ago
        For a datacenter that does not exist on a coastline susceptible to hurricanes, you might have a point. However, we're specifically discussing a cluster of datacenters near Tampa which just missed a direct hit from a very powerful hurricane that had a forecast of a storm surge of 12'-15'. The thing to remember about storm surge is the predicted height does not include the height of wind driven waves on top of that surge. So unless your data center racks are on the 3rd floor, you are screwed
        mlyle 2y ago
        Yes, I am not saying that you can withstand any event. I have seen >3' of water next to a building with infrastructure in the ground level that were fine, though. Seepage barriers and sump pumps and elevating things a little bit can do more than you'd think. It's also worth noting that however high the storm surge is -- your infrastructure is probably at least a little above sea level.
- motrm 2y ago
  I really wanted to reply here with a link to something I recall reading from back when Katrina happened, but I can't find it. If I remember correctly it was someone called Usurper or The Usurper and they chronicled their journey at the time looking after a DC during Katrina. It may not have been a whole DC, possibly it was a business and their (smaller set of) servers, but my memory fails me. Unfortunately I can't remember what medium it was written on - a blog of some sort? Heck, it could even have been on the Something Awful forums given the year. I remember stories of struggling to find fresh sources of gas for the generator and all the fun involved in getting it from A to B. Anyone else remember that? I'd love to have another read of it now, and I think some of you might too. It'll offer a bit of insight of what's to come this year too.
  - juliansimioni 2y ago
    Yes, that was the interdictor blog! https://interdictor.livejournal.com/2005/08/27/ https://interdictor.livejournal.com/2005/08/27/
    - motrm 2y ago
      Interdictor! That's the one! Looks like the posts start at https://interdictor.livejournal.com/?skip=340 https://interdictor.livejournal.com/?skip=340 and newer ones can be found via the 'Next 10' link at the lower right. Thanks Julian!
      - juliansimioni 2y ago
        There's also the infamous bucket brigade from Hurricane Sandy. There was a Stack Overflow podcast about it but it looks like it disappears, here's a read though: https://www.datacenterdynamics.com/en/analysis/reliving-sandy-how-data-center-operators-coped-during-the-record-storm/ https://www.datacenterdynamics.com/en/analysis/reliving-sand...
  - deleted 2y ago
    [deleted]
- gottorf 2y ago
  I used to be a customer of Hivelocity with dedicated servers in their Tampa facilities. Top notch operation, never had an issue. Supposedly they remained unscathed through Milton: https://www.hivelocity.net/blog/hurricane-milton-service-notice/ https://www.hivelocity.net/blog/hurricane-milton-service-not...
staplung 2y ago
It's tempting to interpret this as a proxy for over-all damage but it's perhaps a bit unwise to do so. Alternative possibilities are that internet related infra (power lines, cell towers, roof-mounted dishes, etc.) was disproportionally damaged or that cities in the projected path saw a lot of evacuations. Of course all of these things could be true at the same time. It's an interesting way to try to assess damage in any case because it's automagically collected; you don't need to wait for people on the ground to report damage. So here's hoping that a) those affected can get back on their feet quickly and b) there's some follow-up to see how useful this data was as a proxy for assitance needs.
- mlyle 2y ago
  There's probably better figures. Internet providers themselves probably know what subscribers dropped at what times, so you'd be able to distinguish from evacuations and have a good signal as to whether it happened from power outages or key pieces of their infrastructure failing. It would be really neat to figure out the cooperation necessary to effectively use it. Rapid damage assessment has a ton of value even if the data is somewhat unreliable.
  - dotancohen 2y ago
    Evacuations are, unfortunately, often accompanied by tweets. As early as 2012 or 2014 a team I was on was able to localize the epicenter of an earthquake by examining the Twitter fire hose, half an hour before the earthquake was announced on any of the major news networks.
- ineedaj0b 2y ago
  we try not to use our phones - if the power goes out, you can't charge again for how long? You never know. another problem are power flickers. a short flicker resets modems and routers. so you read a book or talk rather than waste battery
  - dotancohen 2y ago
    > so you read a book or talk rather than waste battery OK, I'm convinced. Next hurricane I'm traveling to Florida with the kids!
deleted 2y ago
[deleted]
lysace 2y ago
Almost on topic: I have this vague memory of some very interesting blog posts of a datacenter sysadmin who bravely dealt with hurricane Katrina and the aftermath, in 2005. Anyone know what I'm talking about?
- rookie 2y ago
  https://en.wikipedia.org/wiki/Interdictor_(blog) https://en.wikipedia.org/wiki/Interdictor_(blog)
  - lysace 2y ago
    Yes! Thanks. (Unfortunately the wayback machine is down...) Nevermind: The actual livejournal posts are still up. Start here: https://interdictor.livejournal.com/2005/08/27/ https://interdictor.livejournal.com/2005/08/27/ Then just move forward day by day. (next day link in the lower right corner)
AlchemistCamp 2y ago
Starlink has been providing emergency internet service in the affected areas: https://news.ycombinator.com/item?id=41803857 https://news.ycombinator.com/item?id=41803857
- tomjakubowski 2y ago
  That article is about areas affected by Helene, not Milton
  - AlchemistCamp 2y ago
    Based on what people are posting, it appears Starlink's emergency services are continuing with Milton: https://x.com/search?q=starlink%20milton&src=typed_query https://x.com/search?q=starlink%20milton&src=typed_query
- karlzt 2y ago
  There's nothing on that link.
  - AlchemistCamp 2y ago
    It’s to a WSJ article, but apparently users flagged it.
worstspotgain 2y ago
Any non-Xitter link? Also, how much of this is less TikTok and similar?
dmitrygr 2y ago
Wouldn't you expect this even if there was no actual damage since so may people evacuated, as they were told?
joecool1029 2y ago
https://nitter.poast.org/CloudflareRadar/status/1844444286911381980 https://nitter.poast.org/CloudflareRadar/status/184444428691... (not X-walled link to thread)
emot 2y ago
An update with today's numbers and Starlink growth in Florida as well: https://x.com/CloudflareRadar/status/1844764342668976159 https://x.com/CloudflareRadar/status/1844764342668976159
eth0up 2y ago
I spent at least an hour in the eye of Milton. At one point, you could hear a leaf drop and the sky was strangely bright in all directions, though power was long gone. This was a 1:1000 miracle and could have annihilated the entire area. Had it not effectively shat itself before climbing on land, I suspect I'd be toast. Still a lot of tree and tree induced damage, but minimal major structural damages. We're forecast to be without power until the 17th and fuel lines exceed a mile, guarded by police. Not great, but beats the hell out of what was mysteriously avoided - the bastard was over 180mph at its prime and spawned more tornadoes than ever in history. Such 'luck' cannot persevere indefinitely. If interested, please see my Ask HN post regarding Google's behavior during this ordeal: https://news.ycombinator.com/item?id=41818861#41819099 https://news.ycombinator.com/item?id=41818861#41819099 Edit: Verizon cellular carried us through most of the storm, becoming inaccessible for most of the proceeding day. It has since resumed with respectable consistency.

91 points by emot 2y ago

1970-01-01 2y ago

There are several DCs in the Tampa area. How are they? https://www.datacentermap.com/usa/florida/tampa/ https://www.datacentermap.com/usa/florida/tampa/

teqsun 2y ago
Aren't datacenters usually pretty hardened buildings? I would think they're okay but disconnected.
- hotspot_one 2y ago
  need to get the electricity from somewhere. Need to feed the people who work there.
- phil21 2y ago
  Honestly not really. Some of the older facilities took great care on site selection and some hardened building features, but modern sites are more based on power grid availability and are simply standard amazon distribution warehouse style buildings. The redundancy is in having your data and infrastructure in multiple buildings and geographic locations. This is due to a myriad of reasons but they all boil down to cost and practicality. The scale of modern facilities typically eclipses anything built 20 years ago by a huge margin which further limits site selection. The days of AT&T long lines are long gone. Simply driving a standard SUV through the right wall or two in most facilities would be enough to cripple them for a long while, much less flooding or a direct hit by hurricane force winds. Of course there are exceptions.
  - quesera 2y ago
    > redundancy is in having your data and infrastructure in multiple buildings aka "RAIDC" -- redundant array of inexpensive data centers. :)
rconti 2y ago
Wow. I didn't expect to see so many so close to the water. They might be in high-rises though.
- dylan604 2y ago
  You could be in the basement of a high-rise though, so that's not quite as helpful. My dad was in commercial construction, and I got to visit several sites. One in particular had a basement with concrete islands dotting the floor. They were support for the computer racks to be raised so if the sprinklers were to open the racks would be above the flooding water. They however would not protect from 12'-15' storm surges
  - mlyle 2y ago
    Getting all the equipment 24 inches off the floor saves you from a pretty big fraction of events. You can often mostly protect against fairly severe events with sump pumps and flood barriers, but some water almost always gets in. If nothing important is on the ground you'll often still be OK.
    - dylan604 2y ago
      For a datacenter that does not exist on a coastline susceptible to hurricanes, you might have a point. However, we're specifically discussing a cluster of datacenters near Tampa which just missed a direct hit from a very powerful hurricane that had a forecast of a storm surge of 12'-15'. The thing to remember about storm surge is the predicted height does not include the height of wind driven waves on top of that surge. So unless your data center racks are on the 3rd floor, you are screwed
      - mlyle 2y ago
        Yes, I am not saying that you can withstand any event. I have seen >3' of water next to a building with infrastructure in the ground level that were fine, though. Seepage barriers and sump pumps and elevating things a little bit can do more than you'd think. It's also worth noting that however high the storm surge is -- your infrastructure is probably at least a little above sea level.
motrm 2y ago
I really wanted to reply here with a link to something I recall reading from back when Katrina happened, but I can't find it. If I remember correctly it was someone called Usurper or The Usurper and they chronicled their journey at the time looking after a DC during Katrina. It may not have been a whole DC, possibly it was a business and their (smaller set of) servers, but my memory fails me. Unfortunately I can't remember what medium it was written on - a blog of some sort? Heck, it could even have been on the Something Awful forums given the year. I remember stories of struggling to find fresh sources of gas for the generator and all the fun involved in getting it from A to B. Anyone else remember that? I'd love to have another read of it now, and I think some of you might too. It'll offer a bit of insight of what's to come this year too.
- juliansimioni 2y ago
  Yes, that was the interdictor blog! https://interdictor.livejournal.com/2005/08/27/ https://interdictor.livejournal.com/2005/08/27/
  - motrm 2y ago
    Interdictor! That's the one! Looks like the posts start at https://interdictor.livejournal.com/?skip=340 https://interdictor.livejournal.com/?skip=340 and newer ones can be found via the 'Next 10' link at the lower right. Thanks Julian!
    - juliansimioni 2y ago
      There's also the infamous bucket brigade from Hurricane Sandy. There was a Stack Overflow podcast about it but it looks like it disappears, here's a read though: https://www.datacenterdynamics.com/en/analysis/reliving-sandy-how-data-center-operators-coped-during-the-record-storm/ https://www.datacenterdynamics.com/en/analysis/reliving-sand...
- deleted 2y ago
  [deleted]
gottorf 2y ago
I used to be a customer of Hivelocity with dedicated servers in their Tampa facilities. Top notch operation, never had an issue. Supposedly they remained unscathed through Milton: https://www.hivelocity.net/blog/hurricane-milton-service-notice/ https://www.hivelocity.net/blog/hurricane-milton-service-not...

staplung 2y ago

It's tempting to interpret this as a proxy for over-all damage but it's perhaps a bit unwise to do so. Alternative possibilities are that internet related infra (power lines, cell towers, roof-mounted dishes, etc.) was disproportionally damaged or that cities in the projected path saw a lot of evacuations. Of course all of these things could be true at the same time. It's an interesting way to try to assess damage in any case because it's automagically collected; you don't need to wait for people on the ground to report damage. So here's hoping that a) those affected can get back on their feet quickly and b) there's some follow-up to see how useful this data was as a proxy for assitance needs.

mlyle 2y ago
There's probably better figures. Internet providers themselves probably know what subscribers dropped at what times, so you'd be able to distinguish from evacuations and have a good signal as to whether it happened from power outages or key pieces of their infrastructure failing. It would be really neat to figure out the cooperation necessary to effectively use it. Rapid damage assessment has a ton of value even if the data is somewhat unreliable.
- dotancohen 2y ago
  Evacuations are, unfortunately, often accompanied by tweets. As early as 2012 or 2014 a team I was on was able to localize the epicenter of an earthquake by examining the Twitter fire hose, half an hour before the earthquake was announced on any of the major news networks.
ineedaj0b 2y ago
we try not to use our phones - if the power goes out, you can't charge again for how long? You never know. another problem are power flickers. a short flicker resets modems and routers. so you read a book or talk rather than waste battery
- dotancohen 2y ago
  > so you read a book or talk rather than waste battery OK, I'm convinced. Next hurricane I'm traveling to Florida with the kids!

deleted 2y ago

[deleted]

lysace 2y ago

Almost on topic: I have this vague memory of some very interesting blog posts of a datacenter sysadmin who bravely dealt with hurricane Katrina and the aftermath, in 2005. Anyone know what I'm talking about?

rookie 2y ago
https://en.wikipedia.org/wiki/Interdictor_(blog) https://en.wikipedia.org/wiki/Interdictor_(blog)
- lysace 2y ago
  Yes! Thanks. (Unfortunately the wayback machine is down...) Nevermind: The actual livejournal posts are still up. Start here: https://interdictor.livejournal.com/2005/08/27/ https://interdictor.livejournal.com/2005/08/27/ Then just move forward day by day. (next day link in the lower right corner)

AlchemistCamp 2y ago

Starlink has been providing emergency internet service in the affected areas: https://news.ycombinator.com/item?id=41803857 https://news.ycombinator.com/item?id=41803857

tomjakubowski 2y ago
That article is about areas affected by Helene, not Milton
- AlchemistCamp 2y ago
  Based on what people are posting, it appears Starlink's emergency services are continuing with Milton: https://x.com/search?q=starlink%20milton&src=typed_query https://x.com/search?q=starlink%20milton&src=typed_query
karlzt 2y ago
There's nothing on that link.
- AlchemistCamp 2y ago
  It’s to a WSJ article, but apparently users flagged it.

worstspotgain 2y ago

Any non-Xitter link? Also, how much of this is less TikTok and similar?

dmitrygr 2y ago

Wouldn't you expect this even if there was no actual damage since so may people evacuated, as they were told?

joecool1029 2y ago

https://nitter.poast.org/CloudflareRadar/status/1844444286911381980 https://nitter.poast.org/CloudflareRadar/status/184444428691... (not X-walled link to thread)

emot 2y ago

An update with today's numbers and Starlink growth in Florida as well: https://x.com/CloudflareRadar/status/1844764342668976159 https://x.com/CloudflareRadar/status/1844764342668976159

eth0up 2y ago

I spent at least an hour in the eye of Milton. At one point, you could hear a leaf drop and the sky was strangely bright in all directions, though power was long gone. This was a 1:1000 miracle and could have annihilated the entire area. Had it not effectively shat itself before climbing on land, I suspect I'd be toast. Still a lot of tree and tree induced damage, but minimal major structural damages. We're forecast to be without power until the 17th and fuel lines exceed a mile, guarded by police. Not great, but beats the hell out of what was mysteriously avoided - the bastard was over 180mph at its prime and spawned more tornadoes than ever in history. Such 'luck' cannot persevere indefinitely. If interested, please see my Ask HN post regarding Google's behavior during this ordeal: https://news.ycombinator.com/item?id=41818861#41819099 https://news.ycombinator.com/item?id=41818861#41819099 Edit: Verizon cellular carried us through most of the storm, becoming inaccessible for most of the proceeding day. It has since resumed with respectable consistency.