7 ms·

US Weather Service Internet Systems Are Crumbling

56 points by julienchastang 6y ago

kevin_thibedeau 6y ago
Accuweather: "I love it when a plan comes together"
burnished 6y ago
Why no mention of third parties that kneecap important infrastructure to serve their own profit margin?
readflaggedcomm 6y ago
Not much detail on decision making. But bandwidth and overtaxed servers are blamed throughout this and the other two articles linked near the top, and their bespoke chat and radar programs seem to make it worse. When will they distribute observation data through some swarmable system like IPFS? Yet even the rogue Alabama office recommended siloed Slack rather than something open.
- Jtsummers 6y ago
  My wager is that NWS is like every other gov't agency. IT is a necessary evil, not a core focus. They do enough to keep it going but until shit hits the fan it won't get enough attention from their leadership to receive the funding (for staff or systems) necessary to actually grow or even properly sustain it (that is, actually replace problem components in servers or properly update/upgrade systems, or properly convert to even minimal degrees of virtualization to allow mobility of services). The ironic/annoying part is that some agencies have figured out how to do it correctly, but they never share with other agencies. Either out of selfishness (fiefdoms, it's mine and I get the accolades for having it, if I shared then it would be ours and I'd have to share those accolades, screw you) or lack of communication (silos, they don't know what anyone else is doing and may not have any good channels of communication to find out). This same thing gets played out over and over again over the years, and it's not just with the government. It's any group that views what's actually their critical infrastructure as a cost center to minimize spending on and, ideally, cut; and any organization that creates overly siloed groups and forces communication up and down a hierarchy rather than direct communication between peer groups. > When will they distribute observation data through some swarmable system like IPFS? Yet even the rogue Alabama office recommended siloed Slack rather than something open. I think I'd also be comfortable making a multi-hundred dollar wager that they've heard of Slack and never heard of IPFS or similar approaches.
  - tablespoon 6y ago
    > My wager is that NWS is like every other gov't agency. IT is a necessary evil, not a core focus. They do enough to keep it going but until shit hits the fan it won't get enough attention from their leadership to receive the funding (for staff or systems) necessary to actually grow or even properly sustain it Not just every other government agency, but every other non-tech organization.
    - Jtsummers 6y ago
      Agreed, see my second paragraph above. Organizations suck at maintaining the infrastructure that they don't see as their core mission or their profit centers (and even then...). IT is one such thing for most businesses and governments. They don't appreciate how integral it is until it fails, then there's a lot of wailing and gnashing of teeth and Congressional hearings.
      - yellowapple 6y ago
        If only organizations would treat things like IT, customer service, operations, fulfillment, logistics, etc. less as cost centers and more as results multipliers.
yellowapple 6y ago
In the vein of "ask not what your country can do for you, but what you can do for your country": how can we help? If the government is (evidently) unable or unwilling to operate these services, it seems like it is our civil duty as Americans to take this upon ourselves and coordinate the creation and operation of a system "by the people, of the people, for the people".
- drewda 6y ago
  The private sector has been trying for years to reduce the National Weather Service's resources. Perhaps that's not the entire story of why NWS has IT constraints right now. But it's still pretty frustrating how Accuweather keeps trying to limit what NWS can distribute for free, because they would rather sell it to the American public. Here's an oldie but a goodie: https://en.wikipedia.org/wiki/National_Weather_Service_Duties_Act_of_2005 https://en.wikipedia.org/wiki/National_Weather_Service_Dutie...
  - throw0101a 6y ago
    John Oliver covered the topic in 2019: * https://www.youtube.com/watch?v=qMGn9T37eR8 https://www.youtube.com/watch?v=qMGn9T37eR8 * https://time.com/5699545/john-oliver-weather-last-week-tonight/ https://time.com/5699545/john-oliver-weather-last-week-tonig... * https://www.theguardian.com/culture/2019/oct/14/john-oliver-last-week-tonight-recap-national-weather-service https://www.theguardian.com/culture/2019/oct/14/john-oliver-...
- 1996 6y ago
  > "ask not what your country can do for you, but what you can do for your country" No. You can and should ask as 1) it is at the basis of check and balances 2) it is not an connection that can not be severed if needed. If you are not willing to look at the dynamic of the relationship, you may help a country commit tyranny: if your country is not willing to help you because of your race / religion / anything else, maybe you should do nothing in return except get the fuck out and grab a bag of popcorn while watching the country go sour. I say that as I love China, but I understand and support my Gay friends who have decided to never return.
kcmastrpc 6y ago
I've used weather.gov for almost 10 years, and I've observed is become more unstable and unusable over the past decade. If it weren't for local weather gurus who pay for the tools professional meteorologists utilize and broadcast through YT and other social media tools I wouldn't know when to shelter or take cover from some of the intense weather that has moved through our area over the past few years.
CWuestefeld 6y ago
Why are they so attached to NWS Chat specifically? Perhaps there are reasons unexplained, but from this article it sounds very much like they've got Not Invented Here syndrome. Sure, they need bespoke computation services for their modeling functions and so forth. But when it comes to the dissemination of information, it seems like that's very much a commodity these days, and can be provided at such an enormous scale at such low prices that it should scarcely be a concern. I suspect, with no evidence whatsoever, that the NWS is holding this hostage for better funding overall, because it's the visible piece that causes pain to the public. There are other areas where they believe they really need better funding, but haven't been able to convince the budgetmakers of that. In order to force their hand, they're starving the public-facing stuff.
- Jtsummers 6y ago
  NWSChat is apparently an XMPP-based solution and people were able to use off-the-shelf clients (did some digging out of curiosity). It was also started before things like chat-as-a-service offerings (like Slack) were widespread. It's also import to remember that they needed to coordinate with both government and non-government users, so a solution like MS's chat (what later became Skype for Business, not sure what it was in the 00s) would be challenging to integrate with non-government users even if it was trivially deployed and scaled across the government side. On top of that, using a commercial service isn't necessarily permissible as government agencies have to track their communications and record them with strict rules on how long recordings have to be maintained and how/when they can be destroyed. Commercial services can be used, but they have to have the right policies in place. WFH in 2020 revealed this issue to many as gov't employees were using non-standard tools without proper recording/tracing that technically violated federal laws and regulations but let them get their job done. For better or worse, in the 00s self-hosting a chat service might have been the least bad option to achieve all the requirements they had. However, it also probably suffered (as I mentioned in another comment) from the fact that many gov't agencies don't really have proper IT staffing and don't treat it as something to continually invest in and maintain/sustain.
  - CWuestefeld 6y ago
johnnyapol 6y ago
Mods - looks like SeattleTimes is syndicating this and lacks the aggressive paywall. Here's an alternative link: https://www.seattletimes.com/nation-world/nation/weather-service-internet-systems-crumbling-as-key-platforms-are-taxed-and-fail/ https://www.seattletimes.com/nation-world/nation/weather-ser...
EMM_386 6y ago
I just tried out the new weather radar, and it was a mess. Extremely slow to update new tiles when scrolling around or zooming. Trying to "play" the radar had a 15 second delay. The browser console is full of errors. And then, after a couple minutes of moving around the map, I can no longer access the site. Is this the "data limiting" the article mentions?? > Access Denied > You don't have permission to access "http://radar.weather.gov/ http://radar.weather.gov/" on this server. > Reference #18.95fa3b17.1617393226.1c0a631b
- julienchastang 6y ago
  There was a WaPo article about this too [1]. I feel bad for the software developers who had to work on this website. They were probably understaffed and under-resourced and had to release something they knew was not working. Edit: Note that the old radar.weather.gov was great and well-liked, but they had to get rid of it due to its reliance on Flash. [1] https://www.washingtonpost.com/weather/2020/12/21/nws-new-radar-website-slow/ https://www.washingtonpost.com/weather/2020/12/21/nws-new-ra...
  - maxerickson 6y ago
    There were gifs. Oh the gifs. The animated loop for a radar station would load in ¼ second. (this is the radar lite that they could have left up without flash)
    - superkuh 6y ago
      I miss them too. Try out this https://s.w-x.co/staticmaps/wu/wu/wxtype1200_cur/conus/animate.png https://s.w-x.co/staticmaps/wu/wu/wxtype1200_cur/conus/anima... It's the best simple animated CONUS weather radar gif I could find with a static URL.
malwarebytess 6y ago
I wonder if it has something to do with the active sabotage in leadership over the last 4 years.
edoceo 6y ago
Well, Mattermost could/would solve that chat problem nearly instantly and cost nearly nothing.
- Jtsummers 6y ago
  Except the cost of deploying, integrating into their authentication system, and maintaining. In 10-15 years a self-hosted solution by a non-IT-centric organization will probably end up in the same situation as their current solution. The US government needs to get better at distributing IT knowledge, capabilities, and services across its numerous agencies instead of having them each (and their subcomponents) act like their own small businesses with little effort on cross-cutting support and capabilities.
nattaylor 6y ago
After the radar launch I inquired about bringing back the GIF loops and they had something within 24 hours. I use the NWS API's in my personal weather app and they work fairly well. It's not all doom and gloom, but they could be better. The NDBC went down for about 7 days last month due to a "catastrophic outage," and that was very unfortunate. But, the deliberately and with warning took the BOX radar offline for maintenance for the first time in decades. A mixed bag.
rckoepke 6y ago
I really miss the old weatherspark. It was extremely performant and also had by far the best/most useful UI visualizations of any weather app to date. http://radar.oreilly.com/2011/09/weatherspark-weather-data-visualization.html http://radar.oreilly.com/2011/09/weatherspark-weather-data-v... https://kk.org/cooltools/weatherspark/ https://kk.org/cooltools/weatherspark/ As I understand it, they didn't have the money to migrate from Flash to HTML5 and the current site is a shadow of it's former self.
8bitsrule 6y ago
A couple months back, I was surpised to see that the radar site's old Windows98-refugee map was gone ... replaced by a slithering mess of SVC-looking intentions gone AI. But when I checked it a week ago, it was much less shaky. Good to see the NWS trying new ideas.
- maxerickson 6y ago
  The old site had good urls and loaded really fast. Those are way more important for an information service than trying new things. And if you want to try new things, you can leave the old one around while you do it.
superkuh 6y ago
I contacted the new NOAA weather radar site team numerous times over a couple years as they began testing the new site. They were completely unresponsive to all input from me or anyone else. It would be so simple to just maintain the animated gif CONUS weather radar map in addition to fancy cloud javascript applications, but no, they went ahead and "replaced" it with a javascript monstrosity that doesn't even work for half the users, myself included. I gave up even trying to get a response when their entire @noaa.gov email system went down due to the solar winds hack and them proxying all their mail through fireeye.gov. This caused all spf checks to fail for any mail coming from anywhere except within google's email empire. Being forced into the cloud is what killed NOAA. They could've upgraded from their 2005 setup to something a bit more modern but instead they migrated to the cloud and it ruined everything.

kevin_thibedeau 6y ago

Accuweather: "I love it when a plan comes together"

burnished 6y ago

Why no mention of third parties that kneecap important infrastructure to serve their own profit margin?

readflaggedcomm 6y ago

Not much detail on decision making. But bandwidth and overtaxed servers are blamed throughout this and the other two articles linked near the top, and their bespoke chat and radar programs seem to make it worse. When will they distribute observation data through some swarmable system like IPFS? Yet even the rogue Alabama office recommended siloed Slack rather than something open.

Jtsummers 6y ago
My wager is that NWS is like every other gov't agency. IT is a necessary evil, not a core focus. They do enough to keep it going but until shit hits the fan it won't get enough attention from their leadership to receive the funding (for staff or systems) necessary to actually grow or even properly sustain it (that is, actually replace problem components in servers or properly update/upgrade systems, or properly convert to even minimal degrees of virtualization to allow mobility of services). The ironic/annoying part is that some agencies have figured out how to do it correctly, but they never share with other agencies. Either out of selfishness (fiefdoms, it's mine and I get the accolades for having it, if I shared then it would be ours and I'd have to share those accolades, screw you) or lack of communication (silos, they don't know what anyone else is doing and may not have any good channels of communication to find out). This same thing gets played out over and over again over the years, and it's not just with the government. It's any group that views what's actually their critical infrastructure as a cost center to minimize spending on and, ideally, cut; and any organization that creates overly siloed groups and forces communication up and down a hierarchy rather than direct communication between peer groups. > When will they distribute observation data through some swarmable system like IPFS? Yet even the rogue Alabama office recommended siloed Slack rather than something open. I think I'd also be comfortable making a multi-hundred dollar wager that they've heard of Slack and never heard of IPFS or similar approaches.
- tablespoon 6y ago
  > My wager is that NWS is like every other gov't agency. IT is a necessary evil, not a core focus. They do enough to keep it going but until shit hits the fan it won't get enough attention from their leadership to receive the funding (for staff or systems) necessary to actually grow or even properly sustain it Not just every other government agency, but every other non-tech organization.
  - Jtsummers 6y ago
    Agreed, see my second paragraph above. Organizations suck at maintaining the infrastructure that they don't see as their core mission or their profit centers (and even then...). IT is one such thing for most businesses and governments. They don't appreciate how integral it is until it fails, then there's a lot of wailing and gnashing of teeth and Congressional hearings.
    - yellowapple 6y ago
      If only organizations would treat things like IT, customer service, operations, fulfillment, logistics, etc. less as cost centers and more as results multipliers.

yellowapple 6y ago

In the vein of "ask not what your country can do for you, but what you can do for your country": how can we help? If the government is (evidently) unable or unwilling to operate these services, it seems like it is our civil duty as Americans to take this upon ourselves and coordinate the creation and operation of a system "by the people, of the people, for the people".

drewda 6y ago
The private sector has been trying for years to reduce the National Weather Service's resources. Perhaps that's not the entire story of why NWS has IT constraints right now. But it's still pretty frustrating how Accuweather keeps trying to limit what NWS can distribute for free, because they would rather sell it to the American public. Here's an oldie but a goodie: https://en.wikipedia.org/wiki/National_Weather_Service_Duties_Act_of_2005 https://en.wikipedia.org/wiki/National_Weather_Service_Dutie...
- throw0101a 6y ago
  John Oliver covered the topic in 2019: * https://www.youtube.com/watch?v=qMGn9T37eR8 https://www.youtube.com/watch?v=qMGn9T37eR8 * https://time.com/5699545/john-oliver-weather-last-week-tonight/ https://time.com/5699545/john-oliver-weather-last-week-tonig... * https://www.theguardian.com/culture/2019/oct/14/john-oliver-last-week-tonight-recap-national-weather-service https://www.theguardian.com/culture/2019/oct/14/john-oliver-...
1996 6y ago
> "ask not what your country can do for you, but what you can do for your country" No. You can and should ask as 1) it is at the basis of check and balances 2) it is not an connection that can not be severed if needed. If you are not willing to look at the dynamic of the relationship, you may help a country commit tyranny: if your country is not willing to help you because of your race / religion / anything else, maybe you should do nothing in return except get the fuck out and grab a bag of popcorn while watching the country go sour. I say that as I love China, but I understand and support my Gay friends who have decided to never return.

kcmastrpc 6y ago

I've used weather.gov for almost 10 years, and I've observed is become more unstable and unusable over the past decade. If it weren't for local weather gurus who pay for the tools professional meteorologists utilize and broadcast through YT and other social media tools I wouldn't know when to shelter or take cover from some of the intense weather that has moved through our area over the past few years.

CWuestefeld 6y ago

Why are they so attached to NWS Chat specifically? Perhaps there are reasons unexplained, but from this article it sounds very much like they've got Not Invented Here syndrome. Sure, they need bespoke computation services for their modeling functions and so forth. But when it comes to the dissemination of information, it seems like that's very much a commodity these days, and can be provided at such an enormous scale at such low prices that it should scarcely be a concern. I suspect, with no evidence whatsoever, that the NWS is holding this hostage for better funding overall, because it's the visible piece that causes pain to the public. There are other areas where they believe they really need better funding, but haven't been able to convince the budgetmakers of that. In order to force their hand, they're starving the public-facing stuff.

Jtsummers 6y ago
NWSChat is apparently an XMPP-based solution and people were able to use off-the-shelf clients (did some digging out of curiosity). It was also started before things like chat-as-a-service offerings (like Slack) were widespread. It's also import to remember that they needed to coordinate with both government and non-government users, so a solution like MS's chat (what later became Skype for Business, not sure what it was in the 00s) would be challenging to integrate with non-government users even if it was trivially deployed and scaled across the government side. On top of that, using a commercial service isn't necessarily permissible as government agencies have to track their communications and record them with strict rules on how long recordings have to be maintained and how/when they can be destroyed. Commercial services can be used, but they have to have the right policies in place. WFH in 2020 revealed this issue to many as gov't employees were using non-standard tools without proper recording/tracing that technically violated federal laws and regulations but let them get their job done. For better or worse, in the 00s self-hosting a chat service might have been the least bad option to achieve all the requirements they had. However, it also probably suffered (as I mentioned in another comment) from the fact that many gov't agencies don't really have proper IT staffing and don't treat it as something to continually invest in and maintain/sustain.
- CWuestefeld 6y ago

johnnyapol 6y ago

Mods - looks like SeattleTimes is syndicating this and lacks the aggressive paywall. Here's an alternative link: https://www.seattletimes.com/nation-world/nation/weather-service-internet-systems-crumbling-as-key-platforms-are-taxed-and-fail/ https://www.seattletimes.com/nation-world/nation/weather-ser...

EMM_386 6y ago

I just tried out the new weather radar, and it was a mess. Extremely slow to update new tiles when scrolling around or zooming. Trying to "play" the radar had a 15 second delay. The browser console is full of errors. And then, after a couple minutes of moving around the map, I can no longer access the site. Is this the "data limiting" the article mentions?? > Access Denied > You don't have permission to access "http://radar.weather.gov/ http://radar.weather.gov/" on this server. > Reference #18.95fa3b17.1617393226.1c0a631b

julienchastang 6y ago
There was a WaPo article about this too [1]. I feel bad for the software developers who had to work on this website. They were probably understaffed and under-resourced and had to release something they knew was not working. Edit: Note that the old radar.weather.gov was great and well-liked, but they had to get rid of it due to its reliance on Flash. [1] https://www.washingtonpost.com/weather/2020/12/21/nws-new-radar-website-slow/ https://www.washingtonpost.com/weather/2020/12/21/nws-new-ra...
- maxerickson 6y ago
  There were gifs. Oh the gifs. The animated loop for a radar station would load in ¼ second. (this is the radar lite that they could have left up without flash)
  - superkuh 6y ago
    I miss them too. Try out this https://s.w-x.co/staticmaps/wu/wu/wxtype1200_cur/conus/animate.png https://s.w-x.co/staticmaps/wu/wu/wxtype1200_cur/conus/anima... It's the best simple animated CONUS weather radar gif I could find with a static URL.

malwarebytess 6y ago

I wonder if it has something to do with the active sabotage in leadership over the last 4 years.

edoceo 6y ago

Well, Mattermost could/would solve that chat problem nearly instantly and cost nearly nothing.

Jtsummers 6y ago
Except the cost of deploying, integrating into their authentication system, and maintaining. In 10-15 years a self-hosted solution by a non-IT-centric organization will probably end up in the same situation as their current solution. The US government needs to get better at distributing IT knowledge, capabilities, and services across its numerous agencies instead of having them each (and their subcomponents) act like their own small businesses with little effort on cross-cutting support and capabilities.

nattaylor 6y ago

After the radar launch I inquired about bringing back the GIF loops and they had something within 24 hours. I use the NWS API's in my personal weather app and they work fairly well. It's not all doom and gloom, but they could be better. The NDBC went down for about 7 days last month due to a "catastrophic outage," and that was very unfortunate. But, the deliberately and with warning took the BOX radar offline for maintenance for the first time in decades. A mixed bag.

rckoepke 6y ago

I really miss the old weatherspark. It was extremely performant and also had by far the best/most useful UI visualizations of any weather app to date. http://radar.oreilly.com/2011/09/weatherspark-weather-data-visualization.html http://radar.oreilly.com/2011/09/weatherspark-weather-data-v... https://kk.org/cooltools/weatherspark/ https://kk.org/cooltools/weatherspark/ As I understand it, they didn't have the money to migrate from Flash to HTML5 and the current site is a shadow of it's former self.

8bitsrule 6y ago

A couple months back, I was surpised to see that the radar site's old Windows98-refugee map was gone ... replaced by a slithering mess of SVC-looking intentions gone AI. But when I checked it a week ago, it was much less shaky. Good to see the NWS trying new ideas.

maxerickson 6y ago
The old site had good urls and loaded really fast. Those are way more important for an information service than trying new things. And if you want to try new things, you can leave the old one around while you do it.

superkuh 6y ago

I contacted the new NOAA weather radar site team numerous times over a couple years as they began testing the new site. They were completely unresponsive to all input from me or anyone else. It would be so simple to just maintain the animated gif CONUS weather radar map in addition to fancy cloud javascript applications, but no, they went ahead and "replaced" it with a javascript monstrosity that doesn't even work for half the users, myself included. I gave up even trying to get a response when their entire @noaa.gov email system went down due to the solar winds hack and them proxying all their mail through fireeye.gov. This caused all spf checks to fail for any mail coming from anywhere except within google's email empire. Being forced into the cloud is what killed NOAA. They could've upgraded from their 2005 setup to something a bit more modern but instead they migrated to the cloud and it ruined everything.