4 ms·

Why are they so attached to NWS Chat specifically? Perhaps there are reasons unexplained, but from this article it sounds very much like they've got Not Invente

Why are they so attached to NWS Chat specifically? Perhaps there are reasons unexplained, but from this article it sounds very much like they've got Not Invented Here syndrome. Sure, they need bespoke computation services for their modeling functions and so forth. But when it comes to the dissemination of information, it seems like that's very much a commodity these days, and can be provided at such an enormous scale at such low prices that it should scarcely be a concern. I suspect, with no evidence whatsoever, that the NWS is holding this hostage for better funding overall, because it's the visible piece that causes pain to the public. There are other areas where they believe they really need better funding, but haven't been able to convince the budgetmakers of that. In order to force their hand, they're starving the public-facing stuff.

Jtsummers 6y ago
NWSChat is apparently an XMPP-based solution and people were able to use off-the-shelf clients (did some digging out of curiosity). It was also started before things like chat-as-a-service offerings (like Slack) were widespread. It's also import to remember that they needed to coordinate with both government and non-government users, so a solution like MS's chat (what later became Skype for Business, not sure what it was in the 00s) would be challenging to integrate with non-government users even if it was trivially deployed and scaled across the government side. On top of that, using a commercial service isn't necessarily permissible as government agencies have to track their communications and record them with strict rules on how long recordings have to be maintained and how/when they can be destroyed. Commercial services can be used, but they have to have the right policies in place. WFH in 2020 revealed this issue to many as gov't employees were using non-standard tools without proper recording/tracing that technically violated federal laws and regulations but let them get their job done. For better or worse, in the 00s self-hosting a chat service might have been the least bad option to achieve all the requirements they had. However, it also probably suffered (as I mentioned in another comment) from the fact that many gov't agencies don't really have proper IT staffing and don't treat it as something to continually invest in and maintain/sustain.
- CWuestefeld 6y ago
  Your point may well be an important aspect of this. But to the extent that's true, we shouldn't be throwing money at a system (in the larger sense, not just NWS) that's broken. Unless of funding, let's find the badly-designed regulation and fix that. But it's more than just NWS Chat: In December, because of an escalating bandwidth shortage, the Weather Service proposed limiting users to 60 connections per minute on a large number of its websites. Constituents complained about the quota and, earlier this month, the Weather Service announced it would instead impose a data limit of 120 requests per minute and only on servers hosting model data, beginning April 20. This is, or should be, bread and butter stuff. The rest of the world has figured out how to do this cheaply and with high availability. And their overseers even recognize this: Neil Jacobs, former acting head of the National Oceanic Atmospheric Administration, which oversees the Weather Service, said many of the agency’s Internet infrastructure problems are tied to the fact they run on internal hardware rather than through cloud service providers such as Amazon Web Services, Microsoft and Google Cloud. “I’ve demanded in writing that NWS transition these applications … to our Cloud partners. It’s part of an internal strategy I’ve laid out,” Jacobs, a Trump administration appointee, told the Capital Weather Gang in an email before he left office. So why are they not making that architectural change?
  - yellowapple 6y ago
    I ain't really convinced by Jacobs' take; if they're having these issues with on-prem hardware, then they're going to have these same issues with cloud services. It's a matter of resource allocation, effort, and the organization itself, not one of specific implementation details like where the servers live. A cloud transition sounds like a boondoggle and a distraction from more fundamental issues - and when (not if) those issues persist, now we'll be in the same boat but with even worse results per tax dollar.
  - Jtsummers 6y ago
    > So why are they not making that architectural change? Because IT is secondary to their primary mission and they don't have a proper dedicated IT team of sufficient size to enable this or a smooth transition if they tried. This is a repeated problem in the federal government. IT is not their mission, and each entity acts like a small-to-medium sized business, even if they're part of a larger scale organization. The US DOD doesn't even have one email system for all services. Each service has its own, and at least until the early 2010s many (most? all?) military bases had their own email servers with military members (and civilians) having to get new email accounts at each new base they worked at (this isn't the case of having a distributed but mutually coordinated system, it was just that each base literally had their own MS Exchange servers running and each maintained them themselves). And that's pretty much the single largest entity in the executive. Now imagine if it was a struggle for them despite a clear mission benefit how much more challenging this is when the smaller agencies are asked to do the same without being provided sufficient personnel and finances to do it, and no leadership (no real leadership) stepping up to create or establish a cross-service solution.

Jtsummers 6y ago

NWSChat is apparently an XMPP-based solution and people were able to use off-the-shelf clients (did some digging out of curiosity). It was also started before things like chat-as-a-service offerings (like Slack) were widespread. It's also import to remember that they needed to coordinate with both government and non-government users, so a solution like MS's chat (what later became Skype for Business, not sure what it was in the 00s) would be challenging to integrate with non-government users even if it was trivially deployed and scaled across the government side. On top of that, using a commercial service isn't necessarily permissible as government agencies have to track their communications and record them with strict rules on how long recordings have to be maintained and how/when they can be destroyed. Commercial services can be used, but they have to have the right policies in place. WFH in 2020 revealed this issue to many as gov't employees were using non-standard tools without proper recording/tracing that technically violated federal laws and regulations but let them get their job done. For better or worse, in the 00s self-hosting a chat service might have been the least bad option to achieve all the requirements they had. However, it also probably suffered (as I mentioned in another comment) from the fact that many gov't agencies don't really have proper IT staffing and don't treat it as something to continually invest in and maintain/sustain.

CWuestefeld 6y ago
Your point may well be an important aspect of this. But to the extent that's true, we shouldn't be throwing money at a system (in the larger sense, not just NWS) that's broken. Unless of funding, let's find the badly-designed regulation and fix that. But it's more than just NWS Chat: In December, because of an escalating bandwidth shortage, the Weather Service proposed limiting users to 60 connections per minute on a large number of its websites. Constituents complained about the quota and, earlier this month, the Weather Service announced it would instead impose a data limit of 120 requests per minute and only on servers hosting model data, beginning April 20. This is, or should be, bread and butter stuff. The rest of the world has figured out how to do this cheaply and with high availability. And their overseers even recognize this: Neil Jacobs, former acting head of the National Oceanic Atmospheric Administration, which oversees the Weather Service, said many of the agency’s Internet infrastructure problems are tied to the fact they run on internal hardware rather than through cloud service providers such as Amazon Web Services, Microsoft and Google Cloud. “I’ve demanded in writing that NWS transition these applications … to our Cloud partners. It’s part of an internal strategy I’ve laid out,” Jacobs, a Trump administration appointee, told the Capital Weather Gang in an email before he left office. So why are they not making that architectural change?
- yellowapple 6y ago
  I ain't really convinced by Jacobs' take; if they're having these issues with on-prem hardware, then they're going to have these same issues with cloud services. It's a matter of resource allocation, effort, and the organization itself, not one of specific implementation details like where the servers live. A cloud transition sounds like a boondoggle and a distraction from more fundamental issues - and when (not if) those issues persist, now we'll be in the same boat but with even worse results per tax dollar.
- Jtsummers 6y ago
  > So why are they not making that architectural change? Because IT is secondary to their primary mission and they don't have a proper dedicated IT team of sufficient size to enable this or a smooth transition if they tried. This is a repeated problem in the federal government. IT is not their mission, and each entity acts like a small-to-medium sized business, even if they're part of a larger scale organization. The US DOD doesn't even have one email system for all services. Each service has its own, and at least until the early 2010s many (most? all?) military bases had their own email servers with military members (and civilians) having to get new email accounts at each new base they worked at (this isn't the case of having a distributed but mutually coordinated system, it was just that each base literally had their own MS Exchange servers running and each maintained them themselves). And that's pretty much the single largest entity in the executive. Now imagine if it was a struggle for them despite a clear mission benefit how much more challenging this is when the smaller agencies are asked to do the same without being provided sufficient personnel and finances to do it, and no leadership (no real leadership) stepping up to create or establish a cross-service solution.