14 ms·

Weather Service faces bandwidth shortage, proposes limiting key data

173 points by scott_s 6y ago

blakesterz 6y ago
Jonathan Porter, a vice president and general manager at the private forecasting firm AccuWeather, warns that the agency’s proposed solution would harm the timeliness and accuracy of forecasts and severe weather warnings. He said the collection, processing and distribution of weather information are the agency’s “most important services.” That's an interesting quote coming from AccuWeather... https://citizentruth.org/the-politics-behind-the-movement-to-privatize-the-weather/ https://citizentruth.org/the-politics-behind-the-movement-to... "Richard Hirn, an employee who represents the National Weather Service’s Employee Organization, told Bloomberg, “We fear that he [Barry Lee Myers] wants to turn the weather service into a taxpayer-funded subsidiary of AccuWeather.”"
- kevin_thibedeau 6y ago
  AccuWeather is in the business of spreading FUD about the NWS. They are directly responsible for the current level of underfunding for their digital products.
- brootstrap 6y ago
  The headlines are misleading. The people leading the call actually said the problem was 'volume' of data. I heard them talk, i'm not sure they really know what is going on or understand the issue. Also the problem isnt just weather.gov and weather API. Sites like accuweather are likely grabbing raw files from the FTP servers. Everytime a new forecast comes out (for say the GFS), all these people hit the server at same time to download gobs and gobs of data. Yall would be surprised at how much data in is weather prediction outputs!
- rnantes 6y ago
  The CEO of AccuWeather Barry Myers who Trump just appointed to head NWS/NOAA is essentially trying to privatize it. NOAA's API is horrible, slow, and you often have to resort to scraping the HTML because it leaves out data. Instead of rate limiting essential 3rd party services they need to rewrite the API and heavily rely on caching and a CDN. Sadly it looks like private money lobbying is ruining an amazing public good.
falcolas 6y ago
Here's the real story, to my mind: >>> The Weather Service held a public forum Tuesday to discuss the proposal and answer questions. When asked about the investment in computing infrastructure that would be required for these limits to not be necessary, agency officials said a one-time cost of about $1.5 million could avert rate limits. The NOAA budget for fiscal 2020 was $5.4 billion. Buchanan, however, stated the actual cost to address the issue would be higher because the $1.5 million “would comprise just one component of what has to be a multifaceted solution.” The officials at the forum also said that senior management at the Weather Service was aware of the relatively small cost of addressing the issue but that the agency faced “competing priorities.” <<< So, the cost to fix would be peanuts, but it's "not a priority". Sounds like pretty typical corporate bureaucracy against improving infrastructure and efficiency until they're forced to. Using their numbers, it'd still only something like .2% of their budget, if the cost was 10x the $1.5m figure.
- deleted 6y ago
  [deleted]
- chrisseaton 6y ago
  > So, the cost to fix would be peanuts, but it's "not a priority" But there are probably thousands of things that cost only peanuts and aren't a priority. If you fix them all they add up to not peanuts and then you don't have any money left and you didn't even fix any priority tasks!
  - Naac 6y ago
    But out of those thousand things only this one made the news no? Doesn't it deserve some extra priority?
  - falcolas 6y ago
    Serving the public should always be a priority for a government institution. In the case of the weather, disseminating the forecasts is just as critical as making the forecasts in the first place. These forecasts directly affect everything from air travel to crop planting/harvesting to sea travel. They are critical for our economic (and citizen's) ongoing health. And the internet, for better or worse, is the primary means of dissemination for the NOAA. Failing in this is failing at one of their greatest duties, IMO.
dsr_ 6y ago
The actual cost of a solution is tiny. (Use four different CDNs.) The political motivation is to portray the government as incapable of doing anything other than defense, so that the functions can be outsourced to the private sector. Expect a solution to appear shortly after the regime changes.
- chrismeller 6y ago
  You still have to pay for those CDNs. If you think this is a red/blue issue you haven’t been following along, though. Obama allocated funds specifically for satellites, but reduced the operating budget [1]. Trump actually increased funding for several areas as part of the stopgap funding in 2018, particularly for satellites [2]. There are other examples over the decades of course, but something that seems to be a trend is the specificity. Satellites are flashy and exciting and get funding from politicians. Servers? Meh... why don’t they just get a couple extra machines? My laptop is fast as hell and only cost like $600. Bandwidth? Why don’t they just get whatever internet I have at home? That costs like $100/m and the Por— I mean the CNBC videos start immediately. 1: https://www.washingtonpost.com/blogs/capital-weather-gang/post/national-weather-service-budget-cuts-misguided-misplaced/2012/03/29/gIQAmm6qiS_blog.html https://www.washingtonpost.com/blogs/capital-weather-gang/po... 2: https://qz.com/1204034/noaa-trump-signed-a-bill-that-gives-millions-to-agency-in-charge-of-climate-and-weather-science/ https://qz.com/1204034/noaa-trump-signed-a-bill-that-gives-m...
  - cobookman 6y ago
    This why I'm so excited about Cloud in the public sector. Its so easy to defer upgrades on a server fleet, even though it might cost someone more money later on. The Public Cloud provides a monthly expense that just becomes part of the operation budget. It's exciting to think about the innovation to government services as an end-result.
    - dingaling 6y ago
      > The Public Cloud provides a monthly expense that just becomes part of the operation budget. Ops budgets aren't infinite. In this example, if the cloud egress costs kept rising due to demand then the weather service would still have to find ways to restrict usage.
saalweachter 6y ago
I've noticed weather.gov/<ZIP> has gotten slow recently. Which is a shame; it's a light website serving a dense block of reasonably formatted data that used to be super snappy.
chrismeller 6y ago
“The Weather Service’s proposed remedy is to limit users to 60 connections per minute on a large number of its websites that provide weather observations, forecasts, warnings, computer model data, air quality information, aviation weather support and ocean conditions.” Well that sounds incredibly reasonable in general, but it completely depends upon what type of interface they’re providing. Do you have to query all of those things separately for every locality you serve? Honestly, this is why so many big companies still prefer things that HN would call “stupid” like an FTP drop of a file 5 times per day/hour/etc. Every FTP implementation I’ve ever seen has included either checking the timestamp, timestamping the filename and filtering on that, or moving to an “archive” folder after processing. Most of the API implementations I see ignore the fact that HEAD requests exist, cache headers can and should be used, etc.
- tenebrisalietum 6y ago
  Still, it's not 1971 anymore. Can we move to SFTP at least?
  - chrismeller 6y ago
    Heh. Well, perhaps, but for public data it doesn’t really matter in general, does it? Just like that installer you just downloaded off the internet, did you check the md5/sha to make sure it wasn’t tampered with? Just because it was over HTTPS doesn’t really mean much.
  - adamcstephens 6y ago
    Move to SFTP requires you to use authentication and increases load on the backend servers.
- brootstrap 6y ago
  You have any experience with numerical weather prediction files? The ftp files you download are typically 'the whole globe' with a bunch of variable stacked like layers of images in a big image file. One of the problems is that a typical numerical weather prediction output has so much fucking data you dont need. GFS for example. IT has maybe 200 or so timesteps in each forecast, runs 4x a day. There are a shit load of configs to run for the GFS. And each config + output has maybe 300/400 variables in each file? So you download 200 files, each with 400 images in it, each image is pretty dense. Each file is ~200 megs, X 200 files. You do the math. Get that 4x a day. And that is just one forecast model. We have lots more, and many more that have higher resolution and denser data. The biggest problem i think is they have a goddamn free-ass oldschool FTP server. They are allowing dimwit grad school scrip kiddies to code up whatever shit they want. Infinite loops to try and download gobs of data you dont understand? sure go ahead. They dont even know how to trakc users. they want to keep an open server, and use IP Address as unique ID. I suggested to use fucking tokens, or username/pass combo at least haha
blantonl 6y ago
I think there are two key solutions for a resolution here: 1) CDNs are a tried a true method for getting content closer to the end user, properly cached, and for less cost. 2) Expose more services that NOAA and the NWS delivers as APIs. If you are providing services for the public good where the consumers are just scraping Web pages to gather the necessary data and content, then you are doing it wrong. It sounds to me like they are serving more API type content over Web pages than they are Web content to Web consumers. I don't want to second guess the NWS/NOAA technical architects, because I know they are probably doing the best they can. Someone simply needs to communicate these goals and solutions to the political appointees...
- tinus_hn 6y ago
  I’m not sure if you can get a CDN for an ftp service. Certainly not as easy as you can for an http service.
  - deleted 6y ago
    [deleted]
  - nitrogen 6y ago
    That just looks like a bunch of mirrors behind a round robin DNS.
    - cratermoon 6y ago
      A CDN is a bunch of mirrors behind a round robin DNS, just with more steps.
  - pridkett 6y ago
    This has been done for decades - long before HTTP took over as the standard we had sites like ftp.cdrom.com and ftp.kernel.org. Usually this is done by using a DNS that either is localized to your or round robin DNS. There were many times when I remember logging into ftp.kernel.org and being surprised that I was connected to a server in Finland, Europe, or Japan. Wasn’t the smartest thing, but with more advanced DNS services, it certainly seems possible.
LatteLazy 6y ago
Isn't this one of the services where Trump appointed a guy who runs a private, for profit, that wants to offer the same data for cash to lead it?
- pbourke 6y ago
  Yes, the former CEO of AccuWeather was appointed but not confirmed to lead NOAA. He was not a scientist and had spent years attempting to undermine the agency from the outside. He also, in keeping with tradition for Trump appointees, was involved with various scandals. https://www.bloomberg.com/news/features/2018-06-14/trump-s-pick-to-lead-weather-agency-spent-30-years-fighting-it https://www.bloomberg.com/news/features/2018-06-14/trump-s-p...
- lotsofpulp 6y ago
  It's unfortunate that your comment is downvoted for simply stating relevant facts.
zetazzed 6y ago
This is a great time to check out Michael Lewis's The Fifth Risk (https://www.amazon.com/Fifth-Risk-Michael-Lewis/dp/1324002646 https://www.amazon.com/Fifth-Risk-Michael-Lewis/dp/132400264...), which does a deep dive into NOAA, the weather service, and the wretched attempts to strangle the usefulness of this public service while keeping it alive enough to act as a massive subsidy to accuweather and related bottom feeders. It should disgust both libertarians and proponents of active government, and really anyone paying attention. Accuweather wants NWS to do all the forecasting work but then funnel all the data through Accu and a few other private hands, who of course want to profit from this data but not pay for it.
- deleted 6y ago
  [deleted]
dboreham 6y ago
The underlying problem is a big one, that drives much of what's bad about the world today: there's no way to charge for internet services.
- fiddlerwoaroof 6y ago
  There is, but it has its own downsides. Also, charging for internet services would have the effect of cementing the monopolies: why pay $5 for signal and $5 for mastodon when I can pay $5 for Facebook and get both?
- toomuchtodo 6y ago
  NWS could push the data commercial users want into a requestor pays S3 bucket. Archived/historical data could be rate limited, and made available for free (and I'd probably upload it into the Internet Archive for good measure). There is a solid case that citizens requesting weather data should have such requests fulfilled at no or minimal cost, similar to GPS services (a public good) or FOIA requests. The government does charge user fees for some commercial services (IRS tax transcripts for income verification, for example). EDIT: Please take the time to submit a comment on this issue [1] (last page of the pdf, December 18th deadline). [1] https://www.weather.gov/media/notification/pdf2/pns20-85ncep_web_access.pdf https://www.weather.gov/media/notification/pdf2/pns20-85ncep...
  - tjohns 6y ago
    > There is a solid case that citizens requesting weather data should have such requests fulfilled at no or minimal cost, similar to GPS services (a public good) or FOIA requests. I totally agree that weather data should be provided for free as a public service. The NWS is one of the true gems of the US government. That said, this view isn't universally shared. Many other countries' meteorology services charge for access to data - especially if you want access to the raw model output. Most famously, the US's GFS output is provided for free while the EU's ECMWF output requires a paid subscription. If open access to weather data is something that's important to you, please make your voice heard!
    - 6y ago
h2odragon 6y ago
The NWS radar web interface is one those great examples of a government service done right. They've been working on an upgraded version, the public deployment was just pushed back a week. The new stuff appears much more bandwidth intensive, especially since "shared cache" is dead. https://preview-radar.weather.gov/ https://preview-radar.weather.gov/ still seems to be getting assets off arcgis.com ; surely theres room for a public map server that can stand this traffic? I haven't looked at the code; I suspect its amenable to hacking up one's own views.
- deleted 6y ago
  [deleted]
- deleted 6y ago
  [deleted]
- ashtonbaker 6y ago
  That site breaks the "back" button for me pretty badly. Small complaint, though - I really love the new UI.
  - maxerickson 6y ago
    The animated displays are miserably slow to load and don't perform very well. I'm in the habit of using the radar lite version of the old system, which has the whole gif loaded fast enough for the first loop to not be delayed. On the new site, the base map tiles start to filter in after about the same amount of time, along with a few pieces of the first radar frame.
- xemoka 6y ago
  Huh, made with Vue/Vuex + map via VueLayers (OpenLayers wrapper). Actual radar data is coming from a geoserver at https://preview-opengeo.ncep.noaa.gov/geoserver/ https://preview-opengeo.ncep.noaa.gov/geoserver/ AGOL (ArcGIS Online) is only providing the basemap and layer overlays (state boundaries etc), those probably come from Esri's "public" data portal and I doubt are part of the bandwidth issue.
- esaym 6y ago
forgotAgain 6y ago
I've had some experience dealing with the Weather Service when trying to develop a weather app using https://www.weather.gov/documentation/services-web-api# https://www.weather.gov/documentation/services-web-api#. It was a frustrating experience and my initial impulse was to vent about that. But I'm choosing not to do that mainly because it seemed at the time that every person that I dealt with at NWS was trying to do the best they could. The situation that NWS is facing is basically that massive tax cuts coupled with growth in defense and social programs has left most other areas of the federal government hollowed out. The amount of money needed to fix the problem being discussed here is minimal but after decades of budget cuts across non-defense and non social-welfare programs there is simply little left to invest in improving services.
- yjftsjthsd-h 6y ago
  Per https://news.ycombinator.com/item?id=25375739 https://news.ycombinator.com/item?id=25375739 - >>> The Weather Service held a public forum Tuesday to discuss the proposal and answer questions. When asked about the investment in computing infrastructure that would be required for these limits to not be necessary, agency officials said a one-time cost of about $1.5 million could avert rate limits. The NOAA budget for fiscal 2020 was $5.4 billion. They do, in fact, appear to have the money.
  - forgotAgain 6y ago
    > They do, in fact, appear to have the money. $5.4 billion is indeed a huge number but gives no information as to the availability of the $1.5 million. Without knowing what the cost is of what they must accomplish with the $5.4 billion no conclusions can be drawn.
  - xxpor 6y ago
    How many of those dollars are going to satellites and Doppler radars? Those aren't exactly cheap.
tyingq 6y ago
Opportunity for Cloudflare or some other CDN?
- agent86 6y ago
  I am by no means an expert in this space, but this was my first thought as well. A government agency with bipartisan support and positive public image seems like a good entity for Cloudflare, Google, Amazon, Microsoft or someone else with infrastructure and bandwidth to be a white knight for.
  - asperous 6y ago
    The government isn't allowed to accept gifts to avoid the appearance of bribes. They can use a free service or offering but it has to be free for everyone
- jrnichols 6y ago
  Or.. isn't this exactly what we had "Internet2" for in the first place? Universities used to host a LOT of stuff. Can't they pick up some of the traffic again? Offer data centers/hosting providers a tax incentive to help out? I don't have all the answers, but "whatever Accuweather wants" is probably not the right decision.
  - vlan0 6y ago
    I question the accuracy of this article. I'll have to poke around for notes from the I2 TechExtra a couple of months ago, but NOAA was there and talked about what they have in the works.
Spooky23 6y ago
Remember with NOAA and NWS you always have GOP pressure originating from Accuweather, which really wants to be a the gatekeeper for this data. Over the years they have attacked the NWS web properties in particular (Rick Santorum was their attack dog a few years back) as their business of selling graphics to TV stations has declined along with traditional TV.
- lotsofpulp 6y ago
  https://www.bloomberg.com/news/features/2018-06-14/trump-s-pick-to-lead-weather-agency-spent-30-years-fighting-it https://www.bloomberg.com/news/features/2018-06-14/trump-s-p...
  - warkdarrior 6y ago
    > The NWS was giving away forecasts on its website, radio stations, and elsewhere, when businesses such as AccuWeather charged its clients for theirs—never mind that AccuWeather relied on the service’s free data to formulate its own predictions. Santorum agreed that commercial weather companies deserved protection. Corruption at its finest.
  - e40 6y ago
    I don't care who you voted for, this should make every American really, really mad.
    - falcolas 6y ago
      Well, not if you don't understand how hard accurate weather forecsts are to get, or why a private entity couldn't do better. After all, that's why the news shows have weathermen, right?! And if the NOAA goes away, the government becomes smaller, and their taxes go down, right? :/ EDIT: Dear downvoters - while this is not my opinion, this is how people think, what people believe. Dismiss it at your own risk.
      - jschwartzi 6y ago
jrochkind1 6y ago
Users are accessing data in high volumes, especially during unusual weather events, which brings down the services. Is there something I'm not understanding about the problem here with bandwidth? Does cloudflare or another CDN for this stuff really cost an amount hard to include in the budget for such an essential public service? Or, shit, can cloudflare just go and donate this to the government? It would be pretty good publicity for them and probably not a huge cost on their scale. What am I missing? This seems ridiculous that the USA cannot afford to provide this high-value public service. When did we become a two-bit country when I didn't notice?
jmount 6y ago
Likely this is on purpose. I remember some congressman (PA's Rick Santorum?) trying to forbid the US Weather Service from sharing weather reports/forecasts so a partner of his would have a better market to sell the same service into.
- pavedwalden 6y ago
  There's a lot about this in Michael Lewis' "The Fifth Risk". The most maddening part is that the company lobbying to weaken the National Weather Service can't replicate much of what the government is doing here and sells a lot of repackaged NWS data. I suppose it doesn't matter to them if dismantling the service leads to less weather data being available overall, as long as their piece of the market gets bigger.
- jdc 6y ago
  John Oliver did an episode on this chicanery last year. https://www.youtube.com/watch?v=qMGn9T37eR8 https://www.youtube.com/watch?v=qMGn9T37eR8
ris 6y ago
Timely reminder that S3 buckets support BitTorrent natively.
sramsay 6y ago
I have written several command-line weather programs for different services -- many of which I had to abandon. For example, I wrote a popular one for Weather Underground that I had to retire because WU stopped allowing free access to their API. My latest effort is one that just deals with NWS/NOAA directly, and reading this article has me wondering if part of their bandwidth trouble is a product of the way their API works. The API is a huge set of linked JSON documents. So it very often happens that a request for data is actually half-a-dozen requests through embedded URLs (all of which are made by the client one after the other). It's all very logical in some ways, but at the same time a bit byzantine. I often wonder which entity is served by the way it's set up. From my perspective, it's easily the hardest API to work with. Weather Underground was wonderful, and so was Dark Sky (which I played with for awhile). I've had people over the years ask if one or another of my tools could be used for large scale or very rapid weather data collection, but all of the services I have worked with throttle connections exactly as NOAA is proposing. I suppose I'm at least relieved to hear that they're not restricting access by independent developers!
- rsync 6y ago
  I'm curious - and perhaps you can help ... I hate wunderground - it is a giant, bloated mess of a website with a UI that jumps all over the place ... However, I can't find anyone else to give me a 10 day forecast. It seems like wunderground is the only place that collates it like that, which I like very much. Am I wrong ? Is there some other source for a proper, detailed, 10 day forecast ?
  - nightski 6y ago
    The NWS directly works well for me. Just type in the location in the upper left. I kind of appreciate the simplicity of their website. https://forecast.weather.gov/ https://forecast.weather.gov/
    - wtallis 6y ago
      Could you be a bit more specific and demonstrate how to get a 10-day forecast out of that URL?
cratermoon 6y ago
Charge AccuWeather by the megabyte. Problem solved.
- paultopia 6y ago
  Seriously, though. I wonder if accuweather or someone got legislation passed to prohibit them from charging heavy users...
plantain 6y ago
I've suggested multiple times to them and even offered to help them setup HTTP BitTorrent to reduce the loads, but there's just no actual interest in solving the problem...
jfrunyon 6y ago
> According to companies that draw large amounts of this data, the proposed limit will substantially harm the services they provide to customers. The possible negative effect on forecasts has also raised concerns among congressional lawmakers. Well then maybe they should be paying for more efficient feeds instead of querying more than once per second for a sustained duration? Really? In what world did anyone think a good solution to ANY problem was "query a website over 60 times per minute continuously"???
BrandonMarc 6y ago
"key data" .... which would be a small subset of all data ... and to solve bandwidth issues they want to make it more scarce. This is a stunt. They want money, and are using their most popular (key) data points as a threat. Just like politicians. Money gets tight and they say they'll cut essential services like fire and police, but what they won't tell you is they really want their pet project funded, using deceit.
maxerickson 6y ago
Is this because they switched to the new radar view? Just throw it away and keep the old one running.
marktangotango 6y ago
Does anyone know where one can get historic wind data (avg direction and speed per hour for example) for a well known location like LAX? I was looking a few months ago and couldn’t find anything. I thought surely NOAA or NWS would have this?
- Buttons840 6y ago
  WeatherSpark?
- brootstrap 6y ago
  YEah i can tell you for 10 bucks. Try to google wind speed LAX for starters. Did you search at all? Airports have weather stations that record shit and you can get at those data.
- a_square_peg 6y ago
  As mentioned in the comment elsewhere in this thread, I run an API service (oikolab.com) that does this. You can get hourly data for any location from 1950 onward.
crazyjncsu 6y ago
My fault guys, that was just me ... but I can’t help if if they have the only national radar animation that isn’t infested with ads and other bloat: https://radar.weather.gov/Conus/Loop/NatLoop.gif https://radar.weather.gov/Conus/Loop/NatLoop.gif
ghthor 6y ago
Why don't they just charge their largest consumers?
ojbyrne 6y ago
John Oliver on this a year or so ago (NSFW): https://youtu.be/qMGn9T37eR8 https://youtu.be/qMGn9T37eR8
creinhardt 6y ago
As far as I know, the weather.gov api is hosted out of a single facility (the NCEP NCO), and if that building has any sort of network issue, the api goes down. At the very least it seems like using some sort of CDN would provide a bit more resiliency, but maybe there are infrastructure constraints that make this difficult.
csense 6y ago
They should publish this data over some protocol that lets users contribute upload bandwidth (BitTorrent, IPFS).

173 points by scott_s 6y ago

Jonathan Porter, a vice president and general manager at the private forecasting firm AccuWeather, warns that the agency’s proposed solution would harm the timeliness and accuracy of forecasts and severe weather warnings. He said the collection, processing and distribution of weather information are the agency’s “most important services.” That's an interesting quote coming from AccuWeather... https://citizentruth.org/the-politics-behind-the-movement-to-privatize-the-weather/ https://citizentruth.org/the-politics-behind-the-movement-to... "Richard Hirn, an employee who represents the National Weather Service’s Employee Organization, told Bloomberg, “We fear that he [Barry Lee Myers] wants to turn the weather service into a taxpayer-funded subsidiary of AccuWeather.”"

kevin_thibedeau 6y ago
AccuWeather is in the business of spreading FUD about the NWS. They are directly responsible for the current level of underfunding for their digital products.
brootstrap 6y ago
The headlines are misleading. The people leading the call actually said the problem was 'volume' of data. I heard them talk, i'm not sure they really know what is going on or understand the issue. Also the problem isnt just weather.gov and weather API. Sites like accuweather are likely grabbing raw files from the FTP servers. Everytime a new forecast comes out (for say the GFS), all these people hit the server at same time to download gobs and gobs of data. Yall would be surprised at how much data in is weather prediction outputs!
rnantes 6y ago
The CEO of AccuWeather Barry Myers who Trump just appointed to head NWS/NOAA is essentially trying to privatize it. NOAA's API is horrible, slow, and you often have to resort to scraping the HTML because it leaves out data. Instead of rate limiting essential 3rd party services they need to rewrite the API and heavily rely on caching and a CDN. Sadly it looks like private money lobbying is ruining an amazing public good.

falcolas 6y ago

Here's the real story, to my mind: >>> The Weather Service held a public forum Tuesday to discuss the proposal and answer questions. When asked about the investment in computing infrastructure that would be required for these limits to not be necessary, agency officials said a one-time cost of about $1.5 million could avert rate limits. The NOAA budget for fiscal 2020 was $5.4 billion. Buchanan, however, stated the actual cost to address the issue would be higher because the $1.5 million “would comprise just one component of what has to be a multifaceted solution.” The officials at the forum also said that senior management at the Weather Service was aware of the relatively small cost of addressing the issue but that the agency faced “competing priorities.” <<< So, the cost to fix would be peanuts, but it's "not a priority". Sounds like pretty typical corporate bureaucracy against improving infrastructure and efficiency until they're forced to. Using their numbers, it'd still only something like .2% of their budget, if the cost was 10x the $1.5m figure.

deleted 6y ago
[deleted]
chrisseaton 6y ago
> So, the cost to fix would be peanuts, but it's "not a priority" But there are probably thousands of things that cost only peanuts and aren't a priority. If you fix them all they add up to not peanuts and then you don't have any money left and you didn't even fix any priority tasks!
- Naac 6y ago
  But out of those thousand things only this one made the news no? Doesn't it deserve some extra priority?
- falcolas 6y ago
  Serving the public should always be a priority for a government institution. In the case of the weather, disseminating the forecasts is just as critical as making the forecasts in the first place. These forecasts directly affect everything from air travel to crop planting/harvesting to sea travel. They are critical for our economic (and citizen's) ongoing health. And the internet, for better or worse, is the primary means of dissemination for the NOAA. Failing in this is failing at one of their greatest duties, IMO.

dsr_ 6y ago

The actual cost of a solution is tiny. (Use four different CDNs.) The political motivation is to portray the government as incapable of doing anything other than defense, so that the functions can be outsourced to the private sector. Expect a solution to appear shortly after the regime changes.

chrismeller 6y ago
You still have to pay for those CDNs. If you think this is a red/blue issue you haven’t been following along, though. Obama allocated funds specifically for satellites, but reduced the operating budget [1]. Trump actually increased funding for several areas as part of the stopgap funding in 2018, particularly for satellites [2]. There are other examples over the decades of course, but something that seems to be a trend is the specificity. Satellites are flashy and exciting and get funding from politicians. Servers? Meh... why don’t they just get a couple extra machines? My laptop is fast as hell and only cost like $600. Bandwidth? Why don’t they just get whatever internet I have at home? That costs like $100/m and the Por— I mean the CNBC videos start immediately. 1: https://www.washingtonpost.com/blogs/capital-weather-gang/post/national-weather-service-budget-cuts-misguided-misplaced/2012/03/29/gIQAmm6qiS_blog.html https://www.washingtonpost.com/blogs/capital-weather-gang/po... 2: https://qz.com/1204034/noaa-trump-signed-a-bill-that-gives-millions-to-agency-in-charge-of-climate-and-weather-science/ https://qz.com/1204034/noaa-trump-signed-a-bill-that-gives-m...
- cobookman 6y ago
  This why I'm so excited about Cloud in the public sector. Its so easy to defer upgrades on a server fleet, even though it might cost someone more money later on. The Public Cloud provides a monthly expense that just becomes part of the operation budget. It's exciting to think about the innovation to government services as an end-result.
  - dingaling 6y ago
    > The Public Cloud provides a monthly expense that just becomes part of the operation budget. Ops budgets aren't infinite. In this example, if the cloud egress costs kept rising due to demand then the weather service would still have to find ways to restrict usage.

saalweachter 6y ago

I've noticed weather.gov/<ZIP> has gotten slow recently. Which is a shame; it's a light website serving a dense block of reasonably formatted data that used to be super snappy.

chrismeller 6y ago

“The Weather Service’s proposed remedy is to limit users to 60 connections per minute on a large number of its websites that provide weather observations, forecasts, warnings, computer model data, air quality information, aviation weather support and ocean conditions.” Well that sounds incredibly reasonable in general, but it completely depends upon what type of interface they’re providing. Do you have to query all of those things separately for every locality you serve? Honestly, this is why so many big companies still prefer things that HN would call “stupid” like an FTP drop of a file 5 times per day/hour/etc. Every FTP implementation I’ve ever seen has included either checking the timestamp, timestamping the filename and filtering on that, or moving to an “archive” folder after processing. Most of the API implementations I see ignore the fact that HEAD requests exist, cache headers can and should be used, etc.

tenebrisalietum 6y ago
Still, it's not 1971 anymore. Can we move to SFTP at least?
- chrismeller 6y ago
  Heh. Well, perhaps, but for public data it doesn’t really matter in general, does it? Just like that installer you just downloaded off the internet, did you check the md5/sha to make sure it wasn’t tampered with? Just because it was over HTTPS doesn’t really mean much.
- adamcstephens 6y ago
  Move to SFTP requires you to use authentication and increases load on the backend servers.
brootstrap 6y ago
You have any experience with numerical weather prediction files? The ftp files you download are typically 'the whole globe' with a bunch of variable stacked like layers of images in a big image file. One of the problems is that a typical numerical weather prediction output has so much fucking data you dont need. GFS for example. IT has maybe 200 or so timesteps in each forecast, runs 4x a day. There are a shit load of configs to run for the GFS. And each config + output has maybe 300/400 variables in each file? So you download 200 files, each with 400 images in it, each image is pretty dense. Each file is ~200 megs, X 200 files. You do the math. Get that 4x a day. And that is just one forecast model. We have lots more, and many more that have higher resolution and denser data. The biggest problem i think is they have a goddamn free-ass oldschool FTP server. They are allowing dimwit grad school scrip kiddies to code up whatever shit they want. Infinite loops to try and download gobs of data you dont understand? sure go ahead. They dont even know how to trakc users. they want to keep an open server, and use IP Address as unique ID. I suggested to use fucking tokens, or username/pass combo at least haha

blantonl 6y ago

I think there are two key solutions for a resolution here: 1) CDNs are a tried a true method for getting content closer to the end user, properly cached, and for less cost. 2) Expose more services that NOAA and the NWS delivers as APIs. If you are providing services for the public good where the consumers are just scraping Web pages to gather the necessary data and content, then you are doing it wrong. It sounds to me like they are serving more API type content over Web pages than they are Web content to Web consumers. I don't want to second guess the NWS/NOAA technical architects, because I know they are probably doing the best they can. Someone simply needs to communicate these goals and solutions to the political appointees...

tinus_hn 6y ago
I’m not sure if you can get a CDN for an ftp service. Certainly not as easy as you can for an http service.
- deleted 6y ago
  [deleted]
- nitrogen 6y ago
  That just looks like a bunch of mirrors behind a round robin DNS.
  - cratermoon 6y ago
    A CDN is a bunch of mirrors behind a round robin DNS, just with more steps.
- pridkett 6y ago
  This has been done for decades - long before HTTP took over as the standard we had sites like ftp.cdrom.com and ftp.kernel.org. Usually this is done by using a DNS that either is localized to your or round robin DNS. There were many times when I remember logging into ftp.kernel.org and being surprised that I was connected to a server in Finland, Europe, or Japan. Wasn’t the smartest thing, but with more advanced DNS services, it certainly seems possible.

LatteLazy 6y ago

Isn't this one of the services where Trump appointed a guy who runs a private, for profit, that wants to offer the same data for cash to lead it?

pbourke 6y ago
Yes, the former CEO of AccuWeather was appointed but not confirmed to lead NOAA. He was not a scientist and had spent years attempting to undermine the agency from the outside. He also, in keeping with tradition for Trump appointees, was involved with various scandals. https://www.bloomberg.com/news/features/2018-06-14/trump-s-pick-to-lead-weather-agency-spent-30-years-fighting-it https://www.bloomberg.com/news/features/2018-06-14/trump-s-p...
lotsofpulp 6y ago
It's unfortunate that your comment is downvoted for simply stating relevant facts.

zetazzed 6y ago

This is a great time to check out Michael Lewis's The Fifth Risk (https://www.amazon.com/Fifth-Risk-Michael-Lewis/dp/1324002646 https://www.amazon.com/Fifth-Risk-Michael-Lewis/dp/132400264...), which does a deep dive into NOAA, the weather service, and the wretched attempts to strangle the usefulness of this public service while keeping it alive enough to act as a massive subsidy to accuweather and related bottom feeders. It should disgust both libertarians and proponents of active government, and really anyone paying attention. Accuweather wants NWS to do all the forecasting work but then funnel all the data through Accu and a few other private hands, who of course want to profit from this data but not pay for it.

deleted 6y ago
[deleted]

dboreham 6y ago

The underlying problem is a big one, that drives much of what's bad about the world today: there's no way to charge for internet services.

fiddlerwoaroof 6y ago
There is, but it has its own downsides. Also, charging for internet services would have the effect of cementing the monopolies: why pay $5 for signal and $5 for mastodon when I can pay $5 for Facebook and get both?
toomuchtodo 6y ago
NWS could push the data commercial users want into a requestor pays S3 bucket. Archived/historical data could be rate limited, and made available for free (and I'd probably upload it into the Internet Archive for good measure). There is a solid case that citizens requesting weather data should have such requests fulfilled at no or minimal cost, similar to GPS services (a public good) or FOIA requests. The government does charge user fees for some commercial services (IRS tax transcripts for income verification, for example). EDIT: Please take the time to submit a comment on this issue [1] (last page of the pdf, December 18th deadline). [1] https://www.weather.gov/media/notification/pdf2/pns20-85ncep_web_access.pdf https://www.weather.gov/media/notification/pdf2/pns20-85ncep...
- tjohns 6y ago
  > There is a solid case that citizens requesting weather data should have such requests fulfilled at no or minimal cost, similar to GPS services (a public good) or FOIA requests. I totally agree that weather data should be provided for free as a public service. The NWS is one of the true gems of the US government. That said, this view isn't universally shared. Many other countries' meteorology services charge for access to data - especially if you want access to the raw model output. Most famously, the US's GFS output is provided for free while the EU's ECMWF output requires a paid subscription. If open access to weather data is something that's important to you, please make your voice heard!
  - 6y ago

h2odragon 6y ago

The NWS radar web interface is one those great examples of a government service done right. They've been working on an upgraded version, the public deployment was just pushed back a week. The new stuff appears much more bandwidth intensive, especially since "shared cache" is dead. https://preview-radar.weather.gov/ https://preview-radar.weather.gov/ still seems to be getting assets off arcgis.com ; surely theres room for a public map server that can stand this traffic? I haven't looked at the code; I suspect its amenable to hacking up one's own views.

deleted 6y ago
[deleted]
deleted 6y ago
[deleted]
ashtonbaker 6y ago
That site breaks the "back" button for me pretty badly. Small complaint, though - I really love the new UI.
- maxerickson 6y ago
  The animated displays are miserably slow to load and don't perform very well. I'm in the habit of using the radar lite version of the old system, which has the whole gif loaded fast enough for the first loop to not be delayed. On the new site, the base map tiles start to filter in after about the same amount of time, along with a few pieces of the first radar frame.
xemoka 6y ago
Huh, made with Vue/Vuex + map via VueLayers (OpenLayers wrapper). Actual radar data is coming from a geoserver at https://preview-opengeo.ncep.noaa.gov/geoserver/ https://preview-opengeo.ncep.noaa.gov/geoserver/ AGOL (ArcGIS Online) is only providing the basemap and layer overlays (state boundaries etc), those probably come from Esri's "public" data portal and I doubt are part of the bandwidth issue.
esaym 6y ago

forgotAgain 6y ago

I've had some experience dealing with the Weather Service when trying to develop a weather app using https://www.weather.gov/documentation/services-web-api# https://www.weather.gov/documentation/services-web-api#. It was a frustrating experience and my initial impulse was to vent about that. But I'm choosing not to do that mainly because it seemed at the time that every person that I dealt with at NWS was trying to do the best they could. The situation that NWS is facing is basically that massive tax cuts coupled with growth in defense and social programs has left most other areas of the federal government hollowed out. The amount of money needed to fix the problem being discussed here is minimal but after decades of budget cuts across non-defense and non social-welfare programs there is simply little left to invest in improving services.

yjftsjthsd-h 6y ago
Per https://news.ycombinator.com/item?id=25375739 https://news.ycombinator.com/item?id=25375739 - >>> The Weather Service held a public forum Tuesday to discuss the proposal and answer questions. When asked about the investment in computing infrastructure that would be required for these limits to not be necessary, agency officials said a one-time cost of about $1.5 million could avert rate limits. The NOAA budget for fiscal 2020 was $5.4 billion. They do, in fact, appear to have the money.
- forgotAgain 6y ago
  > They do, in fact, appear to have the money. $5.4 billion is indeed a huge number but gives no information as to the availability of the $1.5 million. Without knowing what the cost is of what they must accomplish with the $5.4 billion no conclusions can be drawn.
- xxpor 6y ago
  How many of those dollars are going to satellites and Doppler radars? Those aren't exactly cheap.

tyingq 6y ago

Opportunity for Cloudflare or some other CDN?

agent86 6y ago
I am by no means an expert in this space, but this was my first thought as well. A government agency with bipartisan support and positive public image seems like a good entity for Cloudflare, Google, Amazon, Microsoft or someone else with infrastructure and bandwidth to be a white knight for.
- asperous 6y ago
  The government isn't allowed to accept gifts to avoid the appearance of bribes. They can use a free service or offering but it has to be free for everyone
jrnichols 6y ago
Or.. isn't this exactly what we had "Internet2" for in the first place? Universities used to host a LOT of stuff. Can't they pick up some of the traffic again? Offer data centers/hosting providers a tax incentive to help out? I don't have all the answers, but "whatever Accuweather wants" is probably not the right decision.
- vlan0 6y ago
  I question the accuracy of this article. I'll have to poke around for notes from the I2 TechExtra a couple of months ago, but NOAA was there and talked about what they have in the works.

Spooky23 6y ago

Remember with NOAA and NWS you always have GOP pressure originating from Accuweather, which really wants to be a the gatekeeper for this data. Over the years they have attacked the NWS web properties in particular (Rick Santorum was their attack dog a few years back) as their business of selling graphics to TV stations has declined along with traditional TV.

lotsofpulp 6y ago
https://www.bloomberg.com/news/features/2018-06-14/trump-s-pick-to-lead-weather-agency-spent-30-years-fighting-it https://www.bloomberg.com/news/features/2018-06-14/trump-s-p...
- warkdarrior 6y ago
  > The NWS was giving away forecasts on its website, radio stations, and elsewhere, when businesses such as AccuWeather charged its clients for theirs—never mind that AccuWeather relied on the service’s free data to formulate its own predictions. Santorum agreed that commercial weather companies deserved protection. Corruption at its finest.
- e40 6y ago
  I don't care who you voted for, this should make every American really, really mad.
  - falcolas 6y ago
    Well, not if you don't understand how hard accurate weather forecsts are to get, or why a private entity couldn't do better. After all, that's why the news shows have weathermen, right?! And if the NOAA goes away, the government becomes smaller, and their taxes go down, right? :/ EDIT: Dear downvoters - while this is not my opinion, this is how people think, what people believe. Dismiss it at your own risk.
    - jschwartzi 6y ago

jrochkind1 6y ago

Users are accessing data in high volumes, especially during unusual weather events, which brings down the services. Is there something I'm not understanding about the problem here with bandwidth? Does cloudflare or another CDN for this stuff really cost an amount hard to include in the budget for such an essential public service? Or, shit, can cloudflare just go and donate this to the government? It would be pretty good publicity for them and probably not a huge cost on their scale. What am I missing? This seems ridiculous that the USA cannot afford to provide this high-value public service. When did we become a two-bit country when I didn't notice?

jmount 6y ago

Likely this is on purpose. I remember some congressman (PA's Rick Santorum?) trying to forbid the US Weather Service from sharing weather reports/forecasts so a partner of his would have a better market to sell the same service into.

pavedwalden 6y ago
There's a lot about this in Michael Lewis' "The Fifth Risk". The most maddening part is that the company lobbying to weaken the National Weather Service can't replicate much of what the government is doing here and sells a lot of repackaged NWS data. I suppose it doesn't matter to them if dismantling the service leads to less weather data being available overall, as long as their piece of the market gets bigger.
jdc 6y ago
John Oliver did an episode on this chicanery last year. https://www.youtube.com/watch?v=qMGn9T37eR8 https://www.youtube.com/watch?v=qMGn9T37eR8

ris 6y ago

Timely reminder that S3 buckets support BitTorrent natively.

sramsay 6y ago

I have written several command-line weather programs for different services -- many of which I had to abandon. For example, I wrote a popular one for Weather Underground that I had to retire because WU stopped allowing free access to their API. My latest effort is one that just deals with NWS/NOAA directly, and reading this article has me wondering if part of their bandwidth trouble is a product of the way their API works. The API is a huge set of linked JSON documents. So it very often happens that a request for data is actually half-a-dozen requests through embedded URLs (all of which are made by the client one after the other). It's all very logical in some ways, but at the same time a bit byzantine. I often wonder which entity is served by the way it's set up. From my perspective, it's easily the hardest API to work with. Weather Underground was wonderful, and so was Dark Sky (which I played with for awhile). I've had people over the years ask if one or another of my tools could be used for large scale or very rapid weather data collection, but all of the services I have worked with throttle connections exactly as NOAA is proposing. I suppose I'm at least relieved to hear that they're not restricting access by independent developers!

rsync 6y ago
I'm curious - and perhaps you can help ... I hate wunderground - it is a giant, bloated mess of a website with a UI that jumps all over the place ... However, I can't find anyone else to give me a 10 day forecast. It seems like wunderground is the only place that collates it like that, which I like very much. Am I wrong ? Is there some other source for a proper, detailed, 10 day forecast ?
- nightski 6y ago
  The NWS directly works well for me. Just type in the location in the upper left. I kind of appreciate the simplicity of their website. https://forecast.weather.gov/ https://forecast.weather.gov/
  - wtallis 6y ago
    Could you be a bit more specific and demonstrate how to get a 10-day forecast out of that URL?

cratermoon 6y ago

Charge AccuWeather by the megabyte. Problem solved.

paultopia 6y ago
Seriously, though. I wonder if accuweather or someone got legislation passed to prohibit them from charging heavy users...

plantain 6y ago

I've suggested multiple times to them and even offered to help them setup HTTP BitTorrent to reduce the loads, but there's just no actual interest in solving the problem...

jfrunyon 6y ago

> According to companies that draw large amounts of this data, the proposed limit will substantially harm the services they provide to customers. The possible negative effect on forecasts has also raised concerns among congressional lawmakers. Well then maybe they should be paying for more efficient feeds instead of querying more than once per second for a sustained duration? Really? In what world did anyone think a good solution to ANY problem was "query a website over 60 times per minute continuously"???

BrandonMarc 6y ago

"key data" .... which would be a small subset of all data ... and to solve bandwidth issues they want to make it more scarce. This is a stunt. They want money, and are using their most popular (key) data points as a threat. Just like politicians. Money gets tight and they say they'll cut essential services like fire and police, but what they won't tell you is they really want their pet project funded, using deceit.

maxerickson 6y ago

Is this because they switched to the new radar view? Just throw it away and keep the old one running.

marktangotango 6y ago

Does anyone know where one can get historic wind data (avg direction and speed per hour for example) for a well known location like LAX? I was looking a few months ago and couldn’t find anything. I thought surely NOAA or NWS would have this?

Buttons840 6y ago
WeatherSpark?
brootstrap 6y ago
YEah i can tell you for 10 bucks. Try to google wind speed LAX for starters. Did you search at all? Airports have weather stations that record shit and you can get at those data.
a_square_peg 6y ago
As mentioned in the comment elsewhere in this thread, I run an API service (oikolab.com) that does this. You can get hourly data for any location from 1950 onward.

crazyjncsu 6y ago

My fault guys, that was just me ... but I can’t help if if they have the only national radar animation that isn’t infested with ads and other bloat: https://radar.weather.gov/Conus/Loop/NatLoop.gif https://radar.weather.gov/Conus/Loop/NatLoop.gif

ghthor 6y ago

Why don't they just charge their largest consumers?

ojbyrne 6y ago

John Oliver on this a year or so ago (NSFW): https://youtu.be/qMGn9T37eR8 https://youtu.be/qMGn9T37eR8

creinhardt 6y ago

As far as I know, the weather.gov api is hosted out of a single facility (the NCEP NCO), and if that building has any sort of network issue, the api goes down. At the very least it seems like using some sort of CDN would provide a bit more resiliency, but maybe there are infrastructure constraints that make this difficult.

csense 6y ago

They should publish this data over some protocol that lets users contribute upload bandwidth (BitTorrent, IPFS).