3 ms·

Own a national weather service? I want your API to not return 500 errors for hours at a time, multiple times a day... Today it was down from 9AM to 2PM, and t

Own a national weather service? I want your API to not return 500 errors for hours at a time, multiple times a day... Today it was down from 9AM to 2PM, and then from 9:30PM to 11PM. Yesterday was even worse. It isn't rate throttling because I only try to retrieve current conditions every 30 minutes. Also, the load of the API might be a lot lower if it were possible to retrieve just the current measurements, but the response you get back for observations is insanely bloated with 12 hours back of each measurement, plus a bunch of station information that is heavily duplicative of information provided about the station. Nothing about the station should be returned when you ask for the station's measurements should be returned except a link to API call for the station's information.

renewiltord 2y ago
That's fair. I assume it's this? https://www.weather.gov/documentation/services-web-api https://www.weather.gov/documentation/services-web-api Strange that they have outage information via radio. What endpoints were failing? I'll email and see if they'll accept some help.
- b112 2y ago
  Likely what's happening, is someone was put in charge with a purposeful goal of weakening public end points. Similar shenanigans happened with the FCC, and it is a modern trick to denigrating the value of public services. * First, take an excellent Government service and under-fund it by reducing its budget or hobbling it in some capacity * Ensure that the prior does not cut the budget in half, or something so overt. If you do that, your political adversaries will surely drum up support to stop you. Instead just never increase the budget. Let aging infrastructure go unfunded for upgrades and improvements, etc. Small, non-apparent cuts. * Next, put someone in charge who ensures that the service is eating money, yet its output is useless for most tasks, and unreliable. * Later, complain how the service is clearly not useful, yet "look at all this money the taxpayer pays!!", and suggest changing its scope so that it better aligns with "others providing the same service". Key to this is that you didn't cut the budget extensively, so the service is indeed very expensive without value. * Either dismantle the service entirely, or turn it into a shadow of its former self This is an ongoing game. There are those that literally go bananas, at the idea of anything, anything at all being done by Government. It is ridiculously easy to keep services up, if you're even mildly competent, and just handle things with the correct process. Process is vital here, key, how you develop services, and how you deploy and monitor and maintain them, is key. The fact that stuff "just goes down" repeatedly means that they literally have no idea what they're doing. And I suspect that's on purpose.
  - NavinF 2y ago
    https://en.wikipedia.org/wiki/Hanlon%27s_razor https://en.wikipedia.org/wiki/Hanlon%27s_razor
mikeortman 2y ago
I'm fairly sure weather.gov's api doesn't have an SLA of any kind. I highly recommend going through the push vs polling approach if possible and store the data as it comes in live in your own DB. AWS Open Data + MADIS are good sources that are... for reasons... far more stable EDIT: sorry, I misread your commend. If it's only once every 30 secs, that may be a bit overkill for your needs. You may be able to get the appropriate text product directly and parse it out. Unfortunately probably the most stable, yet clunkiest, way
parkrrr 2y ago
> Also, the load of the API might be a lot lower if it were possible to retrieve just the current measurement /stations/{stationId}/observations/latest ?
zorm 2y ago
Let's not forget the time they just turned it off for the winter holidays because they didn't want to be paged about it going down!
kmbfjr 2y ago
Recheck their API spec, you can limit observations to current or any number you want for any station. The problem however, is the 500 or target station readings returning null.
abkfenris 2y ago
They are migrating the api to a new domain and browning out the old one to get folks to switch over and figure out if there are any issues. There is also another infrastructure change going on as well. Normally those would be posted on https://www.weather.gov/notification/ https://www.weather.gov/notification/ but that’s having hiccups, but the email list for announcements still works. A few of the recent announcements: - https://www.weather.gov/media/notification/pdf_2023_24/scn24-44_website_migration_legacy_decommission_aab.pdf https://www.weather.gov/media/notification/pdf_2023_24/scn24... - https://www.weather.gov/media/notification/pdf_2023_24/scn24-50_removal_of_digital_projects_from_forecast_aaa.pdf https://www.weather.gov/media/notification/pdf_2023_24/scn24... - https://www.weather.gov/media/notification/pdf_2023_24/scn24-55_api_v1.13.pdf https://www.weather.gov/media/notification/pdf_2023_24/scn24...

renewiltord 2y ago

That's fair. I assume it's this? https://www.weather.gov/documentation/services-web-api https://www.weather.gov/documentation/services-web-api Strange that they have outage information via radio. What endpoints were failing? I'll email and see if they'll accept some help.

b112 2y ago
Likely what's happening, is someone was put in charge with a purposeful goal of weakening public end points. Similar shenanigans happened with the FCC, and it is a modern trick to denigrating the value of public services. * First, take an excellent Government service and under-fund it by reducing its budget or hobbling it in some capacity * Ensure that the prior does not cut the budget in half, or something so overt. If you do that, your political adversaries will surely drum up support to stop you. Instead just never increase the budget. Let aging infrastructure go unfunded for upgrades and improvements, etc. Small, non-apparent cuts. * Next, put someone in charge who ensures that the service is eating money, yet its output is useless for most tasks, and unreliable. * Later, complain how the service is clearly not useful, yet "look at all this money the taxpayer pays!!", and suggest changing its scope so that it better aligns with "others providing the same service". Key to this is that you didn't cut the budget extensively, so the service is indeed very expensive without value. * Either dismantle the service entirely, or turn it into a shadow of its former self This is an ongoing game. There are those that literally go bananas, at the idea of anything, anything at all being done by Government. It is ridiculously easy to keep services up, if you're even mildly competent, and just handle things with the correct process. Process is vital here, key, how you develop services, and how you deploy and monitor and maintain them, is key. The fact that stuff "just goes down" repeatedly means that they literally have no idea what they're doing. And I suspect that's on purpose.
- NavinF 2y ago
  https://en.wikipedia.org/wiki/Hanlon%27s_razor https://en.wikipedia.org/wiki/Hanlon%27s_razor

mikeortman 2y ago

I'm fairly sure weather.gov's api doesn't have an SLA of any kind. I highly recommend going through the push vs polling approach if possible and store the data as it comes in live in your own DB. AWS Open Data + MADIS are good sources that are... for reasons... far more stable EDIT: sorry, I misread your commend. If it's only once every 30 secs, that may be a bit overkill for your needs. You may be able to get the appropriate text product directly and parse it out. Unfortunately probably the most stable, yet clunkiest, way

parkrrr 2y ago

> Also, the load of the API might be a lot lower if it were possible to retrieve just the current measurement /stations/{stationId}/observations/latest ?

zorm 2y ago

Let's not forget the time they just turned it off for the winter holidays because they didn't want to be paged about it going down!

kmbfjr 2y ago

Recheck their API spec, you can limit observations to current or any number you want for any station. The problem however, is the 500 or target station readings returning null.

abkfenris 2y ago

They are migrating the api to a new domain and browning out the old one to get folks to switch over and figure out if there are any issues. There is also another infrastructure change going on as well. Normally those would be posted on https://www.weather.gov/notification/ https://www.weather.gov/notification/ but that’s having hiccups, but the email list for announcements still works. A few of the recent announcements: - https://www.weather.gov/media/notification/pdf_2023_24/scn24-44_website_migration_legacy_decommission_aab.pdf https://www.weather.gov/media/notification/pdf_2023_24/scn24... - https://www.weather.gov/media/notification/pdf_2023_24/scn24-50_removal_of_digital_projects_from_forecast_aaa.pdf https://www.weather.gov/media/notification/pdf_2023_24/scn24... - https://www.weather.gov/media/notification/pdf_2023_24/scn24-55_api_v1.13.pdf https://www.weather.gov/media/notification/pdf_2023_24/scn24...