5 ms·

I ran a weather site for a year or so (that also showed YESTERDAY'S weather, that was the key feature) using visual crossing's data, until someone started scrap

I ran a weather site for a year or so (that also showed YESTERDAY'S weather, that was the key feature) using visual crossing's data, until someone started scraping all the cities of the world every hour and running up my costs (visual crossing is pay-per-data-point) so I had to shut it down. It surprises me that in 2025 we can't just support global free weather data as some kind of cooperative service. It's not like it's high-bandwidth or even all that high-volume.

dawnerd 1y ago
There is public free weather, for now at least, where do you think the private weather companies source it from? They’re just repackaging it.
- jasonthorsness 1y ago
  The last time I had checked (this was over a year ago) there was nothing that provided world-wide data in a consistent format except the paid APIs, of which only one had a good historical data service. I could switch my site to just use NWS and be US only I suppose; better than just being completely off. Adding all the weather services of every country in the world is too much for me (and why I guess the paid services are value-add).
  - Ueland 1y ago
    The Norwegian MET has had it for years, for free. https://api.met.no/ https://api.met.no/ They also provide yr.no which is widely used worldwide.
    - neurostimulant 1y ago
      gnome use this in their weather app.
  - phillipseamore 1y ago
    https://open-meteo.com/ https://open-meteo.com/
    - jasonthorsness 1y ago
      Looks promising, thanks!
      - jasonthorsness 1y ago
        Update: this worked; I switched to Open Meteo and brought back the site (also bot protections are on and seem to be blocking some stuff so hopefully that keeps things under the Open Meteo limit.)
tempestn 1y ago
Fantastic feature btw; certainly more than once I've wished weather apps/sites would show recent history.
- Leftium 1y ago
  I wanted the same thing, so I built: https://weather-sense.leftium.com https://weather-sense.leftium.com - I even blogged about this over 10 years ago: https://blog.leftium.com/2013/12/how-to-display-temperature-properly.html https://blog.leftium.com/2013/12/how-to-display-temperature-... - Previous version used to show two historical days: https://github.com/Leftium/ultra-weather https://github.com/Leftium/ultra-weather - The current version is capable of showing the previous 90 days. I plan to add a weekly overview with two previous days. BTW the https://open-meteo.com https://open-meteo.com API supports getting historical and forecast data in a single call. All other API's require at least two calls.
whalesalad 1y ago
Your statements are somewhat contradictory. You said that you were being scraped so you had to shut the service down due to costs... but then you also say that this not a high-bandwidth or high-volume service.
- jasonthorsness 1y ago
  I was the only regular user (it's basically a POC/concept site), so it was extremely low volume except for the scraper(s), who suddenly became high-volume. I could have done more research into scraper/bot blocking but I didn't have the time and needed to respond to the scrapers immediately, so I pulled the plug. EDIT: I have also since learned of Vercel's bot/scraper blocking features so I'm also going to try turning those on and see if it stops the scraping.
  - nox_pp 1y ago
    For https://luxweather.com https://luxweather.com I detect bots/scrapers by user agent and just serve them a fake forecast, brought our bill way down :)
    - jasonthorsness 1y ago
      Ha that's a great idea; proactive anti-bot solutions like that are hilarious
Leftium 1y ago
My app also shows yesterday's weather: https://weather-sense.leftium.com https://weather-sense.leftium.com - It uses open-meteo's API. Before open-meteo, I tried with OpenWeatherMap's API. - However, besides requiring an additional call for each previous day's data, OpenWeatherMap's One Call API only supported local time zones for current and forecast data. So depending on the timezone and time, there could be a big gap in the data that needed to be filled with timezone math (and extra API calls. The original DarkSky One Call API did not have this issue.)

dawnerd 1y ago

There is public free weather, for now at least, where do you think the private weather companies source it from? They’re just repackaging it.

jasonthorsness 1y ago
The last time I had checked (this was over a year ago) there was nothing that provided world-wide data in a consistent format except the paid APIs, of which only one had a good historical data service. I could switch my site to just use NWS and be US only I suppose; better than just being completely off. Adding all the weather services of every country in the world is too much for me (and why I guess the paid services are value-add).
- Ueland 1y ago
  The Norwegian MET has had it for years, for free. https://api.met.no/ https://api.met.no/ They also provide yr.no which is widely used worldwide.
  - neurostimulant 1y ago
    gnome use this in their weather app.
- phillipseamore 1y ago
  https://open-meteo.com/ https://open-meteo.com/
  - jasonthorsness 1y ago
    Looks promising, thanks!
    - jasonthorsness 1y ago
      Update: this worked; I switched to Open Meteo and brought back the site (also bot protections are on and seem to be blocking some stuff so hopefully that keeps things under the Open Meteo limit.)

tempestn 1y ago

Fantastic feature btw; certainly more than once I've wished weather apps/sites would show recent history.

Leftium 1y ago
I wanted the same thing, so I built: https://weather-sense.leftium.com https://weather-sense.leftium.com - I even blogged about this over 10 years ago: https://blog.leftium.com/2013/12/how-to-display-temperature-properly.html https://blog.leftium.com/2013/12/how-to-display-temperature-... - Previous version used to show two historical days: https://github.com/Leftium/ultra-weather https://github.com/Leftium/ultra-weather - The current version is capable of showing the previous 90 days. I plan to add a weekly overview with two previous days. BTW the https://open-meteo.com https://open-meteo.com API supports getting historical and forecast data in a single call. All other API's require at least two calls.

whalesalad 1y ago

Your statements are somewhat contradictory. You said that you were being scraped so you had to shut the service down due to costs... but then you also say that this not a high-bandwidth or high-volume service.

jasonthorsness 1y ago
I was the only regular user (it's basically a POC/concept site), so it was extremely low volume except for the scraper(s), who suddenly became high-volume. I could have done more research into scraper/bot blocking but I didn't have the time and needed to respond to the scrapers immediately, so I pulled the plug. EDIT: I have also since learned of Vercel's bot/scraper blocking features so I'm also going to try turning those on and see if it stops the scraping.
- nox_pp 1y ago
  For https://luxweather.com https://luxweather.com I detect bots/scrapers by user agent and just serve them a fake forecast, brought our bill way down :)
  - jasonthorsness 1y ago
    Ha that's a great idea; proactive anti-bot solutions like that are hilarious

Leftium 1y ago

My app also shows yesterday's weather: https://weather-sense.leftium.com https://weather-sense.leftium.com - It uses open-meteo's API. Before open-meteo, I tried with OpenWeatherMap's API. - However, besides requiring an additional call for each previous day's data, OpenWeatherMap's One Call API only supported local time zones for current and forecast data. So depending on the timezone and time, there could be a big gap in the data that needed to be filled with timezone math (and extra API calls. The original DarkSky One Call API did not have this issue.)