6 ms·
In case someone is looking for historical weather data for ML training and prediction, I created an open-source weather API which continuously archives weather
by meteo-jeff 3y ago
In case someone is looking for historical weather data for ML training and prediction, I created an open-source weather API which continuously archives weather data.
Using past and forecast data from multiple numerical weather models can be combined using ML to achieve better forecast skill than any individual model. Because each model is physically bound, the resulting ML model should be stable.
See: https://open-meteo.com https://open-meteo.com
- boxed 3y agoOpen-Meteo has a great API too. I used it to build my iOS weather app Frej (open source and free: https://github.com/boxed/frej https://github.com/boxed/frej) It was super easy and the responses are very fast.
- mdbmdb 3y agoIs it able to provide data on extreme events. Say, the current and potential path of a hurricane? similar to .kml that NOAA provides
- meteo-jeff 3y agoExtreme weather is predicted by numerical weather models. Correctly representing hurricanes has driven development on the NOAA GFS model for centuries. Open-Meteo focuses on providing access to weather data for single locations or small areas. If you look at data for coastal areas, forecast and past weather data will show severe winds. Storm tracks or maps are not available, but might be implemented in the future.
- mdbmdb 3y agoAppreciate the response. Do you know of any services that provide what I described in the previous comments? I'm specifically interested in extreme weather conditions and their visual representation (hurricanes, tornados, hails etc.) with API capabilities
- swells34 3y agoGo to: nhc.noaa.gov/gis There's a list of data and products with kmls and kmzs and geojsons and all sorts of stuff. I haven't actually used the API for retrieving these, but NOAA has a pretty solid track record with data dissemination.
- dmd 3y agoI would love to hear about this centuries-old NOAA GFS model. The one I know about definitely doesn't have that kind of history behind it.
- K2h 3y agoSome of the oldest data may come from ships logs back to 1836 https://www.reuters.com/graphics/CLIMATE-CHANGE-ICE-SHIPLOGS/0100B4QE2FC/index.html https://www.reuters.com/graphics/CLIMATE-CHANGE-ICE-SHIPLOGS...
- meteo-jeff 3y agoSorry, decades. KML files for storm tracks are still the best way to go. You could calculate storm tracks yourself for other weather models like DWD ICON, ECMWF IFS or MeteoFrance ARPEGE, but storm tracks based on GFS ensembles are easy to use with sufficient accuracy
- comment_ran 3y agoHow about https://pirateweather.net/en/latest/ https://pirateweather.net/en/latest/ ? Does anyone have a compare this API with the latest API we have here?
- meteo-jeff 3y agoBoth APIs use weather models from NOAA GFS and HRRR, providing accurate forecasts in North America. HRRR updates every hour, capturing recent showers and storms in the upcoming hours. PirateWeather gained popularity last year as a replacement for the Dark Sky API when Dark Sky servers were shut down. With Open-Meteo, I'm working to integrate more weather models, offering access not only to current forecasts but also past data. For Europe and South-East Asia, high-resolution models from 7 different weather services improve forecast accuracy compared to global models. The data covers not only common weather variables like temperature, wind, and precipitation but also includes information on wind at higher altitudes, solar radiation forecasts, and soil properties. Using custom compression methods, large historical weather datasets like ERA5 are compressed from 20 TB to 4 TB, making them accessible through a time-series API. All data is stored in local files; no database set-up required. If you're interested in creating your own weather API, Docker images are provided, and you can download open data from NOAA GFS or other weather models.
- Fatnino 3y agoIs there somewhere to see historical forecasts? So not "the weather on 25 December 2022 was such and such" but rather "on 20 December 2022 the forecast for 25 December 2022 was such and such"
- meteo-jeff 3y agoNot yet, but I am working towards it: https://github.com/open-meteo/open-meteo/issues/206 https://github.com/open-meteo/open-meteo/issues/206
- berniedurfee 3y agoI’ve always wanted to see something like that. I always wonder if forecasts are a coin flip beyond a window of a few hours.
- CSMastermind 3y agoI know at a minimum that hurricane forecasts have gotten significantly better over time. We can now https://www.nhc.noaa.gov/verification/verify5.shtml https://www.nhc.noaa.gov/verification/verify5.shtml Our 96 hour projections are as accurate today as the 24 hour projections were in 1990.
- mjs 3y agoLooks like https://sites.research.google/weatherbench/ https://sites.research.google/weatherbench/ attempts to "benchmark" different forecast models/systems. They're very cautious about naming a "best" model though! > Weather forecasting is a multi-faceted problem with a variety of use cases. No single metric fits all those use cases. Therefore,it is important to look at a number of different metrics and consider how the forecast will be applied.
- rrr_oh_man 3y agoThat last paragraph sounds like something ChatGPT would write.
- Auracle 3y ago
- Vagantem 3y agoThat’s awesome! I’ve hooked something similar up to my service - https://dropory.com https://dropory.com which predicts which day it will rain the least for any location Based on historical data!
- polygamous_bat 3y agoYikes, after completed three steps I was asked for my email. No to your bait and switch, thanks!
- Vagantem 3y agoIt can take up to 10 min to generate a report - I had a spinner before but people just left the page. So I implemented a way to send it to them instead. I’ve never used the emails for anything else than that. Try it with a 10 min disposable email address if you like. Thanks for your feedback!
- polygamous_bat 3y agoOk, seems like your UI is not coming from a place of malice. However, pulling out an email input form at the final step is a very widespread UI dark pattern, so if nothing else please let people know that you will ask their email before they start interacting with your forms.
- willsmith72 3y agothis is really cool, I've been looking for good snow-related weather APIs for my business. I tried looking on the site, but how does it work, being coordinates-based? I'm used to working with different weather stations, e.g. seeing different snowfall prediction at the bottom of a mountain, halfway up, and at the top, where the coordinates are quite similar.
- ryanlitalien 3y agoYou'll need a local weather expert to assist, as terrain, geography and other hyper-local factors create forecasting unpredictability. For example, Jay Peak in VT has its own weather, the road in has no snow, but it's a raging snowstorm on the mountain.
- just_testing 3y agoI was going to ask about air quality, but just opened the site and you have air quality as well! Thanks!
- _visgean 3y agoThere is also https://github.com/google-research/weatherbench2 https://github.com/google-research/weatherbench2 which has baselines of numerical weather models.
- Omnipresent 3y agoThis is great. I am very curious about the architectural decisions you've taken here. Is there a blog post / article about them? 80 yrs of historical data -- are you storing that somewhere in PG and the APIs are just fetching it? If so, what indices have you set up to make APIs fetch faster etc. I just fetched 1960 to 2022 in about 12 secs.
- meteo-jeff 3y agoTraditional database systems struggle to handle gridded data efficiently. Using PG with time-based indices is memory and storage extensive. It works well for a limited number of locations, but global weather models at 9-12 km resolution have 4 to 6 million grid-cells. I am exploiting on the homogeneity of gridded data. In a 2D field, calculating the data position for a graphical coordinate is straightforward. Once you add time as a third dimension, you can pick any timestamp at any point on earth. To optimize read speed, all time steps are stored sequentially on disk in a rotated/transposed OLAP cube. Although the data now consists of millions of floating-point values without accompanying attributes like timestamps or geographical coordinates, the storage requirements are still high. Open-Meteo chunks data into small portions, each covering 10 locations and 2 weeks of data. Each block is individually compressed using an optimized compression scheme. While this process isn't groundbreaking and is supported by file systems like NetCDF, Zarr, or HDF5, the challenge lies in efficiently working with multiple weather models and updating data with each new weather model run every few hours. You can find more information here: https://openmeteo.substack.com/i/64601201/how-data-are-stored-and-made-available-through-api https://openmeteo.substack.com/i/64601201/how-data-are-store...
- caseyf7 3y agoHow did you handle missing data? I’ve used NOAA data a few times and I’m always surprised at how many days of historical data are missing. They have also stopped recording in certain locations and then start in new locations over time making it hard to get solid historical weather information.
- Guestmodinfo 3y agoI always suspect that they don't tell me the actual temperature. Maybe I am totally wrong but I suspect. I need to get my own physical thermometer not the digital one in my room and outside my house and have a camera focussed on it. So that later I can speed up the video and see how much the weather varied the previous night.
- kubiton 3y agoWhat? Why?
- 3abiton 3y agoAre multiple data sources supported?
- brna 3y agoHi Jeff, Great work, Respect! I just hit the daily limit on the second request at https://climate-api.open-meteo.com/v1/climate https://climate-api.open-meteo.com/v1/climate I see the limit for non-commercial use should be "less than 10.000 daily API calls". Technically 2 is less than 10.000, I know, but still I decided to drop you a comment. :)
- wodenokoto 3y ago10.000 requests / (24 hours * 60 minutes * 60 seconds) = 0.11 requests / second or 1 request every ~9 seconds. Maybe you just didn't space them enough.
- brna 3y agoMaybe, that would be funny. ~7 requests per minute would be a more dev-friendly way of enforcing the same quota.
- tomaskafka 3y agoI confirm, open-meteo is awesome and has a great API (and API playground!). And is the only source I know to offer 2 weeks of hourly forecasts (I understand at that point they are more likely to just show a general trend, but it still looks spectacular). It's a pleasure being able to use it in https://weathergraph.app https://weathergraph.app
- brahbrah 3y ago> And is the only source I know to offer 2 weeks of hourly forecasts Enjoy the data directly from the source producing them. American weather agency: https://www.nco.ncep.noaa.gov/pmb/products/gfs/ https://www.nco.ncep.noaa.gov/pmb/products/gfs/ European weather agency: https://www.ecmwf.int/en/forecasts/datasets/open-data https://www.ecmwf.int/en/forecasts/datasets/open-data The data’s not necessarily east to work with, but it’s all there, and you get all the forecast ensembles (potential forecasted weather paths) too
- tomaskafka 3y agoThank you, I didn't know! I'd love to, but I'd need another 24 hours in a day to also process the data - I'm glad I can build on a work of others and use the friendly APIs :).
- aaarrm 3y agoThis is awesome. I was trying to do a weather project a while ago, but couldn't find an API to suit my needs for the life of me. It looks like yours still doesn't have exactly everything I'd want but it still has plenty. Mainly UV index is something I've been trying to find wide historical data for, but it seems like it just might not be out there. I do see you have solar radiation, so I wonder if I could calculate it using that data. But I believe UV index also takes into account things like local air pollution and ozone forecast as well.