3 ms·
Show HN: Ranking weather models by how their forecasts turned out
Weather apps all claim to be accurate but never show their work. So I built a scoreboard that checks: it takes the forecasts each model published (ECMWF, GFS, ICON, AIFS and others, plus Apple Weather, Foreca, OpenWeather and Visual Crossing), waits for the weather to happen, and scores temperature, wind and rain against observations.
A few things that surprised me:
- AIFS performs very well, yet almost no commercial apps give you access to it or uses it in their blend (I suspect some do without disclosing it tho)
- Foreca scores surprisingly well compared to other apps and raw models
- ICON is very accurate around the mediterranean, but performs quite poor everywhere else
There's also a history page that scores each model back through its full archive (about 5 years for GFS) to see if forecasts have actually gotten better.
It's a static page and open-source: https://github.com/NickLeenders/verisky-scoreboard https://github.com/NickLeenders/verisky-scoreboard. Public models are scored in your browser against Open-Meteo's archive. Commercial scores come as small aggregates from my server, because those providers' terms don't allow redistributing raw forecasts.
It powers an app that does the same per location in more detail, link is on the page.
Happy to answer questions about the scoring method.
- AlexRenders 2mo agovery cool project. I imagine there is a commercial market with the weather polymarket betting. I would love to see a legend, or some guidance on comprehended your data. I will be self promotional here: my project, slickfast, is free and open source. SlickFast is all about visualizing data, its super powerful and made for agentic workflows. It could buttress your project and won't cost anything to try it out, or free if you meet AGPL. I'd love to talk to you about it.
- nickoteeno 2mo agoThanks! I haven't heard about weather polymarket betting, sounds fun. I will have a look at slickfast, do I understand correctly it's an mcp?
- roryirvine 2mo agoReally interesting! Might be useful to expand the full name of each model somewhere (perhaps with a one para description and/or or a link). On the trends tab, would it be worth zooming in a little? There's less value in showing 2021-24 since only GFS data is included for those years.
- nickoteeno 2mo agoAdding a link would be fair indeed! On the zooming in, you're right, I thought I did that, and I did for all but temperature, which is default, I'll change that right away
- phillipseamore 2mo agoSo this doesn't need to be hourly. Use the ECMWF IFS 00 and 12 as observations only. RMSE really favors average models, AI models are bad... but look good in statistics because they are so average. Look at the output of AI models visually on a map and they are just a blurry mess. In my area AIFS and AIGFS get the top results but the actual best performing model (because I'm a weather nerd) ends up in third place. There also seems to be some issue with short and long run models in you comparison. I also see a few models on the list that don't cover my area so it looks like you are not querying for just the specific models but the model+ECMWF fill.
- nickoteeno 2mo agoFair point, what would you recommend instead of RSME (for temp and wind as rain uses F1 anyways)? I query the seamless variant yes, which for longer lead times replaces with ECMWF IFS. I'll take a good look at that. Which would you normally say is the best for your area?
- nickoteeno 2mo ago[dead]