6 ms·

Hopefully IBM can fix the weather data industry. It seems ridiculously broken right now that if I want to get a "prediction" of how much it's going to rain next

Hopefully IBM can fix the weather data industry. It seems ridiculously broken right now that if I want to get a "prediction" of how much it's going to rain next week there are a million free sources, but if I want to see how much it actually rained in the past week then I need to pay thousands of dollars or else figure out how to parse a bunch of obscure and undocumented formats.

huhtenberg 11y ago
Wolfram|Alpha has the data you are looking for.
- feider 11y ago
  ...and operates like North Korea when it comes to openness.
jonknee 11y ago
Wunderground has lots of data available. They may have actually been part of this deal as they were owned by The Weather Channel. http://www.wunderground.com/history/airport/KSFO/2015/10/28/MonthlyCalendar.html?req_city=San%20Francisco&req_state=CA&reqdb.zip=94101&reqdb.magic=1&reqdb.wmo=99999#calendar http://www.wunderground.com/history/airport/KSFO/2015/10/28/...
- ISL 11y ago
  International Business Machines on Wednesday announced an agreement to acquire The Weather Co.'s B2B, mobile, and cloud-based web properties, including WSI, weather.com, Weather Underground and The Weather Company brand. Source: http://www.cnbc.com/2015/10/28/ibm-plans-to-acquire-the-weather-companys-product-and-tech-businesses.html http://www.cnbc.com/2015/10/28/ibm-plans-to-acquire-the-weat...
- hangtime79 11y ago
  Wunderground is included in the deal. http://www-03.ibm.com/press/us/en/pressrelease/47952.wss http://www-03.ibm.com/press/us/en/pressrelease/47952.wss
  - free2rhyme214 11y ago
    Aww I love this app except their ads. I'm not sure what IBM's plans for this is.
- acomjean 11y ago
  I use weather underground for my personal server (developer accounts are free up to 500 calls a day...). We used them for work to get weather data for lots of locations hourly ("drizzle" plan). Very reliable. api: http://www.wunderground.com/weather/api/d/docs http://www.wunderground.com/weather/api/d/docs National weather service, http://weather.gov http://weather.gov is my goto weather forecaster, but at the time I wrote the code, getting hourly data based on lat/long for several thousand locations (some outside the US) was problematic. and XML..
- rdtsc 11y ago
  Part of IBM now as well.
- Redoubts 11y ago
  That doesn't look free, unless you want to scrape. http://www.wunderground.com/weather/api/d/pricing.html http://www.wunderground.com/weather/api/d/pricing.html
- rconti 11y ago
  That's what I love about wunderground. I actually have an Ambient Weather ws-1001-wifi weather station at my house, and I'm part of wunderground's weather station network. Whenever I pop open the weather app on my iPhone, it shows the weather in my backyard which I love.
deuill 11y ago
Have you tried using http://forecast.io/ http://forecast.io/? You can get historical forecasts via the JSON API[0], which is, from what I can tell, free for low-volume use. [0]: https://developer.forecast.io/docs/v2#time_call https://developer.forecast.io/docs/v2#time_call
- tertius 11y ago
  I think he's trying to get historical data of actual weather, not forecasts.
  - drewbug 11y ago
    > Please note that we only store the best data we have for a given location and time: in the past, this will usually be observations from weather stations (though we may fall back to forecasted data if we don't have any observations)
- Redoubts 11y ago
  That looks really nice.
cwal37 11y ago
Hey buddy, I use this resource[1] all the time (and they're actually housed into the building next to me here at work). It's the long-term climate records from monitoring stations across the country, the United States Historical Climatology Network, and you can get daily data going back over a hundred years in some cases. For example, I've used it to crank out random temperature histograms in the past[2], and even plotted some long-term averages to decide on temperature and precipitation ranges when going on camping trips. It's really useful if you're willing to work with the data a tiny bit. You're getting csv files, so it's not terrible to work with in any way via Python or R, or whichever data-friendly language/tool you want to use. [1] http://cdiac.ornl.gov/epubs/ndp/ushcn/ushcn_map_interface.html http://cdiac.ornl.gov/epubs/ndp/ushcn/ushcn_map_interface.ht... [2] http://btus.us/decadal-histograms-of-monthly-yosemite-temperatures/ http://btus.us/decadal-histograms-of-monthly-yosemite-temper...
- Alex3917 11y ago
  So apparently in order to use this I need to enter a NESDIS ID. When I Google that, the first result I get informs me that a NESDIS ID is "an 8-character identifier assigned by NESDIS that becomes the RAWS identification number for use with the GOES satellite." Clicking on 'glossary' in the breadcrumbs then fails to take me to an actual glossary. I realize that it's theoretically possible to get this working. But it's not easy. What I want is a way to enter a postal code and get a JSON response with the precipitation for the last week. Without having to convert to scientology, learn GeoDjango, license GIS software, read an 800 page manual on parsing a bytestream from a satellite, etc.
  - cwal37 11y ago
    Um, maybe to use it in an commercial setting? I've literally no idea what you're talking about as I've never come across that before. I've always just pulled down the station stuff I need by hand (since you can get all the station's data at once), or by some lightly mechanized webscraping. Your comment sounded significantly more chill than what it seems like you might need. It wouldn't be terribly difficult to mechanize this and get all the data at once, but I thought you wanted to occasionally see how much it rained, not build a service out of this. Does this page[1] not contain what you need? [1] http://cdiac.ornl.gov/ftp/ushcn_daily/ http://cdiac.ornl.gov/ftp/ushcn_daily/
subnaught 11y ago
For the most part, the Weather Company is merely redistributing data collected by the federal government, specifically the National Oceanic and Atmospheric Administration. You can easily access historical rainfall data here: http://www.nws.noaa.gov/oh/hads/ http://www.nws.noaa.gov/oh/hads/
chinathrow 11y ago
Those formats are very well documented and standardised world wide. Commercial use is another thing though...
dublinben 11y ago
The National Weather Service makes historical weather data available for free in a standardized format.[0] I'm sure there's already a parser written for any language you might be using. [0]http://w2.weather.gov/climate/index.php?wfo=mtr http://w2.weather.gov/climate/index.php?wfo=mtr
- lelandbatey 11y ago
  That's true, but that data is for only a few locations.
vegardx 11y ago
You can actually get all that for free from Norwegian Meteorological Institute, both forecasts and observations. The only caveat is that it's all in Norwegian. Looks like that is a little intentional[1]. [1] http://om.yr.no/verdata/free-weather-data/ http://om.yr.no/verdata/free-weather-data/
arikrak 11y ago
I once found a link that showed useful charts for historical weather data including averages and highs and lows. I can't find it right now, but Googling for it brings up a similar site: https://weatherspark.com https://weatherspark.com
exelius 11y ago
I wouldn't be too hopeful; this situation is the entire reason IBM made this purchase. IBM is in the business of providing data about a whole lot of things. I would expect the cost for access to this data to go up, not down -- the majority people who care about how much it rained in the last week only care because they can make more money by knowing that.
- hangtime79 11y ago
  I couldn't disagree more. I bet we actually see a service exposed via Bluemix in the not distant future.
Keats 11y ago
I scraped http://en.tutiempo.net/climate http://en.tutiempo.net/climate when I was writing the following article: https://blog.wearewizards.io/comparing-the-weather-of-places-ive-lived-in https://blog.wearewizards.io/comparing-the-weather-of-places... There is also a link to the python script I made to scrape it
PretzelFisch 11y ago
There is Noaa http://w2.weather.gov/climate/ http://w2.weather.gov/climate/ you can get past results there.
vram22 11y ago
Check out products like ForecastWatch from Intellovations. Not sure, but might be of help. Not affiliated, just had conversations with the founder a while ago. http://www.intellovations.com/products/ http://www.intellovations.com/products/
feider 11y ago
Sorry for non value adding cynicism, but when has a big corp fixed any industry? Especially IBM? Sure, more competition is welcome, but I would not get my hopes up.
alphataraxian 11y ago
Free quality controlled historical data available from NOAA as well: https://www.ncdc.noaa.gov/data-access/land-based-station-data/land-based-datasets/quality-controlled-local-climatological-data-qclcd https://www.ncdc.noaa.gov/data-access/land-based-station-dat...
anderiv 11y ago
Weather Underground makes nearly all climatological data available in easy-to-consume formats (csv mostly). Using your rainfall example, here's how I'd get a listing of rainfall by day for the month of September, from my local National Weather Service monitoring site: $ curl -s "http://www.wunderground.com/history/airport/KMSP/2015/09/31/MonthlyHistory.html?format=1" | grep -v Precip | cut -d "," -f 20

huhtenberg 11y ago

Wolfram|Alpha has the data you are looking for.

feider 11y ago
...and operates like North Korea when it comes to openness.

jonknee 11y ago

Wunderground has lots of data available. They may have actually been part of this deal as they were owned by The Weather Channel. http://www.wunderground.com/history/airport/KSFO/2015/10/28/MonthlyCalendar.html?req_city=San%20Francisco&req_state=CA&reqdb.zip=94101&reqdb.magic=1&reqdb.wmo=99999#calendar http://www.wunderground.com/history/airport/KSFO/2015/10/28/...

ISL 11y ago
International Business Machines on Wednesday announced an agreement to acquire The Weather Co.'s B2B, mobile, and cloud-based web properties, including WSI, weather.com, Weather Underground and The Weather Company brand. Source: http://www.cnbc.com/2015/10/28/ibm-plans-to-acquire-the-weather-companys-product-and-tech-businesses.html http://www.cnbc.com/2015/10/28/ibm-plans-to-acquire-the-weat...
hangtime79 11y ago
Wunderground is included in the deal. http://www-03.ibm.com/press/us/en/pressrelease/47952.wss http://www-03.ibm.com/press/us/en/pressrelease/47952.wss
- free2rhyme214 11y ago
  Aww I love this app except their ads. I'm not sure what IBM's plans for this is.
acomjean 11y ago
I use weather underground for my personal server (developer accounts are free up to 500 calls a day...). We used them for work to get weather data for lots of locations hourly ("drizzle" plan). Very reliable. api: http://www.wunderground.com/weather/api/d/docs http://www.wunderground.com/weather/api/d/docs National weather service, http://weather.gov http://weather.gov is my goto weather forecaster, but at the time I wrote the code, getting hourly data based on lat/long for several thousand locations (some outside the US) was problematic. and XML..
rdtsc 11y ago
Part of IBM now as well.
Redoubts 11y ago
That doesn't look free, unless you want to scrape. http://www.wunderground.com/weather/api/d/pricing.html http://www.wunderground.com/weather/api/d/pricing.html
rconti 11y ago
That's what I love about wunderground. I actually have an Ambient Weather ws-1001-wifi weather station at my house, and I'm part of wunderground's weather station network. Whenever I pop open the weather app on my iPhone, it shows the weather in my backyard which I love.

deuill 11y ago

Have you tried using http://forecast.io/ http://forecast.io/? You can get historical forecasts via the JSON API[0], which is, from what I can tell, free for low-volume use. [0]: https://developer.forecast.io/docs/v2#time_call https://developer.forecast.io/docs/v2#time_call

tertius 11y ago
I think he's trying to get historical data of actual weather, not forecasts.
- drewbug 11y ago
  > Please note that we only store the best data we have for a given location and time: in the past, this will usually be observations from weather stations (though we may fall back to forecasted data if we don't have any observations)
Redoubts 11y ago
That looks really nice.

cwal37 11y ago

Hey buddy, I use this resource[1] all the time (and they're actually housed into the building next to me here at work). It's the long-term climate records from monitoring stations across the country, the United States Historical Climatology Network, and you can get daily data going back over a hundred years in some cases. For example, I've used it to crank out random temperature histograms in the past[2], and even plotted some long-term averages to decide on temperature and precipitation ranges when going on camping trips. It's really useful if you're willing to work with the data a tiny bit. You're getting csv files, so it's not terrible to work with in any way via Python or R, or whichever data-friendly language/tool you want to use. [1] http://cdiac.ornl.gov/epubs/ndp/ushcn/ushcn_map_interface.html http://cdiac.ornl.gov/epubs/ndp/ushcn/ushcn_map_interface.ht... [2] http://btus.us/decadal-histograms-of-monthly-yosemite-temperatures/ http://btus.us/decadal-histograms-of-monthly-yosemite-temper...

Alex3917 11y ago
So apparently in order to use this I need to enter a NESDIS ID. When I Google that, the first result I get informs me that a NESDIS ID is "an 8-character identifier assigned by NESDIS that becomes the RAWS identification number for use with the GOES satellite." Clicking on 'glossary' in the breadcrumbs then fails to take me to an actual glossary. I realize that it's theoretically possible to get this working. But it's not easy. What I want is a way to enter a postal code and get a JSON response with the precipitation for the last week. Without having to convert to scientology, learn GeoDjango, license GIS software, read an 800 page manual on parsing a bytestream from a satellite, etc.
- cwal37 11y ago
  Um, maybe to use it in an commercial setting? I've literally no idea what you're talking about as I've never come across that before. I've always just pulled down the station stuff I need by hand (since you can get all the station's data at once), or by some lightly mechanized webscraping. Your comment sounded significantly more chill than what it seems like you might need. It wouldn't be terribly difficult to mechanize this and get all the data at once, but I thought you wanted to occasionally see how much it rained, not build a service out of this. Does this page[1] not contain what you need? [1] http://cdiac.ornl.gov/ftp/ushcn_daily/ http://cdiac.ornl.gov/ftp/ushcn_daily/

subnaught 11y ago

For the most part, the Weather Company is merely redistributing data collected by the federal government, specifically the National Oceanic and Atmospheric Administration. You can easily access historical rainfall data here: http://www.nws.noaa.gov/oh/hads/ http://www.nws.noaa.gov/oh/hads/

chinathrow 11y ago

Those formats are very well documented and standardised world wide. Commercial use is another thing though...

dublinben 11y ago

The National Weather Service makes historical weather data available for free in a standardized format.[0] I'm sure there's already a parser written for any language you might be using. [0]http://w2.weather.gov/climate/index.php?wfo=mtr http://w2.weather.gov/climate/index.php?wfo=mtr

lelandbatey 11y ago
That's true, but that data is for only a few locations.

vegardx 11y ago

You can actually get all that for free from Norwegian Meteorological Institute, both forecasts and observations. The only caveat is that it's all in Norwegian. Looks like that is a little intentional[1]. [1] http://om.yr.no/verdata/free-weather-data/ http://om.yr.no/verdata/free-weather-data/

arikrak 11y ago

I once found a link that showed useful charts for historical weather data including averages and highs and lows. I can't find it right now, but Googling for it brings up a similar site: https://weatherspark.com https://weatherspark.com

exelius 11y ago

I wouldn't be too hopeful; this situation is the entire reason IBM made this purchase. IBM is in the business of providing data about a whole lot of things. I would expect the cost for access to this data to go up, not down -- the majority people who care about how much it rained in the last week only care because they can make more money by knowing that.

hangtime79 11y ago
I couldn't disagree more. I bet we actually see a service exposed via Bluemix in the not distant future.

Keats 11y ago

I scraped http://en.tutiempo.net/climate http://en.tutiempo.net/climate when I was writing the following article: https://blog.wearewizards.io/comparing-the-weather-of-places-ive-lived-in https://blog.wearewizards.io/comparing-the-weather-of-places... There is also a link to the python script I made to scrape it

PretzelFisch 11y ago

There is Noaa http://w2.weather.gov/climate/ http://w2.weather.gov/climate/ you can get past results there.

vram22 11y ago

Check out products like ForecastWatch from Intellovations. Not sure, but might be of help. Not affiliated, just had conversations with the founder a while ago. http://www.intellovations.com/products/ http://www.intellovations.com/products/

feider 11y ago

Sorry for non value adding cynicism, but when has a big corp fixed any industry? Especially IBM? Sure, more competition is welcome, but I would not get my hopes up.

alphataraxian 11y ago

Free quality controlled historical data available from NOAA as well: https://www.ncdc.noaa.gov/data-access/land-based-station-data/land-based-datasets/quality-controlled-local-climatological-data-qclcd https://www.ncdc.noaa.gov/data-access/land-based-station-dat...

anderiv 11y ago

Weather Underground makes nearly all climatological data available in easy-to-consume formats (csv mostly). Using your rainfall example, here's how I'd get a listing of rainfall by day for the month of September, from my local National Weather Service monitoring site: $ curl -s "http://www.wunderground.com/history/airport/KMSP/2015/09/31/MonthlyHistory.html?format=1" | grep -v Precip | cut -d "," -f 20