6 ms·

It's a very exciting time in weather forecasting. We're getting all kinds of new observations available, from cheap LEO satellites to smartphone sensors to smar

by cryptoz 11y ago

It's a very exciting time in weather forecasting. We're getting all kinds of new observations available, from cheap LEO satellites to smartphone sensors to smart umbrellas and everything else. And faster computers! And more parallel computers! Think about every single iPhone and every single Galaxy S in the world as a useful sensor array _and_ a parallel processor... Weather forecasting accuracy is still constrained by both the available observations and the ability to run the forecast models. This new supercomputer won't be nearly enough to process all the new data that will arrive in the next few years, so the pendulum will swing back to the bottleneck being computing power soon enough. I think the most exciting things will happen over the next 5-10 years, as we end up with another 2-3 order of magnitude increase in the available observations to the models and we'll build new supercomputers to process them. I bet we can get incredible improvements, things like reliable 2-week weather forecasts.

reubenmorais 11y ago
> Think about every single iPhone and every single Galaxy S in the world as a useful sensor array _and_ a parallel processor... Sadly, until we have a breakthrough in battery technology, all of those will be parallel processors programmed to be asleep as often as they can.
- cryptoz 11y ago
  Sure, that's true - but they can just as happily run their GPUs if they are plugged in and charging without any battery issues. Although for the moment you are right, they aren't that useful as a powerful distributed supercomputer. They ARE, however, really really good at collecting live weather data and sending that to the supercomputer to process. One step at a time.
  - nradov 11y ago
    Smart phones are good at collecting live barometric pressure data but not much else. Temperature sensors aren't very useful because they are poorly calibrated, you can't reliably tell whether someone is outdoors, and the reading can be thrown off by the heat generated by the device or the person holding it. UV sensors are seldom pointed in the right direction. And I haven't seen a phone with a wind speed or precipitation sensor yet.
    - avar 11y ago
      I genuinely don't mean to sound passive aggressive when I ask, do you know anything about this subject or are you just guessing? I don't know anything about it, but I can think of numerous ways to mitigate the issues you mention. E.g.: * Does it actually matter that the absolute value of temperature sensors is inaccurate, or is it enough that they have good relative accuracy (do they?) * Do you really need to tell if someone is indoors or outdoors given a large enough sample of users in a given area? That also seems like something you could detect with heuristics (work hours, jumps up/down in temperature). There's a lot of ways to tease useful signals from noisy data. Is it really the case that smart phone temperature sensors in the aggregate are useless as an input into weather forecasting models?
      - mrow84 11y ago
        I also know nothing about the limitations of the temperature sensors on smartphones (I don't even know of any phones that have temperature sensors). What I do know is that whilst teasing signals out of noisy data is achievable when the data are unbiased, the possibility of biases makes it significantly harder, especially when you don't know what the biases might be. This can be a significant problem with all sorts of data that you might think were fairly good, including both in-situ and satellite measurements.
      - NickNameNick 11y ago
        I'm not sure how useful the usual techniques for normalising sensors would be for moving sensors. Fixed weather stations have corrective mappings associated with them. For example the station at the airport might be assumed to be 1 degree warmer than the generalised temperature for your city. It might be known to collect 5% more rain when the wind is northerly, and 8% less for southerly. These mappings are very important, because they let you shut-down a weather station, and build a new one in a different location, and compare the data from the two. if the weather stations aren't in consistent locations, with discoverable local behaviors, then converting the raw data into generalised data about the area is going to be very difficult.
      - nradov 11y ago
        I'm not an expert but have played around with sensors on some Android devices. The absolute thermometer accuracy will vary by device. When the temperature rises by 1℃ over the course of an hour is that because it actually got warmer, or is because the user moved inside or put it in a warm jacket pocket or ran another app that caused the CPU to heat up? In theory I suppose if you had thousands of devices in a small area uploading simultaneous temperature readings you could perform a regression analysis and extract a little bit of signal from the noise. But in practice I just don't see how you could ever get enough device density in a small area to make that work. Remember that in urban areas the vast majority of smart phones spend most of their time inside climate-controlled vehicles and buildings, so any temperature readings from those are worthless for weather prediction. But they do still give useful barometric pressure data because the pressure is fairly consistent inside and outside for a given location and altitude.
kuya 11y ago
just curious, do you have references to smartphone sensors being utilized in numerical weather prediction? i'd been browsing through some WMO reports and other associated NWP lit recently and i'd seen stuff on GPS-RO, but nothing on anyone assimilating "smart" devices.
- cryptoz 11y ago
  Sure. I've been working on this problem for about 5 years now, starting with the collection of barometric pressure from Android devices. I'm currently working on this with iPhones at Sunshine [1], where we collect pressure data (along with other metrics), but pressure is the most valuable. There are researchers who use this data - we have collected about 4 billion atmospheric pressure measurements that we have distributed for academic and government research. The primary researchers are Cliff Mass and his lab at the University of Washington. There are also groups in Canada and the US that are using the data. IBM is now also collecting and using smartphone pressure data through their mobile apps. [2] Generally speaking, the current trend is to take the live data stream, run it through a quality-control algorithm and then use kalman filters in the WRF data assimilation package. There are some papers published, but it is still early. I will find some links to papers if you'd like to read them. [3] [1] https://thesunshine.co/ https://thesunshine.co/ [2] http://www.nytimes.com/2015/10/29/technology/ibm-to-acquire-the-weather-company.html http://www.nytimes.com/2015/10/29/technology/ibm-to-acquire-... [3] Utility of Dense Pressure Observations for Improving Mesoscale Analyses and Forecasts: http://www.atmos.washington.edu/~hakim/papers/madaus_hakim_mass_2013_pressure_assimilation.pdf http://www.atmos.washington.edu/~hakim/papers/madaus_hakim_m...
  - kuya 11y ago
    ah, thanks, that's fascinating. "It also learns every time you actively report to the community on sky conditions and hazards, translating this information into weather predictions." does sunshine actively run a numerical weather prediction model like WRF? it'd be great to assimilate all these extra sensors into the NWP centre's models, but i imagine things like cal/val and WMO agreements to share data might make things difficult for commercial companies?
ThomPete 11y ago
In other words. This is how the actual IoT revolution (if one can call it that) looks like.
oh_sigh 11y ago
No one in the foreseeable future is going to use cellphones as parallel processors. First of all, the communication latencies are way too large to coordinate anything usefully. Second of all, users are already upset about the lack of battery life available for their phones.
- nether 11y ago
  It's not exactly parallel processing, but the Dark Sky app already does its forecasting calculations on the iPhone: http://jackadam.github.io/2011/how-dark-sky-works/ http://jackadam.github.io/2011/how-dark-sky-works/ > The reason we encode velocity data as an image is so we can pass it off to the GPU on the iPhone and iPad. Both the storm prediction and the smooth animations are calculated on the device itself, rather than the server, and all the magic happens directly on the GPU.
  - mrow84 11y ago
    In case anyone starts wondering, in the article it says that they're assuming that atmospheric motion is linear, which they justify (reasonably) because they're only looking over short time scales. So, although it is a forecast, it's a pretty simple one ;-).

by cryptoz 11y ago

reubenmorais 11y ago

> Think about every single iPhone and every single Galaxy S in the world as a useful sensor array _and_ a parallel processor... Sadly, until we have a breakthrough in battery technology, all of those will be parallel processors programmed to be asleep as often as they can.

cryptoz 11y ago
Sure, that's true - but they can just as happily run their GPUs if they are plugged in and charging without any battery issues. Although for the moment you are right, they aren't that useful as a powerful distributed supercomputer. They ARE, however, really really good at collecting live weather data and sending that to the supercomputer to process. One step at a time.
- nradov 11y ago
  Smart phones are good at collecting live barometric pressure data but not much else. Temperature sensors aren't very useful because they are poorly calibrated, you can't reliably tell whether someone is outdoors, and the reading can be thrown off by the heat generated by the device or the person holding it. UV sensors are seldom pointed in the right direction. And I haven't seen a phone with a wind speed or precipitation sensor yet.
  - avar 11y ago
    I genuinely don't mean to sound passive aggressive when I ask, do you know anything about this subject or are you just guessing? I don't know anything about it, but I can think of numerous ways to mitigate the issues you mention. E.g.: * Does it actually matter that the absolute value of temperature sensors is inaccurate, or is it enough that they have good relative accuracy (do they?) * Do you really need to tell if someone is indoors or outdoors given a large enough sample of users in a given area? That also seems like something you could detect with heuristics (work hours, jumps up/down in temperature). There's a lot of ways to tease useful signals from noisy data. Is it really the case that smart phone temperature sensors in the aggregate are useless as an input into weather forecasting models?
    - mrow84 11y ago
      I also know nothing about the limitations of the temperature sensors on smartphones (I don't even know of any phones that have temperature sensors). What I do know is that whilst teasing signals out of noisy data is achievable when the data are unbiased, the possibility of biases makes it significantly harder, especially when you don't know what the biases might be. This can be a significant problem with all sorts of data that you might think were fairly good, including both in-situ and satellite measurements.
    - NickNameNick 11y ago
      I'm not sure how useful the usual techniques for normalising sensors would be for moving sensors. Fixed weather stations have corrective mappings associated with them. For example the station at the airport might be assumed to be 1 degree warmer than the generalised temperature for your city. It might be known to collect 5% more rain when the wind is northerly, and 8% less for southerly. These mappings are very important, because they let you shut-down a weather station, and build a new one in a different location, and compare the data from the two. if the weather stations aren't in consistent locations, with discoverable local behaviors, then converting the raw data into generalised data about the area is going to be very difficult.
    - nradov 11y ago
      I'm not an expert but have played around with sensors on some Android devices. The absolute thermometer accuracy will vary by device. When the temperature rises by 1℃ over the course of an hour is that because it actually got warmer, or is because the user moved inside or put it in a warm jacket pocket or ran another app that caused the CPU to heat up? In theory I suppose if you had thousands of devices in a small area uploading simultaneous temperature readings you could perform a regression analysis and extract a little bit of signal from the noise. But in practice I just don't see how you could ever get enough device density in a small area to make that work. Remember that in urban areas the vast majority of smart phones spend most of their time inside climate-controlled vehicles and buildings, so any temperature readings from those are worthless for weather prediction. But they do still give useful barometric pressure data because the pressure is fairly consistent inside and outside for a given location and altitude.

kuya 11y ago

just curious, do you have references to smartphone sensors being utilized in numerical weather prediction? i'd been browsing through some WMO reports and other associated NWP lit recently and i'd seen stuff on GPS-RO, but nothing on anyone assimilating "smart" devices.

cryptoz 11y ago
Sure. I've been working on this problem for about 5 years now, starting with the collection of barometric pressure from Android devices. I'm currently working on this with iPhones at Sunshine [1], where we collect pressure data (along with other metrics), but pressure is the most valuable. There are researchers who use this data - we have collected about 4 billion atmospheric pressure measurements that we have distributed for academic and government research. The primary researchers are Cliff Mass and his lab at the University of Washington. There are also groups in Canada and the US that are using the data. IBM is now also collecting and using smartphone pressure data through their mobile apps. [2] Generally speaking, the current trend is to take the live data stream, run it through a quality-control algorithm and then use kalman filters in the WRF data assimilation package. There are some papers published, but it is still early. I will find some links to papers if you'd like to read them. [3] [1] https://thesunshine.co/ https://thesunshine.co/ [2] http://www.nytimes.com/2015/10/29/technology/ibm-to-acquire-the-weather-company.html http://www.nytimes.com/2015/10/29/technology/ibm-to-acquire-... [3] Utility of Dense Pressure Observations for Improving Mesoscale Analyses and Forecasts: http://www.atmos.washington.edu/~hakim/papers/madaus_hakim_mass_2013_pressure_assimilation.pdf http://www.atmos.washington.edu/~hakim/papers/madaus_hakim_m...
- kuya 11y ago
  ah, thanks, that's fascinating. "It also learns every time you actively report to the community on sky conditions and hazards, translating this information into weather predictions." does sunshine actively run a numerical weather prediction model like WRF? it'd be great to assimilate all these extra sensors into the NWP centre's models, but i imagine things like cal/val and WMO agreements to share data might make things difficult for commercial companies?

ThomPete 11y ago

In other words. This is how the actual IoT revolution (if one can call it that) looks like.

oh_sigh 11y ago

No one in the foreseeable future is going to use cellphones as parallel processors. First of all, the communication latencies are way too large to coordinate anything usefully. Second of all, users are already upset about the lack of battery life available for their phones.

nether 11y ago
It's not exactly parallel processing, but the Dark Sky app already does its forecasting calculations on the iPhone: http://jackadam.github.io/2011/how-dark-sky-works/ http://jackadam.github.io/2011/how-dark-sky-works/ > The reason we encode velocity data as an image is so we can pass it off to the GPU on the iPhone and iPad. Both the storm prediction and the smooth animations are calculated on the device itself, rather than the server, and all the magic happens directly on the GPU.
- mrow84 11y ago
  In case anyone starts wondering, in the article it says that they're assuming that atmospheric motion is linear, which they justify (reasonably) because they're only looking over short time scales. So, although it is a forecast, it's a pretty simple one ;-).