7 ms·

Using Machine Learning to “Nowcast” Precipitation in High Resolution

72 points by luthfur 7y ago

luthfur 7y ago
A physics-free approach to precipitation nowcasting using a CNN.
signa11 7y ago
dup: https://news.ycombinator.com/item?id=22051889 https://news.ycombinator.com/item?id=22051889
- dang 7y ago
  On HN, it's not a dupe if a story hasn't had significant attention yet. See the FAQ: https://news.ycombinator.com/newsfaq.html https://news.ycombinator.com/newsfaq.html. Allowing reposts in such cases is a way of mitigating the randomness of /newest and giving good stories multiple cracks at the bat. If you submit a lot of good stories (which you have!) it evens out in the long run. By the way, from the IDs you can tell that this submission was earlier than yours. It made the front page later because we put it in the second-chance pool (described at https://news.ycombinator.com/item?id=11662380 https://news.ycombinator.com/item?id=11662380).
  - signa11 7y ago
    ah got it thanks !
labster 7y ago
I don't get why the put nowcast in quotes. Nowcasting is pretty common term in meteorology. It's also cool that they could get this far without physics. Of course, "HRRR model begins to outperform our current results when the prediction horizon reaches roughly 5 to 6 hours". Simple associations that could be discovered by neural networks work in the short term, but the atmospheric physics is needed to understand the long term evolution of storms.
- franch 7y ago
  Also, physics-based models (Numerical weather predictions NWP) assimilate much more data than just the last few radars scan. The assimilation phase takes a couple of hours, but when the physics kicks in, the NWP models win hands down.
  - labster 7y ago
    I've been out of the field for about a decade now. So how often are they running data assimilation into models now? It was basically just 0Z and 12Z back then, because it's computationally intensive. (And of course, you only have radiosondes every 12 hours, and satellites only get you so far.) As are model runs. Like, of course NWP models win. It's not just the physics, it's all of the other assimilated obs that are advecting over your area of interest. But it's the problem is always always computation time, to an extent that most people on HN won't get. Maybe the finance guys. But you have to process a mountain of new data, then run the model very quickly for it to be any use at all to the public. We should use reanalysis for nowcasting, that would be super accurate. /s
- geoalchimista 7y ago
  I think this blog article did not frame the novelty of this study very well. It is not the nowcast that is the problem, it is getting nowcast at a spatial resolution higher than weather forecast models. But in that case, the limitation is more about computation rather than physics. What their U-Net does seems to be a fancy way of doing statistical downscaling. Needless to say, this should not be interpreted as machine learning can replace physics in weather forecast or operational meteorology (see Lorenz 1963 paper). It shines as a great data assimilation technique though.
franch 7y ago
I chime in on this because I'm working on deep learning for precipitation nowcasting using radar for my Ph.D. I was very excited when google released the press statement at NeurIPS about their work in this area. Unfortunately, after reading the paper, I have to say that their approach is fairly basic. Basically they threshold the precipitation in 4 thresholds (no rain, light, medium, heavy) and then use a U-Net like architecture, treating it as a classification problem. I think that the works of Shi et al are much more interesting in this regard: https://papers.nips.cc/paper/5955-convolutional-lstm-network-a-machine-learning-approach-for-precipitation-nowcasting.pdf https://papers.nips.cc/paper/5955-convolutional-lstm-network... http://papers.nips.cc/paper/7145-deep-learning-for-precipitation-nowcasting-a-benchmark-and-a-new-model.pdf http://papers.nips.cc/paper/7145-deep-learning-for-precipita... What I think is that Google wanted to use a lighter model that can be applied to the whole continental US. I expect them to integrate this in google assistant, like: "hey google, tell me when it's going to rain"
londons_explore 7y ago
In the UK, netweather.tv shows weather radar with a 5 minute temporal resolution, 50 meter spacal resolution, and a 10 minute latency (less if you pay I think). I regularly use it 'by eye' to predict when a big band of rain is coming. I can very effectively figure out if there will be more or less rain 5 minutes from now. "Shall I walk to the car now, or should I wait 5 mins?". In comparison, the results of this work seem disappointing.
m3kw9 7y ago
Now cast, let me look outside
denimboy 7y ago
Sounds a lot like darksky.net they have an interesting story starting from kickstarter to getting acquired by longnow.

72 points by luthfur 7y ago

luthfur 7y ago

A physics-free approach to precipitation nowcasting using a CNN.

signa11 7y ago

dup: https://news.ycombinator.com/item?id=22051889 https://news.ycombinator.com/item?id=22051889

dang 7y ago
On HN, it's not a dupe if a story hasn't had significant attention yet. See the FAQ: https://news.ycombinator.com/newsfaq.html https://news.ycombinator.com/newsfaq.html. Allowing reposts in such cases is a way of mitigating the randomness of /newest and giving good stories multiple cracks at the bat. If you submit a lot of good stories (which you have!) it evens out in the long run. By the way, from the IDs you can tell that this submission was earlier than yours. It made the front page later because we put it in the second-chance pool (described at https://news.ycombinator.com/item?id=11662380 https://news.ycombinator.com/item?id=11662380).
- signa11 7y ago
  ah got it thanks !

labster 7y ago

I don't get why the put nowcast in quotes. Nowcasting is pretty common term in meteorology. It's also cool that they could get this far without physics. Of course, "HRRR model begins to outperform our current results when the prediction horizon reaches roughly 5 to 6 hours". Simple associations that could be discovered by neural networks work in the short term, but the atmospheric physics is needed to understand the long term evolution of storms.

franch 7y ago
Also, physics-based models (Numerical weather predictions NWP) assimilate much more data than just the last few radars scan. The assimilation phase takes a couple of hours, but when the physics kicks in, the NWP models win hands down.
- labster 7y ago
  I've been out of the field for about a decade now. So how often are they running data assimilation into models now? It was basically just 0Z and 12Z back then, because it's computationally intensive. (And of course, you only have radiosondes every 12 hours, and satellites only get you so far.) As are model runs. Like, of course NWP models win. It's not just the physics, it's all of the other assimilated obs that are advecting over your area of interest. But it's the problem is always always computation time, to an extent that most people on HN won't get. Maybe the finance guys. But you have to process a mountain of new data, then run the model very quickly for it to be any use at all to the public. We should use reanalysis for nowcasting, that would be super accurate. /s
geoalchimista 7y ago
I think this blog article did not frame the novelty of this study very well. It is not the nowcast that is the problem, it is getting nowcast at a spatial resolution higher than weather forecast models. But in that case, the limitation is more about computation rather than physics. What their U-Net does seems to be a fancy way of doing statistical downscaling. Needless to say, this should not be interpreted as machine learning can replace physics in weather forecast or operational meteorology (see Lorenz 1963 paper). It shines as a great data assimilation technique though.

franch 7y ago

I chime in on this because I'm working on deep learning for precipitation nowcasting using radar for my Ph.D. I was very excited when google released the press statement at NeurIPS about their work in this area. Unfortunately, after reading the paper, I have to say that their approach is fairly basic. Basically they threshold the precipitation in 4 thresholds (no rain, light, medium, heavy) and then use a U-Net like architecture, treating it as a classification problem. I think that the works of Shi et al are much more interesting in this regard: https://papers.nips.cc/paper/5955-convolutional-lstm-network-a-machine-learning-approach-for-precipitation-nowcasting.pdf https://papers.nips.cc/paper/5955-convolutional-lstm-network... http://papers.nips.cc/paper/7145-deep-learning-for-precipitation-nowcasting-a-benchmark-and-a-new-model.pdf http://papers.nips.cc/paper/7145-deep-learning-for-precipita... What I think is that Google wanted to use a lighter model that can be applied to the whole continental US. I expect them to integrate this in google assistant, like: "hey google, tell me when it's going to rain"

londons_explore 7y ago

In the UK, netweather.tv shows weather radar with a 5 minute temporal resolution, 50 meter spacal resolution, and a 10 minute latency (less if you pay I think). I regularly use it 'by eye' to predict when a big band of rain is coming. I can very effectively figure out if there will be more or less rain 5 minutes from now. "Shall I walk to the car now, or should I wait 5 mins?". In comparison, the results of this work seem disappointing.

m3kw9 7y ago

Now cast, let me look outside

denimboy 7y ago

Sounds a lot like darksky.net they have an interesting story starting from kickstarter to getting acquired by longnow.