4 ms·

I've worked in "climate intelligence" for many years. The list overlooks one of the largest and most immediate opportunities around that market: the data infras

I've worked in "climate intelligence" for many years. The list overlooks one of the largest and most immediate opportunities around that market: the data infrastructure and analysis tools we have today are profoundly unfit for purpose. Just about everyone is essentially using cartography tools to do large-scale spatiotemporal analysis of sensor and telemetry data. The gaps for both features and practical scalability are massive. It has made most of the climate intelligence analysis we'd like to do, and for which data is available, intractable. And what we can do is so computationally inefficient that we figuratively burn down a small forest every time we run an analysis on a non-trivial model, which isn't very green either. (This is definitely something I'd work on if I had the bandwidth, it is a pretty pure deep tech software problem.)

babelfish 4y ago
Where could I read more about this problem and how it's being tackled today?
- tony_cannistra 4y ago
  I hesitate to link to Twitter here, but Joe Morrison has his finger on the pulse of this and offers a tongue-in-cheek perspective that I appreciate: https://twitter.com/mouthofmorrison https://twitter.com/mouthofmorrison
  - 2devnull 4y ago
    Thanks for posting. His substack piece on the 3 m’s is very appropriate.
a_square_peg 4y ago
Out of curiosity, is accessing & working with large datasets a problem in your areas of work? I run a weather/climate site that makes some of this less painful, taking datasets such as GFS or ERA5/ERA5-Land much faster to access. We have some enterprise clients who really value the time-saving aspect of this but I also feel like everyone has their own data-processing set up and problems are different for everyone.
- jandrewrogers 4y ago
  There are a couple issues I see with basic access and working with large datasets. Ease of access for typical users is also a valid issue. First, we still mostly move the data to the computation when we should be moving the computation to the data. Moving the data works fine when data is small but if the data volumes are large (as sensor/geo data tends to be) then it can take an incredibly long time to move the data. In many cases, more time is spent shoveling data over the network than actually doing the computation. This has become worse as storage density has increased, hundreds of TB/server is ordinary. Second, the data is rarely organized in a way that makes it efficient to extract arbitrary subsets. There is still a lot of what is essentially "grep at scale" going on. Again, not a problem if the data is small but if I need a specific 50TB subset of a 10PB source, this becomes prohibitively slow. The data needs to be organized such that we can slice and dice it with high selectivity in place, much more like a proper database and less of a distributed filesystem. Because spatiotemporal analysis tends to involve iterative join-like operations, you want this to be efficient as possible. The other big problem is many of these data sources are too large for everyone to have their own copy. Or if they did have their own copy, it would be extraordinarily wasteful. This is adjacent to the first issue. EDIT: And herein is the likely business model.
  - counters 4y ago
    Want to make sure you're familiar with the Pangeo community: www.pangeo.io I don't think any of these challenges are "solved", but there's a groundswell of technology that is well-situated to make a big impact in these domains. The largest barriers that still remain are the ownership of engineering processes/workflow to transform larger gridded datasets to ARCO (analysis-ready, cloud-optimized) formats, as well as tooling to mediate between heterogeneous datasets (e.g. combining regular vs irregular or arbitrarily gridded data, such as land surveys or ZIP codes). There are definitely players in the space working on these, but much is left to be done here.
    - a_square_peg 4y ago
      +1 for Pangeo. We use these toolsets heavily (Xarray, Zarr, Dask) to run our service, which is essentially what you described as taking the larger gridded datasest to ARCO format. I think this is still a bit too heavy for casual Excel/GIS analysts so we try to make it as simple as possible for them to get climate data in CSV or NetCDF format for their work.
bmitc 4y ago
I would also love any references to existing companies, research groups, or the problems in this space if you have the time to share. I found the posted list underwhelming and more of a marketing shotgun approach to try and take advantage of the push for "climate tech" but not solve any real problems.
- tony_cannistra 4y ago
  Not OP, but in my experience: Jupiter Intelligence (https://jupiterintel.com/ https://jupiterintel.com/) Descartes Labs (https://descarteslabs.com/ https://descarteslabs.com/) Microsoft Planetary Computer (https://planetarycomputer.microsoft.com/ https://planetarycomputer.microsoft.com/) Coiled (managed Dask - python HPC) (https://www.coiled.io/ https://www.coiled.io/) CarbonPlan (https://carbonplan.org/ https://carbonplan.org/) Salo Sciences (acquired by Planet, satellite imagery company) (https://salo.ai/ https://salo.ai/) lots of others
  - bmitc 4y ago
    Thank you!
tony_cannistra 4y ago
This is an excellent point. I think the problem is that because it's such a pure software problem it doesn't have an immediate "climate tech" alignment, so it stands to "dilute" these kinds of calls for funding.
tdaltonc 4y ago
Companies with good climate intelligence tech tend to evolve in to marketplaces because that gets them closer to the money. Climate projects can't afford SaaS, but offset buyers are willing to pay a premium for offsets re-verified by high-tech climate intelligence SaaS. Example: Pachama https://pachama.com/ https://pachama.com/
2devnull 4y ago
I guess it would help to be more specific about how this differs from some of the measurement related startups they list. Taxonomies are difficult, so maybe they do need an entirely separate category, or enlarge the one related to measurement.
PakG1 4y ago
> Just about everyone is essentially using cartography tools to do large-scale spatiotemporal analysis of sensor and telemetry data. The gaps for both features and practical scalability are massive. Could you point to any readings or resources that would explain these gaps? I'd be quite curious why our current spatiotemporal analysis techniques would be insufficient. Is it the analysis tools that just need new techniques or is the problem at the source (i.e. the sensors)? Or?
- Demiurge 4y ago
  There aren’t really any problems. Some people see that because all of this data can be spatially and temporarily referenced, it can be accessed better, as if there could be one format and one application that could let you do anything you want across the space time, regardless of the source. The reality is that a lots of models using this data are developed against a specific sensor or dataset, and just don’t work or scale. I don’t think this will be solved in this domain by pangeo or any startup in particular. There is an awesome new STACspec that every geo company should be adopting, and that’s the direction to move towards. But each step towards standardizing the GIS process will require something that truly everyone can adopt, sort of like JSON.
dheera 4y ago
Would you mind elaborating on say a few specific asks for tools climate people would want to have, that are low-hanging fruit that people might be able to write in their spare time? I'm very interested in doing something for climate change but I'd like to know what people want.
worldsayshi 4y ago
What potential customers are there for climate intelligence systems?
- jandrewrogers 4y ago
  Don't think of it as a "climate intelligence" system. It is really an "analyze complex dynamics in the physical world at global scale" system, one application of which is climate intelligence. Everything is fundamentally anchored to space and time, and connected by it. The applicability cuts across industries. It would enable managing many kinds of risks that have nothing to do with climate.
  - worldsayshi 4y ago
    Makes sense. Although to have such a big scope and understand how to satisfy such cross cutting concerns where tools aren't built yet it sounds like you you kind of need to understand a lot about everything. I do see that when you really understand the problem you can build such a tool in a non-domain specific way as you describe without inconceivable effort but understanding up to that point is kind of the whole problem? Edit: Btw, really sounds like a space where I would like to build stuff in. I kind of understand what you're aiming for but I feel like it's very easy to jump to conclusions and build air castles.
emmelaich 4y ago
We need sensors for carbon presence over distance and time in the ocean. At huge scale, to test the viability of various carbon sequestration schemes. That's pretty expensive, with a large non-software component. I know this from a peripheral involvement in one the XPRIZE projects.
rabernat 4y ago
Agree 100%. This is big part of the motivation behind our new startup Earthmover: https://earthmover.io/ https://earthmover.io/ Our mission is to make it easier to work with scientific data at scale in the cloud, focusing mainly on the climate, weather, and geospatial vertical. My cofounder Joe Hamman and I are climate scientists who helped create the Pangeo project. We are also core devs on the Python packages Xarray and Zarr. We think that a layer of managed services (think a "modern data stack" oriented around the multidimensional array data model) is exactly what this ecosystem needs to make it easier for teams to build data-intensive products in the climate-tech space. And we're hiring! https://earthmover.io/posts/earthmover-is-hiring/ https://earthmover.io/posts/earthmover-is-hiring/
- smaddox 4y ago
  I'm curious, have you considered using PyTorch or JAX for tensor processing? ML libraries seem to be much further along when it comes to performing compute-intensive, hardware-accelerated operations on Tensor's. And you get gradients basically for free (in terms of developer time). Also, the kernel compiler being added PyTorch 2 looks very promising.
  - rabernat 4y ago
    PyTorch and JAX are used heavily in climate science on the ML side. For more general analytics, not so much. Many of our users like to use Xarray as a high-level API. There has been some work to integrate Xarray with PyTorch (https://github.com/pydata/xarray/issues/3232 https://github.com/pydata/xarray/issues/3232) but we're not there yet. The Python Array API standard should help align these different back-ends: https://data-apis.org/array-api/latest/ https://data-apis.org/array-api/latest/
  - counters 4y ago
    The primary issue in this domain is not compute - it's I/O, especially when you need to perform complex computations with intermediate data that doesn't fit into memory.
- darkmarmot 4y ago
m0llusk 4y ago
Many lessons to be learned from Climate Corporation. They used satellite data and drones to make robust crop insurance and ended up being bought by Monsanto and changing their direction.

babelfish 4y ago

Where could I read more about this problem and how it's being tackled today?

tony_cannistra 4y ago
I hesitate to link to Twitter here, but Joe Morrison has his finger on the pulse of this and offers a tongue-in-cheek perspective that I appreciate: https://twitter.com/mouthofmorrison https://twitter.com/mouthofmorrison
- 2devnull 4y ago
  Thanks for posting. His substack piece on the 3 m’s is very appropriate.

a_square_peg 4y ago

Out of curiosity, is accessing & working with large datasets a problem in your areas of work? I run a weather/climate site that makes some of this less painful, taking datasets such as GFS or ERA5/ERA5-Land much faster to access. We have some enterprise clients who really value the time-saving aspect of this but I also feel like everyone has their own data-processing set up and problems are different for everyone.

jandrewrogers 4y ago
There are a couple issues I see with basic access and working with large datasets. Ease of access for typical users is also a valid issue. First, we still mostly move the data to the computation when we should be moving the computation to the data. Moving the data works fine when data is small but if the data volumes are large (as sensor/geo data tends to be) then it can take an incredibly long time to move the data. In many cases, more time is spent shoveling data over the network than actually doing the computation. This has become worse as storage density has increased, hundreds of TB/server is ordinary. Second, the data is rarely organized in a way that makes it efficient to extract arbitrary subsets. There is still a lot of what is essentially "grep at scale" going on. Again, not a problem if the data is small but if I need a specific 50TB subset of a 10PB source, this becomes prohibitively slow. The data needs to be organized such that we can slice and dice it with high selectivity in place, much more like a proper database and less of a distributed filesystem. Because spatiotemporal analysis tends to involve iterative join-like operations, you want this to be efficient as possible. The other big problem is many of these data sources are too large for everyone to have their own copy. Or if they did have their own copy, it would be extraordinarily wasteful. This is adjacent to the first issue. EDIT: And herein is the likely business model.
- counters 4y ago
  Want to make sure you're familiar with the Pangeo community: www.pangeo.io I don't think any of these challenges are "solved", but there's a groundswell of technology that is well-situated to make a big impact in these domains. The largest barriers that still remain are the ownership of engineering processes/workflow to transform larger gridded datasets to ARCO (analysis-ready, cloud-optimized) formats, as well as tooling to mediate between heterogeneous datasets (e.g. combining regular vs irregular or arbitrarily gridded data, such as land surveys or ZIP codes). There are definitely players in the space working on these, but much is left to be done here.
  - a_square_peg 4y ago
    +1 for Pangeo. We use these toolsets heavily (Xarray, Zarr, Dask) to run our service, which is essentially what you described as taking the larger gridded datasest to ARCO format. I think this is still a bit too heavy for casual Excel/GIS analysts so we try to make it as simple as possible for them to get climate data in CSV or NetCDF format for their work.

bmitc 4y ago

I would also love any references to existing companies, research groups, or the problems in this space if you have the time to share. I found the posted list underwhelming and more of a marketing shotgun approach to try and take advantage of the push for "climate tech" but not solve any real problems.

tony_cannistra 4y ago
Not OP, but in my experience: Jupiter Intelligence (https://jupiterintel.com/ https://jupiterintel.com/) Descartes Labs (https://descarteslabs.com/ https://descarteslabs.com/) Microsoft Planetary Computer (https://planetarycomputer.microsoft.com/ https://planetarycomputer.microsoft.com/) Coiled (managed Dask - python HPC) (https://www.coiled.io/ https://www.coiled.io/) CarbonPlan (https://carbonplan.org/ https://carbonplan.org/) Salo Sciences (acquired by Planet, satellite imagery company) (https://salo.ai/ https://salo.ai/) lots of others
- bmitc 4y ago
  Thank you!

tony_cannistra 4y ago

This is an excellent point. I think the problem is that because it's such a pure software problem it doesn't have an immediate "climate tech" alignment, so it stands to "dilute" these kinds of calls for funding.

tdaltonc 4y ago

Companies with good climate intelligence tech tend to evolve in to marketplaces because that gets them closer to the money. Climate projects can't afford SaaS, but offset buyers are willing to pay a premium for offsets re-verified by high-tech climate intelligence SaaS. Example: Pachama https://pachama.com/ https://pachama.com/

2devnull 4y ago

I guess it would help to be more specific about how this differs from some of the measurement related startups they list. Taxonomies are difficult, so maybe they do need an entirely separate category, or enlarge the one related to measurement.

PakG1 4y ago

> Just about everyone is essentially using cartography tools to do large-scale spatiotemporal analysis of sensor and telemetry data. The gaps for both features and practical scalability are massive. Could you point to any readings or resources that would explain these gaps? I'd be quite curious why our current spatiotemporal analysis techniques would be insufficient. Is it the analysis tools that just need new techniques or is the problem at the source (i.e. the sensors)? Or?

Demiurge 4y ago
There aren’t really any problems. Some people see that because all of this data can be spatially and temporarily referenced, it can be accessed better, as if there could be one format and one application that could let you do anything you want across the space time, regardless of the source. The reality is that a lots of models using this data are developed against a specific sensor or dataset, and just don’t work or scale. I don’t think this will be solved in this domain by pangeo or any startup in particular. There is an awesome new STACspec that every geo company should be adopting, and that’s the direction to move towards. But each step towards standardizing the GIS process will require something that truly everyone can adopt, sort of like JSON.

dheera 4y ago

Would you mind elaborating on say a few specific asks for tools climate people would want to have, that are low-hanging fruit that people might be able to write in their spare time? I'm very interested in doing something for climate change but I'd like to know what people want.

worldsayshi 4y ago

What potential customers are there for climate intelligence systems?

jandrewrogers 4y ago
Don't think of it as a "climate intelligence" system. It is really an "analyze complex dynamics in the physical world at global scale" system, one application of which is climate intelligence. Everything is fundamentally anchored to space and time, and connected by it. The applicability cuts across industries. It would enable managing many kinds of risks that have nothing to do with climate.
- worldsayshi 4y ago
  Makes sense. Although to have such a big scope and understand how to satisfy such cross cutting concerns where tools aren't built yet it sounds like you you kind of need to understand a lot about everything. I do see that when you really understand the problem you can build such a tool in a non-domain specific way as you describe without inconceivable effort but understanding up to that point is kind of the whole problem? Edit: Btw, really sounds like a space where I would like to build stuff in. I kind of understand what you're aiming for but I feel like it's very easy to jump to conclusions and build air castles.

emmelaich 4y ago

We need sensors for carbon presence over distance and time in the ocean. At huge scale, to test the viability of various carbon sequestration schemes. That's pretty expensive, with a large non-software component. I know this from a peripheral involvement in one the XPRIZE projects.

rabernat 4y ago

Agree 100%. This is big part of the motivation behind our new startup Earthmover: https://earthmover.io/ https://earthmover.io/ Our mission is to make it easier to work with scientific data at scale in the cloud, focusing mainly on the climate, weather, and geospatial vertical. My cofounder Joe Hamman and I are climate scientists who helped create the Pangeo project. We are also core devs on the Python packages Xarray and Zarr. We think that a layer of managed services (think a "modern data stack" oriented around the multidimensional array data model) is exactly what this ecosystem needs to make it easier for teams to build data-intensive products in the climate-tech space. And we're hiring! https://earthmover.io/posts/earthmover-is-hiring/ https://earthmover.io/posts/earthmover-is-hiring/

smaddox 4y ago
I'm curious, have you considered using PyTorch or JAX for tensor processing? ML libraries seem to be much further along when it comes to performing compute-intensive, hardware-accelerated operations on Tensor's. And you get gradients basically for free (in terms of developer time). Also, the kernel compiler being added PyTorch 2 looks very promising.
- rabernat 4y ago
  PyTorch and JAX are used heavily in climate science on the ML side. For more general analytics, not so much. Many of our users like to use Xarray as a high-level API. There has been some work to integrate Xarray with PyTorch (https://github.com/pydata/xarray/issues/3232 https://github.com/pydata/xarray/issues/3232) but we're not there yet. The Python Array API standard should help align these different back-ends: https://data-apis.org/array-api/latest/ https://data-apis.org/array-api/latest/
- counters 4y ago
  The primary issue in this domain is not compute - it's I/O, especially when you need to perform complex computations with intermediate data that doesn't fit into memory.
darkmarmot 4y ago

m0llusk 4y ago

Many lessons to be learned from Climate Corporation. They used satellite data and drones to make robust crop insurance and ended up being bought by Monsanto and changing their direction.