3 ms·

It's puzzling that there's so much documentation, both about how to run it and the internals, while failing to explain what it does. What are the inputs and out

It's puzzling that there's so much documentation, both about how to run it and the internals, while failing to explain what it does. What are the inputs and outputs? It looks like it does something with satellite photos?

simonw 3y ago
After a bit of digging I found this in their FAQ on https://madewithclay.org/ https://madewithclay.org/, it was the best answer I've seen to your question: > Clay harnesses AI, satellite images, and other spatial data to organize information about what’s happening in precise locations around the world. We give Clay millions of satellite data and use the latest AI tools so it can supervise itself learning about Earth through those images. As it learns, we benchmark how those skills improve its capacity to do important tasks like creating land cover maps, detecting crops or burn scars, or predicting carbon stock.
- skybrian 3y ago
  Thanks. It sounds like the input is raw satellite photos and output might be cleaned-up maps based on the data. The page on "location embeddings" [1] seems interesting: > In CLAY, the encoder receives unmasked patches, latitude-longitude data, and timestep information. Notably, the last 2 embeddings from the encoder specifically represent the latitude-longitude and timestep embeddings. Perhaps it could figure out the latitude, longitude, and orientation from a satellite photo (geolocation). [1] https://clay-foundation.github.io/model/clay-v0-location-embeddings.html https://clay-foundation.github.io/model/clay-v0-location-emb...
- icyfox 3y ago
  Right now it seems that they've trained a true foundational model. It can recreate its training inputs, but that only matters as far as downstream performance goes. Based on a brief scanning of the repo & website: - Their training process is a self-supervised MAE to recover the original satellite images and some additional metadata. - They're trying to then use this for plastic pollution detection, deforestation, mining, etc. I'd liken this to a llama-base. It's exposed to a lot of satellite data, but to do something useful really has to be fine-tuned (or clustered in the case of unsupervised embeddings). No word yet on the performance against these specific downstream tasks.

simonw 3y ago

After a bit of digging I found this in their FAQ on https://madewithclay.org/ https://madewithclay.org/, it was the best answer I've seen to your question: > Clay harnesses AI, satellite images, and other spatial data to organize information about what’s happening in precise locations around the world. We give Clay millions of satellite data and use the latest AI tools so it can supervise itself learning about Earth through those images. As it learns, we benchmark how those skills improve its capacity to do important tasks like creating land cover maps, detecting crops or burn scars, or predicting carbon stock.

skybrian 3y ago
Thanks. It sounds like the input is raw satellite photos and output might be cleaned-up maps based on the data. The page on "location embeddings" [1] seems interesting: > In CLAY, the encoder receives unmasked patches, latitude-longitude data, and timestep information. Notably, the last 2 embeddings from the encoder specifically represent the latitude-longitude and timestep embeddings. Perhaps it could figure out the latitude, longitude, and orientation from a satellite photo (geolocation). [1] https://clay-foundation.github.io/model/clay-v0-location-embeddings.html https://clay-foundation.github.io/model/clay-v0-location-emb...
icyfox 3y ago
Right now it seems that they've trained a true foundational model. It can recreate its training inputs, but that only matters as far as downstream performance goes. Based on a brief scanning of the repo & website: - Their training process is a self-supervised MAE to recover the original satellite images and some additional metadata. - They're trying to then use this for plastic pollution detection, deforestation, mining, etc. I'd liken this to a llama-base. It's exposed to a lot of satellite data, but to do something useful really has to be fine-tuned (or clustered in the case of unsupervised embeddings). No word yet on the performance against these specific downstream tasks.