4 ms·
Keep in mind, even though this is the largest geospatial model, it's still a tiny model, with only 100M parameters. I'd be excited to see what a more substanti
by cs702 3y ago
Keep in mind, even though this is the largest geospatial model, it's still a tiny model, with only 100M parameters.
I'd be excited to see what a more substantial, state-of-the-art model could do with geospatial data.
Say, a model based on something like ViT-22B, with 22 billion parameters: https://arxiv.org/abs/2302.05442 https://arxiv.org/abs/2302.05442.
- junon 3y agoI'm not disagreeing with you but just pointing out that it's kind of wild that "100M parameters" isn't considered SotA anymore. Amazing how far we've come in such a short time.
- naillo 3y ago"What about a bigger model" is basically feature creep for ML.
- lolinder 3y agoPersonally, I'm excited to see ML researchers doing cool stuff with small models again! With LLMs taking over the spotlight it's easy for people to forget that not everything needs billions or trillions of parameters. Stable Diffusion fits comfortably in my 8GB of VRAM and can generate amazing images. I'd love to see more research like this in smaller models that can be used on cheap consumer hardware.
- cs702 3y agoWe want much larger models, not because they're "cool," but because they exhibit capabilities that tiny models don't exhibit, including the ability to perform new tasks for which they were not trained, without requiring finetuning.
- lolinder 3y agoI question the assumption that fine-tuning should always be avoided. If a model is going to be used many times for a specific use case, it is far cheaper and uses far less energy to fine tune a small model once and run it on cheap low-power hardware than it is to continuously run a huge, do-everything model on expensive, high-power hardware. Enormous models are great for exploration and for general purpose applications like ChatGPT, but I think that we will find over the next few years that smaller, purpose-built models will continue to dominate in applications like geospatial analysis.
- cs702 3y agoWe're talking about different things. You're talking about finetuning models to tasks known in advance. I'm talking about the ability to generalize to new tasks: https://arxiv.org/pdf/2206.07682 https://arxiv.org/pdf/2206.07682. Please don't argue against a straw-man.
- lolinder 3y agoI'm not attacking a straw man, it seems that I just don't understand the distinction you're drawing. As I understand it we're contrasting two opposite approaches to ML: fine tuning small models for specific applications versus training a single large model that can generalize to new tasks without preparing them ahead of time. I'm arguing that in fine tuning is far more useful than people are currently giving it credit for, and that generalizing a single massive model to new tasks is overrated. Can you clarify where you're seeing a straw man?
- cs702 3y agoYou're talking about tasks known in advance; I'm not.
- lolinder 3y agoAnd I'm saying that any task that wasn't known in advance immediately becomes a task known in advance once it's done once. I'm not arguing that there is no place for large, general models—they're great for exploration—just that a smaller foundation model shouldn't be dismissed offhand based solely on parameter count.
- RosanaAnaDana 3y agoIts the largest *published* geospatial model. There are plenty of models being developed daily with more than 100m parameters. I think its just that there is a dearth of publishing on the advances in geospatial ml.