Yes this has been considered by bloom/petals but it was eclipsed by llama-type models on single machines. But the recent 'leak' from google indicated that lora

Yes this has been considered by bloom/petals but it was eclipsed by llama-type models on single machines. But the recent 'leak' from google indicated that lora (low rank update) of model weights is enabling a de facto distributed effort. Individuals can do cheap low-rank updates fine-tuning models, and combine their efforts additively, effectively performing full-rank updates. So maybe it will be possible!