3 ms·
small players will never have a chance to push the state of the art, as whatever optimization there is will also be applied at large scale with more money
by make3 4y ago
small players will never have a chance to push the state of the art, as whatever optimization there is will also be applied at large scale with more money
- cypress66 4y agoA lot of SOTA comes from small players. It just isn't the case for LLMs.
- awestroke 4y agoGood point, but perhaps a leap could take small players into territories of language models that are large enough to be useful. GPT-3 crossed that threshold
- hankman86 4y agoTake a leaf from Seti@Home‘s book and try to come up with a distributed, volunteer-based approach to training an open source LLM. There is already an enormous amount of suitable ML hardware on end user devices.
- Der_Einzige 4y agoHuggingface actually recently did this, but I think it's for inference on their giant models like BLOOM