4 ms·
>Garbage in garbage out, and there is a ton of garbage in the Falcon/Llama (and OpenAI?) datasets. It feels like such a waste of compute and parameter space. T
by famouswaffles 3y ago
>Garbage in garbage out, and there is a ton of garbage in the Falcon/Llama (and OpenAI?) datasets. It feels like such a waste of compute and parameter space.
This is true and most researchers understand this on some level (https://arxiv.org/abs/2305.07759 https://arxiv.org/abs/2305.07759) but understanding it doesn't really make the problem any easier. How do you curate a general purpose model's dataset without throwing the baby out with the bathwater ?
EDIT: This paper seems to take a good stab at the question.
https://arxiv.org/abs/2309.04564 https://arxiv.org/abs/2309.04564
- brucethemoose2 3y ago> How do you curate a general purpose model's dataset without throwing the baby out with the bathwater Time, money, and human work. The pruning process needs focused input from experts other than ML researchers and data scientists.
- mycall 3y agoMaybe the Cyc team could provide some generalization tips.
- waste_monk 3y ago>How do you curate a general purpose model's dataset without throwing the baby out with the bathwater ? Maybe we could train an AI to do it ;)