4 ms·
Data isn't the hard part here, plenty is available, even with all the necessary preprocessing.
by sinenomine 4y ago
Data isn't the hard part here, plenty is available, even with all the necessary preprocessing.
- lossolo 4y agoA large amount of data is not an issue, but obtaining a high amount of high-quality data is challenging. This is why open-source models do not perform as well as GPT-3 models in real-world usage.
- sinenomine 4y agoSorry, since the Pile and C4 https://huggingface.co/datasets/c4 https://huggingface.co/datasets/c4 (and, more generally, common crawl) and BigCode https://www.bigcode-project.org/ https://www.bigcode-project.org/ became available, this argument ceased being the real moat. The real moat is more about the lack of concentrated compute, ML engineering, and, more generally, prosaic lack of political with outside a few orgs.