3 ms·
Technically the original BERT was mostly trained on Wikipedia (with some potentially-copyrighted books in BookCorpora)
by minimaxir 3y ago
Technically the original BERT was mostly trained on Wikipedia (with some potentially-copyrighted books in BookCorpora)
- simonw 3y agoYeah that's what I was trying to hint at when I said "useful" - I'm looking for a general purpose LLM, GPT-3 level or higher. (BERT was useful but not GPT-3 useful.)
- eli 3y agoWikipedia content is also copyrighted, and it’s unlikely that an LLM reproducing it is compatible with its license terms.