3 ms·
You probably don't need knowledge about Pokemon or the Diamond Sutra in your enterprise coding LLM. That's one of the biggest remaining head-scratchers in this
by CamperBob2 6mo ago
You probably don't need knowledge about Pokemon or the Diamond Sutra in your enterprise coding LLM.
That's one of the biggest remaining head-scratchers in this whole business. You do need all that unrelated stuff to make a good coding model.
Nobody knows why you can't build a coding model by training on nothing but code, CS texts, specifications, and case studies, but so far it appears that you can't.
- otabdeveloper4 6mo agoThis one is kind of obvious - because people prompt coding LLMs with natural language. That's unrelated to stuffing the pre-train set with trivia factoids. An LLM that knows English very well isn't actually very large and certainly not hundreds of billions of parameters.