8 ms·

> You may have read hundres of books in your life, maybe even thousands. You haven't read a million. You don't need to. Sometimes online forums are the only pl

> You may have read hundres of books in your life, maybe even thousands. You haven't read a million. You don't need to. Sometimes online forums are the only place where you can find solutions for niche situations and edge cases. Tricks which would have been very difficult to figure out on your own. LLMs can train on the official documentation of tools l/libraries but they can't experiment and figure out solutions to weird problems which are unfortunately very common in tech industry. If people stop sharing such solutions with others, it might become a big problem.

skydhash 2y ago
> Sometimes online forums are the only place where you can find solutions for niche situations and edge cases. That's the most valuable aspect of it. When you find yourself in these niches situations, it's nice when you see someone has encountered it and has done the legwork to solve it, saving you hours and days. And that's why Wikis like the Arch Wiki are important. You need people to document the system, not just individual components.
- genr8 2y ago
  the Gentoo Wiki is also high on my list, actually first because Gentoo is my preferred choice, equal in terms of useful content, only second in userbase - to Arch. Gentoo also implemented a policy prohibiting people from contributing LLM output. Theres not a culty rule-following edit-sanitization mod team, and the wiki doesnt require special permission, just the honor system, but so far its enough to make me like contributing to it is a form of charity to help future mankind.
simonw 2y ago
"LLMs can train on the official documentation of tools l/libraries but they can't experiment and figure out solutions to weird problems" LLMs train on way more than just the official documentation: they train on the code itself, the unit tests for that code (which, for well written projects, cover all sorts of undocumented edge-based) and - for popular projects - thousands of examples of that library being used (and unit tested) "in the wild". This is why LLMs are so effective at helping figure out edge-cases for widely used libraries. The best coding LLMs are also trained on additional custom examples written by humans who were paid to build proprietary training data for those LLMs. I suspect they are increasingly trained on artificially created examples which have been validated (to a certain extent) through executing that code before adding it to the training data. That's a unique advantage for code - it's a lot harder to "verify" non-code generated prose since you can't execute that and see if you get an error.
- 0x1ceb00da 2y ago
  > they train on the code itself, the unit tests for that code If understanding the code was enough, we wouldn't have any bugs or counterintuitive behaviors. > and - for popular projects - thousands of examples of that library being used (and unit tested) "in the wild". If people stopped contributing to forums, we won't have any such data for new things that are being made.
  - simonw 2y ago
    The examples I'm talking about come from openly licensed code in sources like GitHub, not from StackOverflow. I would argue that code in GitHub is much more useful, because it's presented in the context of a larger application and is also more likely to work.

skydhash 2y ago

> Sometimes online forums are the only place where you can find solutions for niche situations and edge cases. That's the most valuable aspect of it. When you find yourself in these niches situations, it's nice when you see someone has encountered it and has done the legwork to solve it, saving you hours and days. And that's why Wikis like the Arch Wiki are important. You need people to document the system, not just individual components.

genr8 2y ago
the Gentoo Wiki is also high on my list, actually first because Gentoo is my preferred choice, equal in terms of useful content, only second in userbase - to Arch. Gentoo also implemented a policy prohibiting people from contributing LLM output. Theres not a culty rule-following edit-sanitization mod team, and the wiki doesnt require special permission, just the honor system, but so far its enough to make me like contributing to it is a form of charity to help future mankind.

simonw 2y ago

"LLMs can train on the official documentation of tools l/libraries but they can't experiment and figure out solutions to weird problems" LLMs train on way more than just the official documentation: they train on the code itself, the unit tests for that code (which, for well written projects, cover all sorts of undocumented edge-based) and - for popular projects - thousands of examples of that library being used (and unit tested) "in the wild". This is why LLMs are so effective at helping figure out edge-cases for widely used libraries. The best coding LLMs are also trained on additional custom examples written by humans who were paid to build proprietary training data for those LLMs. I suspect they are increasingly trained on artificially created examples which have been validated (to a certain extent) through executing that code before adding it to the training data. That's a unique advantage for code - it's a lot harder to "verify" non-code generated prose since you can't execute that and see if you get an error.

0x1ceb00da 2y ago
> they train on the code itself, the unit tests for that code If understanding the code was enough, we wouldn't have any bugs or counterintuitive behaviors. > and - for popular projects - thousands of examples of that library being used (and unit tested) "in the wild". If people stopped contributing to forums, we won't have any such data for new things that are being made.
- simonw 2y ago
  The examples I'm talking about come from openly licensed code in sources like GitHub, not from StackOverflow. I would argue that code in GitHub is much more useful, because it's presented in the context of a larger application and is also more likely to work.