4 ms·
I was thinking about this when reading an article on HN a couple weeks ago, "LLMs Aren’t Just “Trained On the Internet” Anymore" [1] (HN discussion [2]). The t
by SamBam 2y ago
I was thinking about this when reading an article on HN a couple weeks ago, "LLMs Aren’t Just “Trained On the Internet” Anymore" [1] (HN discussion [2]).
The thesis was that it won't matter if LLMs exhaust the available corpus of text that humanity has generated thus far, because it will always be easy to pay humans to create an inexhaustible supply of more text.
The article suggested that it was all going to be PhDs, lawyers, and poets creating this training data, but that all sounded like hand-wavy BS to me. I assume that, if more text is needed to train LLMs in the future, the vast majority of it is going to be pennies-on-the-page, mechanical turk-type workers. And the LLMs are going to reflect that kind of input.
1. https://allenpike.com/2024/llms-trained-on-internet https://allenpike.com/2024/llms-trained-on-internet
2. https://news.ycombinator.com/item?id=40549021 https://news.ycombinator.com/item?id=40549021
- jmbwell 2y agoIn which case, why not just have humans write the text we need instead of firing all our writers in the first place?
- simmerup 2y agoEase of use? Like why pay more for AWS when you can host things locallly. People just can't be bothered with the admin
- iaaan 2y agoIf corporations can simply augment some generated text with the cheapest human labor available, without audiences noticing or caring so much that it affects their bottom line, they will do so. Corporations exist to optimize for cost/profit, nothing more.
- dleeftink 2y ago> Corporations exist to optimize for cost/profit, nothing more But LLMs are not only used in corporate settings. Cutting the bottom line in medical professions, case law, academic journals, etc. may seem enticing, but their audiences will definitely bear the brunt of the loss of specialist diagnoses, case work and peer/lit reviews.
- iaaan 2y agoAgreed, their audiences will definitely bear the brunt of those losses, but they will still absolutely try, and your average person has very little (if any) control over this.
- Bluestein 2y ago(Which is very very scary - at the very least: Law, medicine ... ... how many (and to what extent) are today's students/trainees and future practicing pros already passing classes that they would have otherwise failed, undetected -for example- and undermining future professional services which we will all avail ourselves of?
- xattt 2y agoR/AskReddit has prompt posts that seem to tap into people’s opinions about current events and how people feel about various things. My best guess is that it’s LLM trainers tapping into the oblivious masses.
- mcmcmc 2y agoLatency
- HPsquared 2y agoData quality matters for training, at least as much as quantity.
- bunderbunder 2y agoA possible counterpoint here is The Unreasonable Effectiveness of Data by Halevy, Norvig and Pereira. https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/35179.pdf https://static.googleusercontent.com/media/research.google.c... Say what you will about the merits of this paper, it's hard to deny that it has been influential within the deep learning community.
- rolisz 2y agoThat's a bit outdated. At the scale LLMs are trained, quality starts to matter again.
- charlieyu1 2y agoThe pay that workers earn suggest otherwise
- vidarh 2y agoThere's AI annotation work paying into the $100+/hour range. Not nearly as much as at the lower rates, but the amount in the $20-$40/hour range might surprise you. (been offered contracts, have done some on the top end of the market, have access to data on some of the volume on the lower end)
- charlieyu1 2y agoCan’t found anything near that, $40 for expert roles is the most common I find. I worked with one of these vendors, and the working condition is so horrible that people who are actual experts would not tolerate
- mistrial9 2y agothe Internet has generated vast digital language data in forums and publications, ads and opinion writing.. this is discussed in detail already.. there is concern about training on low-quality thinking and writing, and very much concern about using LLM output to train new LLM.. there is so much content needed for training, no group of humans can review it all. "bias" is the first word they use but really there are endless flaws that are possible.
- some_random 2y agoUnless there's huge leverage specific text/topics, I can't imagine MTurk anything will make a dent in the amount of text available for ML training. Your best bet imo is to somehow detect and filter out enough LLM generated text from reddit or wherever to not poison yourself.
- a2128 2y agoThe issue is that as AI gets better and more accessible, more and more people will use AI to help them with MTurk-style work that requires writing text[0]. The only sure way to get human-generated text will be to either have a person go to an office (expensive), or more likely just force all workers to install 5 cameras surveilling their every move with some algorithm that may fire a few people for taking a bathroom break. Or, more easily, just scrape more data [0] https://techcrunch.com/2023/06/14/mechanical-turk-workers-are-using-ai-to-automate-being-human/ https://techcrunch.com/2023/06/14/mechanical-turk-workers-ar...
- what 2y ago>just scrape more data… …that was generated by LLMs. So no you’re back to problem of getting data generated by humans.
- insane_dreamer 2y agoDon't we also eventually run into a situation where so much of the online content is AI generated that LLMs are being trained mostly on their own data. In that situation, it seems we could get stuck in a self-reinforcing loop rather than branching out with creativity and ingenuity, because even if such content still exists out there it risks being drowned out.