4 ms·
I’ve heard the argument made that LLMs are simply models overfit on the entirety of the internet. Double decent is a good counter point to that claim, it sugges
by valine 3y ago
I’ve heard the argument made that LLMs are simply models overfit on the entirety of the internet. Double decent is a good counter point to that claim, it suggests something more interesting than simple overfitting is happening with large parameter counts.
- hackernewds 3y agoHow is double descent a good counterpoint to the claim?
- qumpis 3y agoThat novel reasoning is due to the model transitioning to a generalization regime, not merely parroting training data?
- malux85 3y agoOverfitting is lack of genralisation, LLMs can produce novel output by using abstraction composition, which you cannot do without generalisation. Other signs of non-overfitting: Abstraction laddering, task decomposition, novel (i.e. unseen) joke explanation
- version_five 3y agoI'd be interested to know if anyone has studied how overfitting translates into the domain of llm output: it's easy to understand when you're fitting a line though data, or building a classifier, you overfit and your test set loss is higher than your training set loss, and this directly relates to worse performance of the model. For an llm picking probably next words, what's the analogy, and does overfitting make it "worse" even if a test set loss is higher?
- bheadmaster 3y agoI find it hard to believe that every single problem has a solution on the internet, which is what "overfitting" would imply. I was playing Fallout: New Vegas on Wine, and for some reason, the music on my pipboy wasn't playing. I searched the internet for the Wine errors from the terminal to no avail, and as a last resort asked ChatGPT. It gave me step-by-step instructions on how to fix it, and it worked. If that doesn't demonstrate that LLMs have some kind of internal model of the world and understanding of it, then I don't know what will.
- alphydan 3y agoAn alternative explanation is that your google-fu is not as good as openai's crawlers or the WebText corpus (which can go a lot deeper than any of your searches).
- Buttons840 3y agoMakes me wonder how Google would compare if a good portion of the entire world hadn't spent decades trying to game their system. OpenAI created a completely new system and reaps the benefits of training on data that hasn't been twisted-half-way-to-hell to exploit it.
- ChatGTP 3y agoI can’t find anything on Google anymore so I’m not sure this proves much except open ai has better search.
- fauxpause_ 3y agohttps://www.gog.com/forum/fallout_series/new_vegas_music_is_not_playing_on_the_radio https://www.gog.com/forum/fallout_series/new_vegas_music_is_... I haven’t played this game so I don’t know what I’m searching for but this was my first result. Seems on the money, no?
- bheadmaster 3y agoThat's the thing - it isn't. I've been through this forum, many reddit posts and other sites - none of the solutions worked. What worked was that ChatGPT figured out that I need to add the following line to ~/.wine/system.reg: [Software\\Wine\\GStreamer] "DllOverrides"="mscoree,mshtml=" And install 32-bit version of gstreamer good plugins: sudo apt-get install gstreamer1.0-plugins-good:i386 If you happen to find these exact instructions anywhere on the internet, please share, as that will be enough to convince me that LLMs aren't anything more than glorified search engines. Otherwise, I can't help but be skeptical. If nothing else, it's plausible that LLMs have some kind of internal representation of the world.