4 ms·
The serious science of trolling LLMs
- ysofunny 2y ago> unlike humans, language models can’t be humiliated and do not learn from mistakes. wut. I was pretty sure they do learn from mistakes. but they call it 'training'
- jsheard 2y agoThe authors point is that LLMs dont learn "on the job" so to speak, as a user if you tell something like ChatGPT that it has made a mistake then it will put that in its short term memory (the session context) but it won't be committed to long term memory (the actual model weights) that persists across sessions. Training is usually a separate process that end-users have little to no direct influence over.
- buildbot 2y agoThat's how it's currently deployed, but there is no requirement or restriction that you can't learn online. It's just kinda a bad idea, see Microsoft Tay...
- ysofunny 2y agonot the ones publicly available I bet. but it's one of those things that's only hard to do because of the capitalist race ongoing in that space but very easy to consider... what if Gemini learned on the job??? like self-training? (alpha go zero?) these is a low-hanging fruit sort of idea is all I'm saying
- voiceblue 2y agoIn the context of prediction, there is no such thing as a “mistake”, any more than a d20 die can make a mistake. You might get a 1 when you wanted a 20, but you can hardly blame the machine, only the manufacturer who forgot to weight it with the appropriate (or inappropriate) bias!
- talldayo 2y ago> It’s when it fails at a simple task that we know what the limitations are — and trolls are the torch-bearers of this new englightenment. I really don't agree. Watching AI fail spectacularly is a great bread and circus, but identifying something's deficiencies is not the same as fixing it. The thing these trolls get so damningly right is their denouncement of AI; we can't rely on shit text-generators that fly by the seat of their pants. The insinuated "but it's useful and does real work" shtick is over; I don't even see people talking about RAG value-add anymore. The bubble popped, and now expectations are returning to normal. It's less of an enlightenment period and more of the "AI winter" that we've always known was going to occur once we exhausted the current scaling limits. The trolls are scratching beneath the surface and proving the prophecy right.
- mnk47 2y ago> The insinuated "but it's useful and does real work" shtick is over > The bubble popped NVIDIA has a 3.12 trillion market cap. AI investments are reportedly surging: > Despite a decline in overall AI private investment last year, funding for generative AI surged, nearly octupling from 2022 to reach $25.2 billion. Major players in the generative AI space, including OpenAI, Anthropic, Hugging Face, and Inflection, reported substantial fundraising rounds. https://aiindex.stanford.edu/report/ https://aiindex.stanford.edu/report/
- geon 2y agoGenerative AI is basically like playing the game where each participant says one word of a sentence, one after the other. https://youtu.be/kNJNV0GXO-s?si=3TKUpFoFHV7FNSHR https://youtu.be/kNJNV0GXO-s?si=3TKUpFoFHV7FNSHR The AI has no idea where it is going before it starts outputting. If it ends up with a sentence that can’t be completed sensibly, it will output nonsense. I think a better model needs to be able to plan out the idea to convey in advance and possibly backtrack.
- dTal 2y agoI am curious, when you started writing your comment, did you know the last word of it would be "backtrack"?
- geon 2y agoYes.
- geon 2y agoStable diffusion doesn’t start with the top left pixel and generate them one by one. Why do llms?
- XenophileJKO 2y agoThis isn't entirely correct. The models clearly demonstrate anticipatory behavior.
- deleted 2y ago[deleted]
- cptcobalt 2y agoI wanted more examples of trolls and how the behavior degrades. The one scripted joke example doesn't land with a punch. > In short, I believe that the revealed objectives of LLM vendors differ starkly from what they say. They pour considerable resources into responding to every trollish LLM transcript that goes viral on the internet — and it’s useful to ponder why. I'd be pretty interested in reading at how people do poke at LLMs and what this actually reveals about the vendors priorities.
- XenophileJKO 2y agoI also thought the humor angle was odd as the models can create some very good humor when the context is set correctly.
- emilamlom 2y agoI am curious what the author would think about LLM based AI like Neuro-sama (AI vtuber that streams on twitch). Different use-case than the llm's the author was "trolling", but much better at humor and more resilient to trolls than usual.
- thih9 2y agoSummarizing, we’re dealing with closed source software coupled with an AI model trained on unspecified data. Vendors glorify the latter and never say how much work is handled by the former. Trust seems a big factor here.