7 ms·
Training is not the same as chatting: LLMs don’t remember everything you say
- gmerc 2y agoApart from the forced memories, which are super annoying in ChatGPT.
- simonw 2y agoYeah, I turned that feature off - it kept on making decidedly non-useful decisions about what to remember.
- phillipcarter 2y agoYeah, the difference between "train on this data" and "these interactions might be collected and aggregated across sessions to be included in a future training dataset" is a difficult one to message to most people in software, let alone consumers without that background. By the time you're done explaining how most data is largely garbage, labeling and curating data is difficult and time-consuming, and updates may make things worse such that a rollback is forced, etc. you're up to several paragraphs of explanation and most people won't read that.
- AlexandrB 2y agoIf you're worried that what you're putting into the text box might be used to train the AI, I don't see how any of this matters. There's no transparency (or control) on whether your specific input is being used for training or not, so the only safe assumption is that it will be used.
- simonw 2y agoThis is what I mean when I talk about the AI trust crisis. Plenty of AI vendors will swear that they don't train on input to their models. People straight up don't believe them.
- AlexandrB 2y agoThe economic incentives are too stacked. You see companies like reddit signing contracts to sell their user generated data to AI companies[1]. If OpenAI is willing to pay (probably a large sum) for reddit's data, why would they refrain from exploiting data they're already getting for "free"? This also feels like a rerun of the advertising economy's development and how tracking and data collection became more invasive and more common over time. [1] https://www.reuters.com/markets/deals/openai-strikes-deal-bring-reddit-content-chatgpt-2024-05-16/ https://www.reuters.com/markets/deals/openai-strikes-deal-br...
- phillipcarter 2y agoThere's several reasons why this may be: 1. Most of the data might be junk, and so it's just not worth it 2. They want paying customers, paying customers don't want this, so they don't offer it for paying customers 3. It's really, really hard to annotate data effectively at that scale. The reason why Reddit data is being bought is because it's effectively pre-annotated. Most users don't follow up with chat responses to give feedback. But people absolutely provide answers and upvotes on forums.
- janalsncm 2y agoIt’s like the popcorn button on a microwave. For it to work properly your microwave needs a humidity sensor. Cheap microwaves don’t, but they still include a popcorn button. As a result, no one trusts the popcorn button, even if it works.
- jerkstate 2y agoThere’s an opt-out: https://www.groovypost.com/howto/opt-out-your-data-on-chatgpt/ https://www.groovypost.com/howto/opt-out-your-data-on-chatgp... Of course, there’s no real way to know if it’s respected or not.
- fsmv 2y agoHow did they take this long to figure it out? I would have expected this article when ChatGPT first came out.
- croes 2y agoMaybe they thought the users would understand how LLMs get trained. Seems that's not the case and some think the model could br trained instantly by their input instead of much later when a good amount of new training data is collected.
- pbhjpbhj 2y agoIn part I suspect that's down to prompting, which is somewhat obscured (on purpose) from most users. "Prompts" use symbols within the conversation that's gone before, along with hidden prompting by the organisation that controls access to an LLM, as a part of the prompt. So when you ask, 'do you remember what I said about donuts' the LLM can answer -- it doesn't remember, but that's [an obscured] part of the current prompt issued to the LLM. It's not too surprising users are confused when purposeful deception is part of standard practice. I can ask ChatGPT what my favourite topics are and it gives an answer ...
- kordlessagain 2y agoSimon has been working on this technology for a while, and is fairly prolific about writing about it. It's more likely that it didn't "take long" for Simon to figure it out, but more that it was come to be realized that others had misconceptions about how it worked, thus the usefulness of posting about it presented itself. Things usually aren't black and white, unless of course they are bits...
- simonw 2y agoRight; I've understood this since ChatGPT first launched, but I've only recently understood how common and damaging the misconception about it is.
- Matticus_Rex 2y ago
- SilverBirch 2y agoThe author of this isn't wrong, everything he says is correct but I think the order in which he says things is at the very least misleading. Yes, there is technically training which is separate, but as he points out, the companies that are running these things are recording and storing everything you write and it more than likely will influence the next model they design. So sure, today the model you interact with won't update based on your input, but the next iteration might, and at some point it wouldn't be particularly surprising to see models which do live train on the data as you interact. It's like the bell curve meme - the person who knows nothing "This system is going to learn from me", the person in the middle "The system only learns from the training data" and the genius "This system is going to learn from me". No one is re-assured that the AI only learns from you with a 6 month lag.
- simonw 2y agoThe key message I'm trying to convey in this piece is that models don't instantly remember what you tell them. I see that as an independent issue from the "will it train on my data" thing. Some people might be completely happy to have a model train on their inputs, but if they assume that's what happens every time they type something in the box they can end up wasting a lot of time thinking they are "training" the model when their input is being instantly forgotten every time they start a new chat.
- mistercow 2y agoYeah, this is something I’ve found that it’s important to explain early to users when building an LLM-based product. They need to know that testing the solution out is useful if they share the results back with you. Otherwise, people tend to think they’re helping just by exercising the product so that it can learn from them.
- pornel 2y agoYou address the "personalization" misconception, but to people who don't have this misconception, but are concerned about data retention in a more general sense, this article is unclear and seems self-contradictory.
- huac 2y agoit's probably correct to think of functionally all ML models as being stateless. even something like twitter/fb feed - the models themselves remain the same (usually updated 1-2x per month IIRC) - only the data and the systems change. an illustrative example: say you open twitter, load some posts, then refresh. the model's view of you is basically the same, even the data is basically the same. you get different posts, however, because there is a system (read: bloom filter) on top of the model that chooses which posts go into ranking and that system removes posts that you've already seen. similarly, if you view some posts or like them, that updates a signal (e.g. time on user X profile) but not the actual model. what's weird about LLM's is that they're modeling the entire universe of written language, which does not actually change that frequently! now, it is completely reasonable to instead consider the problem to be modeling 'a given users' preference for written language' - which is personalized and can change. this is a different feedback to really gather and model towards. recall the ranking signals - most people don't 'like' posts even if they do like them, hence reliance on implicit signals like 'time spent.' one approach I've considered is using user feedback to steer different activation vectors towards user-preferred responses. that is much closer to the traditional ML paradigm - user feedback updates a signal, which is used at inference time to alter the output of a frozen model. this certainly feels doable (and honestly kinda fun) but challenging without tons of users and scale :)
- charles_f 2y ago> LLM providers don’t want random low-quality text [...] making it into their training data Don't they though? Most the training data is somewhat contentious, between the accusation of purely stealing it and the issues around copyright. While it's not text organized directly as knowledge the way Wikipedia is, it's still got some value, and at least that data is much more arguably "theirs" (even if it's not, it's their users'). Short of a pure "we won't train our stuff on your data" in their conditions, I would assume they do.
- CuriouslyC 2y agoThey really don't. Pretraining on low quality QA is a bad idea so at the very least they're filtering down the data set. Low quality QA is still useful for RLHF/task training though if the question is well answered.
- sanxiyn 2y agoPeople often correct bad generation and I think such feedback is valuable. eg "Training Language Models with Language Feedback at Scale" https://arxiv.org/abs/2303.16755 https://arxiv.org/abs/2303.16755
- simonw 2y agoI don't think they do, for a few reasons: 1. Evidence keeps adding up that the quality of your training data really matters. That's one of the reason's OpenAI are signing big dollar deals with media companies The Atlantic, Vox etc have really high quality tokens! 2. Can you imagine the firestorm that will erupt the first time a company credibly claims that their private IP leaked to another user via a ChatGPT session that was later used as part of another response? But as I said in the article... I can't say this with confidence because OpenAI are so opaque about what they actually use that data for! Which is really annoying.
- FeepingCreature 2y agoReally, what you'd want is to train a LoRA or something on user input. Then you can leave it specialized to that user.
- ankit219 2y agoI understand where you are coming from, but I kind of see where the misconception comes from too. Over the last year and a half, "training" has come to mean everything from pretraining an LLM from scratch, instruction tuning, finetuning, RLHF, and even creating a RAG pipeline that can augment a response. Many startups and Youtube videos use the word loosely to mean anything which augments the response of an LLM. To be fair, in context of AI, training and learning are elegant words which sort of convey what the product would do. In case of RAG pipeline, both for memory features or for augmenting answers, I think the concern is genuine. There would be someone looking at the data (in bulk) and chats to improve the intermediary product and (as a next step) use that to tweak the product or the underlying foundational model in some way. Given the way every foundational model company launched their own Chatbot service (mistral, Anthropic, Google) after ChatGPT, there seems to be some consensus that the real world conversation data is what can improve the models further. For the users who used these services (and not APIs), there might be a concerning scenario. These companies would look at queries a model failed to answer, and there is a subset which would be corporate employees asking questions on proprietary data (with context in prompt) for which ChatGPT is giving a bad answer because it's not part of the training data.
- ChicagoDave 2y agoI’d point out that this is the “missing link” to a next-gen LLM. When the scientists figure out how to enable a remembered collaboration that continually progresses results, then we’ll have something truly remarkable.
- danielmarkbruce 2y agoYou can do this with a sophisticated RAG system.
- ChicagoDave 2y agoTo some degree, sure, but I'm suggesting a productized client. I envision an IDE with separate windows for collaborative discussions. One window is the dialog, and another is any resulting code as generated files. The files get versioned with remarks on how and why they've changed in relation to the conversation. The dev can bring any of those files into their project and the IDE will keep a link to which version of the LLM generated file is in place or if the dev has made changes. If the dev makes changes, the file is automatically pushed to the LLM files window and the LLM sees those changes and adds them to the "conversation". The collaboration continually has "why" answers for every change. And all of this without anyone digging into models, python, or RAG integrations.
- danielmarkbruce 2y agoYou could build this now, using gpt-4 (or any of the top few models) and a whole bunch of surrounding systems. It would be difficult to pull off, but there is not technology change required. If you build it you'll need to be digging into models, python and various pieces of "RAG" stuff, but your users wouldn't need to know anything about it.
- gwern 2y agoYou can do this with dynamic evaluation (https://gwern.net/doc/ai/nn/dynamic-evaluation/index https://gwern.net/doc/ai/nn/dynamic-evaluation/index): literally just train on the data as you are fed new data to predict, in the most obvious possible way. It works well, and most SOTAs set in text prediction 2010-2018 would use dynamic evaluation for the boost. (And it has equivalents for other tasks, like test-time adaptation for CNN classifiers.) In fact, dynamic evaluation is so old and obvious that the first paper to demonstrate RNNs really worked better than n-grams as LLMs did dynamic evaluation: https://gwern.net/doc/ai/nn/rnn/2010-mikolov.pdf#page=2 https://gwern.net/doc/ai/nn/rnn/2010-mikolov.pdf#page=2 Dynamic evaluation offers large performance boosts in Transformers just as well as RNNs. The fact that it's not standard these days has more to do with cloud providers strongly preferring models to not change weights to simplify deployment and enable greater batching, and preferring to crudely approximate it with ever greater context windows. (Also probably the fact that no one but oldtimers knows it is a thing.) I think it might come back for local LLMs, where there is only one set of model weights running persistently, users want personalization and the best possible performance, and where ultra-large context windows are unacceptably memory-hungry & high-latency.
- euniceee3 2y agoAnyone know if there is a LM Studio style tool for training models?
- danielmarkbruce 2y agoThis is just one of many misconceptions about LLMs. If you are at all involved in trying to build/deploy a product based around LLMs, your head will explode as you talk to users and realize the vast chasm between how folks who understand them think about and use them, and folks who don't understand them. It's frustrating on both sides. It's quite unclear if we'll ever hit the point where (like, a car for example) someone who has no idea how it works gets as much out of it as someone who does.
- SunlitCat 2y agoAnd yet, you need a drivers license to operate a car. Computers, AI and other technology everyone is allowed to use, even without basic understanding of how stuff works.
- danielmarkbruce 2y agoGenerally, any situation that is as dangerous for other people as driving a car requires a license or is illegal. It has nothing to do with understanding the underlying technology. About the last thing we need is a law requiring a license to use ChatGPT.
- SunlitCat 2y agoOperating technology without understanding what you're doing, can be dangerous for other people as well and shouldn't be underestimated. Of course, the license requirement wasn't meant literally, but at least some basic understanding that not downloading everything from everywhere is a good thing to do, or that clicking on that link and entering your details is okay just because the mail said "WE ARE YOUR BANK AND YOU NEED TO GIVE ALL DETAILS!!!" and so on. If you point that out, you are often meet with "Ah. Technology is so difficult to understand. They should make it easier."
- siva7 2y agoI really think that an AI license would be the best solution, at least for school kids but ideally for everyone in a professional context. Operating an AI is very different than operating a computer as the former can cause real damage to the tech-"illiterate".
- kazinator 2y agoThe LLMs don't learn during the chat, but what you type into them is being collected, the same as, for instance, your Google searches.
- kordlessagain 2y agoI took Simon's post and pasted it into two different ChatGPT 4o windows. In one, I put "what's wrong with this post, especially with the way it's written?". In the other, I put "what's really great about this post and what does it do well?". Both sessions worked toward those implied goals. The one where I asked what was wrong indicated long-winded sentences, redundancy, lack of clear structure, and focus issues. Sorry, Simon! The one where I asked what was good about the post indicated clarity around misconceptions, good explanations of training, consistent theme, clear sections, logical progression, and focused arguments. I tend to agree with this one more than the other, but clearly the two are conflicted. So, what's my point? My point is that AI/LLMs interactions are driven by the human's intent during the interaction. It is literally our own history and training that brings the stateless function responses to where they arrive. Yes, facts and whatnot matter in the training data, and the history, if there is one, but the output is very much dependent on the user. If we think about this more, it's clear that training data does matter, but likely doesn't matter as much as we think it does. It's probably just as equally important to consider the historic data, and the data coming in from RAG/search processes, as making as big of an impact on the output.
- roywiggins 2y agoIt's at least sometimes a "Clever Hans" effect, or an "ELIZA" effect. These models have been RLHFed to just go along and agree with whatever the user asks for, even if the user doesn't realize what's going on. This is a bit of a cheap trick- of course users will find something more intelligent if it agrees with them. When models start arguing back you get stuff like early Bing Chat going totally deranged.
- Arthur_ODC 2y agoWow, I figured this would have been the most basic level of common knowledge for LLM end-users by this point. I guess there is still a surprising amount of people out there who haven't jumped on the bandwagon.
- swiftcoder 2y agoRegardless of whether the data is actually being used to train an LLM, as long as it's going over the network to an LLM provider, there's the possibility that it will be used. Or exfiltrated from the provider's servers and sold on the dark web. Or a million other things that can go wrong when you provide a 3rd party access to your personal data. If LLMs are actually here to stay, then we need to keep shrinking the inferencing costs to the point that everyone can run it on the client, and avoid all of these problems at once...
- deleted 2y ago[deleted]
- janalsncm 2y agoSometimes ChatGPT will ask me which response I like better. That is almost certainly used for something like DPO.
- caeril 2y agoWow. Water is wet. The sky is blue. FFNN models don't update their weights during inference. Who doesn't already know this?
- sinaid 2y agoOpenAI could of course send all your chat history to a DB and then use that on a future training round. Some of it is free RLHF data that needs a bit of cleaning up. Because the user side of the conversation has clues about whether the answer was correct. Also are we that far from fine tuning in real time? Maybe next year? I think while the current model might be what he says the model in 2025 could be vastly different.
- BensonGao 2y agoThe article is well written. It is precisely because there is no memory that frameworks like langchain exist. If a way to implement LLM with built-in memory capabilities can be found, I feel it will change the game rules of the entire large model market.
- mosseater 2y agoI dunno. Nowadays you can store the the conversation's history and use different RAG methods to pull relevant pieces into the LLM's context.
- ummonk 2y agoI’ve never seen anyone believe the straw man that the author seems to think people believe. People are rightly concerned about their data being trained on for the very real concerns which the author later acknowledges are legitimate risks.
- simonw 2y agoI was surprised when I found out people believe this too. Ask around some more, it's not the kind of thing that comes up in conversation.
- labrador 2y agoTrue, the model is already trained. But it can still be tuned. When I hit the dislike or like button in Claude and leave a comment what I found bad or good I assume my actions are not for nothing, just like a busy box to make me think I'm doing something. If that's true I'll stop doing it. I hope by doing it I'll help tune it and help the next model be better.
- gsuuon 2y agoA 'remember everything I say' model would actually be fantastic if ran locally.
- Syzygies 2y agoI chat a lot with ChatGPT-4o, usually endless everyday questions like how to replace the screened washer in a kitchen faucet supply line. It stunned me recently by remembering I had an argon tank, something I had only mentioned in a thread days earlier. I asked it to recall which conversations these were, for sharing on HN. That conversation was comically stupid (REALLY bad, like "Perplexity" bad) and went nowhere.
- vismit2000 2y agoGreat article! Agree with everything mentioned. Good to share with all tech people who don't work in the AI space and "hallucinate" about working of LLMs. Having said that, although LLMs don't directly learn from what we say to them, they can for sure create a representation or persona using Representation Engineering. Quoting from an excellent article by Theia Vogel [1]: "In October 2023, a group of authors from the Center for AI Safety, among others, published Representation Engineering: A Top-Down Approach to AI Transparency. That paper looks at a few methods of doing what they call 'Representation Engineering': calculating a 'control vector' that can be read from or added to model activations during inference to interpret or control the model's behavior, without prompt engineering or finetuning. (There was also some similar work published in May 2023 on steering GPT-2-XL.) Previously on HN 3 months ago: [2] [1] https://vgel.me/posts/representation-engineering/ https://vgel.me/posts/representation-engineering/ [2] https://news.ycombinator.com/item?id=39414532 https://news.ycombinator.com/item?id=39414532
- croes 2y agoSeems like OpenAI needs more data >All ChatGPT Free users can now use browse, vision, data analysis, file uploads, and GPTs https://news.ycombinator.com/item?id=40524031 https://news.ycombinator.com/item?id=40524031