7 ms·
The author of this isn't wrong, everything he says is correct but I think the order in which he says things is at the very least misleading. Yes, there is techn
by SilverBirch 2y ago
The author of this isn't wrong, everything he says is correct but I think the order in which he says things is at the very least misleading. Yes, there is technically training which is separate, but as he points out, the companies that are running these things are recording and storing everything you write and it more than likely will influence the next model they design. So sure, today the model you interact with won't update based on your input, but the next iteration might, and at some point it wouldn't be particularly surprising to see models which do live train on the data as you interact.
It's like the bell curve meme - the person who knows nothing "This system is going to learn from me", the person in the middle "The system only learns from the training data" and the genius "This system is going to learn from me".
No one is re-assured that the AI only learns from you with a 6 month lag.
- simonw 2y agoThe key message I'm trying to convey in this piece is that models don't instantly remember what you tell them. I see that as an independent issue from the "will it train on my data" thing. Some people might be completely happy to have a model train on their inputs, but if they assume that's what happens every time they type something in the box they can end up wasting a lot of time thinking they are "training" the model when their input is being instantly forgotten every time they start a new chat.
- mistercow 2y agoYeah, this is something I’ve found that it’s important to explain early to users when building an LLM-based product. They need to know that testing the solution out is useful if they share the results back with you. Otherwise, people tend to think they’re helping just by exercising the product so that it can learn from them.
- pornel 2y agoYou address the "personalization" misconception, but to people who don't have this misconception, but are concerned about data retention in a more general sense, this article is unclear and seems self-contradictory.
- simonw 2y agoWhat's unclear? I have a whole section about "Reasons to worry anyway".
- pornel 2y ago"ChatGPT and other LLMs don’t remember everything you say" in the title is contradicted by the "Reasons to worry anyway", because OpenAI does remember (store) everything I say in non-opted-out chat interface, and there's no guarantee that a future ChatGPT based on the next model won't "remember" it in some way. The article reads as "no, but actually yes".
- simonw 2y agoMaybe I should have put the word "instantly" in there: Training is not the same as chatting: ChatGPT and other LLMs don’t instantly remember everything you say
- swiftcoder 2y agoThey may not internalise it instantly. They certainly do "remember" (in the colloquial sense) by writing it to a hard drive somewhere. This article feels like a game of semantics.
- goatlover 2y agoIn the sense of a chat bot that people are interacting with, it doesn't remember. That's an important distinction, regardless of what OpenAI does to save your interactions somewhere for whatever purposes they may have in mind.
- deleted 2y ago[deleted]
- amenhotep 2y agoIt's funny, I completely know that ChatGPT won't remember a thing I tell it but when I'm using it to try and solve a problem and it can't quite do it and I end up figuring out the answer myself, I very frequently feel compelled to helpfully inform it what the correct answer was. And it always responds something along the lines of "oh, yes, of course! I will remember that next time." No you won't! But that doesn't stop me. Not as bad as spending weeks pasting stuff in, but enough that I can sympathise with the attempt. Brains are weird.
- simonw 2y agoYeah I hate that bug! The thing where models suggest that they will take your feedback into account, when you know that they can't do that.
- sebzim4500 2y agoIt's possible that OpenAI scrapes people's chat history for cases where that happens in order to improve their fine tuning data, in which case it isn't a total lie.
- outofpaper 2y agoThey say that will do this so long as users don't opt out.
- radicality 2y agoIsn’t that kind of how the “ChatGPT memory” feature works? I’ve recently seen it tell me that it’s updating memory, and whatever I said does appear under the “Memory” in the settings. I’m not familiar though with how the Memory works, ie whether it’s using up context length in every chat or doing something else.
- simonw 2y agoYeah, memory works by injecting everything it has "remembered" as part of the system prompt at the beginning of each new chat session: https://simonwillison.net/2024/Feb/14/memory-and-new-controls-for-chatgpt/ https://simonwillison.net/2024/Feb/14/memory-and-new-control...
- jedberg 2y ago> The key message I'm trying to convey in this piece is that models don't instantly remember what you tell them. Are you sure? We have no idea how OpenAI runs their models. The underlying transformers don't instantly remember, but we have no idea what other kinds of models they have in their pipeline. There could very well be a customization step that accounts for everything else you've just said.
- simonw 2y agoThat's a good point: I didn't consider that they might have additional non-transformer models running in the ChatGPT layer that aren't exposed via their API models. I think that's quite unlikely, given the recent launch of their "memory" feature which wouldn't be necessary if they had a more sophisticated mechanism for achieving the same thing. As always, the total lack of transparency really hurts them here.
- Matticus_Rex 2y agoAn element in the layer collecting data for training future models would almost certainly not be able to fulfill the same function as the memory feature.
- glandium 2y agoThey could very well have an "agent" that scrapes all conversations and tags elements that it finds might be interesting to feed the training of a future model. At least, that's what I would do if I were in their shoes. They presumably have a moderator agent that runs separately of ChatGPT and that causes the "This content may violate our content policy" orange box via `type: moderation` messages. They could just as well have a number of other agents.
- ClassyJacket 2y agoPeople are using and picking apart and blogging about their services so much that I feel like if they were doing something like this we would know about it in some sense, even if we didn't know how they were doing it
- tsunamifury 2y agoActually latest interfaces use cognitive compression to keep memory inside the context window. It’s a widely used trick and pretty easy to implement.
- simonw 2y agoDo you know of any chat tools that are publicly documented as using this technique?
- tsunamifury 2y agoNo but we all talk about it behind the scenes and everyone seems to use some form of it. Just have the model reflect and summarize so far and remember key concepts based on the trajectory and goals of the conversation. There are a couple different techniques based on how much compression you want: key pairing for high compression and full statement summaries for low compression. There is also a survey model where you have the llm fill in and update a questioneire every new input with things like “what is the goal so far” and “what are the key topics” It’s essentially like a therapists notepad that the model can write to behind the scenes of the session. This all conveniently lets you do topical and intent analytics more easily on these notepads rather than the entire conversation.
- simonw 2y agoRight, I know the theory of how this can work - I just don't know who is actually running that trick in production.
- joquarky 2y agoI'm curious what summarizing prompts or specific verbs (e.g. concise, succinct, brief, etc.) achieve the best "capture" of the context.
- tsunamifury 2y ago“One sentence” does the trick
- Suppafly 2y agoHonestly the whole thing with these chat AIs not continually learning is the most disappointing thing about them and really removes a lot of utility they provide. I don't really understand why they are essentially fixed in time to whenever the model was originally developed, why doesn't the model get continuously improved, not just from users but from external data sources?
- scarface_74 2y agoDo we need for the model to be be continuously updated from data sources or is it good enough that they can now figure out either by themselves or with some prompting when they need to search the web and find current information? https://chatgpt.com/share/0a5f207c-2cca-4fc3-be33-7db947c64b70 https://chatgpt.com/share/0a5f207c-2cca-4fc3-be33-7db947c64b... Compared to 3.5 https://chatgpt.com/share/8ff2e419-03df-4be2-9e83-e9d915921b0d https://chatgpt.com/share/8ff2e419-03df-4be2-9e83-e9d915921b...
- Suppafly 2y agoThe links aren't loading for me, but there is a difference between the output when the AI is trained vs having it google something for you, no? Having the ability to google something for you vs just making up an answer or being unable to answer is definitely a step in the right direction, but isn't the same as the AI incorporating the new data into it's model on an ongoing basis as a way of continuous improvement.
- scarface_74 2y agoI don’t know why the links aren’t working correctly. The idea is that if an LLM can now search the web and analyze data, it will be more up to date than training it. Another unrelated improvement with the newer versions of ChatGPT is that while LLMs are notoriously bad at math, they are pretty good at writing Python. ChatGPT can write Python code to solve a problem, run it and give you an answer based on the code. You can also verify the code yourself. I used ChatGPT to model financial scenarios and the answers were correct in 4x. I verified the generated code it was using. But here is what it answered when I asked ChatGPT “What’s going on in Atlanta this weekend”? > This weekend in Atlanta, there are several exciting events happening: 1. Lakewood Antiques Market (May 31 - June 2): A large market in Cumming featuring antiques and collectibles . 2. Bark Meow-Ket (June 1): A craft and pet vendor market at The Lawn at Uptown Atlanta, with food and pet adoptions . (and a few other events) But a better example and where I forced it to look on the web is “Compare Amex’s current transfer partners to Chase’s partners” For context, when you earn credit card points with Amex and Chase, you can transfer them to airline and hotel loyalty programs. It searches the web (and provides the links it used to find the information) and then answers with: ——- ### American Express (Amex) Transfer Partners Amex has a robust list of transfer partners, spanning both airlines and hotels. As of 2024, Amex Membership Rewards points can be transferred to the following partners: #### Airlines: 1. Aer Lingus 2. Aeromexico … (A list of the airlines) #### Hotels: 1. Choice Privileges 2. Hilton Honors 3. Marriott Bonvoy ### Chase Transfer Partners Chase Ultimate Rewards points can be transferred to a variety of airline and hotel partners. Here are the current partners for 2024: #### Airlines: 1. Aer Lingus (AerClub) 2. Air Canada (Aeroplan) (a list of airlines) #### Hotels: 1. Hyatt (World of Hyatt) 2. IHG (IHG One Rewards) 3. Marriott Bonvoy ### Comparison Both Amex and Chase offer a strong selection of transfer partners, but there are some differences: - *Unique to Amex*: ANA, Avianca, Cathay Pacific, Delta, Etihad, Hawaiian, and Qatar. - *Unique to Chase*: Southwest Airlines, United Airlines, and Hyatt. - *Common Partners*: Aer Lingus, Air Canada, British Airways, Emirates, Flying Blue (Air France/KLM), Iberia Plus, JetBlue, Singapore Airlines, Virgin Atlantic, and Marriott Bonvoy. Amex tends to have a larger selection of airline partners, especially those in the SkyTeam and oneworld alliances. Chase, on the other hand, includes Southwest and United, which are popular with domestic travelers in the U.S., and Hyatt, which is highly regarded among hotel loyalty programs (a list of web citations)
- math_dandy 2y agoPeople need to distinguish between ChapGPT-the-model and ChatGPT-the-service. The latter has memory; the former does not (as far as we know).
- zamfi 2y agoI love this whole series on misconceptions! Our expectations here are very much set by human-human interactions: we expect memory, introspection, that saying approximately-the-same-thing will give us approximately-the-same-result, that instructions are better than examples, that politeness helps, and many more [1] -- and some of these expectations are so deeply rooted that even when we know, intellectually, that our expectations are off, it can be hard to modify our behavior. That said, it will be super interesting to see how people's expectations shift -- and how we bring new expectations from human-AI interactions back to human-human interactions. [1]: https://dl.acm.org/doi/pdf/10.1145/3544548.3581388 https://dl.acm.org/doi/pdf/10.1145/3544548.3581388 (open access link)
- JohnMakin 2y ago> Our expectations here are very much set by human-human interactions True, but also a healthy dose of marketing these tools as hyper-intelligent, anthropomorphizing them constantly, and hysterical claims of them being "sentient" or at least possessing a form of human intelligence by random "experts" including some commenters on this site. That's basically all you hear about when you learn about these language models, with a big emphasis on "safety" because they are ohhhh so intelligent just like us (that's sarcasm).
- zamfi 2y agoI hear you, and that certainly plays a role -- but we actually did the work in that paper months before ChatGPT was released (June-July 2022), and most of the folks who participated in our study had not heard much about LLMs at the time. (Obviously if you ran the same study today you'd get a lot more of what you describe!)
- wkat4242 2y agoYeah I've seen people go out of their way to correct a model giving wrong information. Even though it apologizes and corrects itself, they don't understand it will give the same wrong answer again in the next session :)
- wruza 2y agoTraining on chats with an LLM is considered useless in my circles (enthusiast level). The argument is that it already knows its answers, also user input is just too bad to train on, because it’s mostly clueless questions, corrections and rants, due to the nature of chats with LLMs. It just isn’t a discussion worth including into a dataset. Some people experimented with loras trained on such chats and reported degradation.
- scarface_74 2y agoThe author also points out why would any sensible AI company want what is more likely low quality data with personal information in its training set?
- lukan 2y ago"and at some point it wouldn't be particularly surprising to see models which do live train on the data as you interact" That would probably be a big step towards true AI. Any known promising approaches towards that?
- zitterbewegung 2y agoIf LLMs did remember everything said it would be a lossless compression algorithm too….
- Hugsun 2y agoIt would certainly be lossy. We don't know how to make them reliably know any particular facts. Neither training or in context learning does this without fail.