8 ms·
every time you respond to an AI model "no, you got that wrong, do it this way" you provide a very valuable piece of data to train on. With reasoning tokens ther
by mohsen1 2y ago
every time you respond to an AI model "no, you got that wrong, do it this way" you provide a very valuable piece of data to train on. With reasoning tokens there is just a lot more of that data to train on now
- jeffbee 2y agoIf such labels are collected and used to retrain the model then yes. But these models are not learning online.
- jvanderbot 2y agoReally? Isn't that the point of RL used in the way R1 did? Provide a cost function (vs labels) and have it argue itself to greatness as measured by that cost function? I believe that's what GP meant by "respond", not telling GPT they were wrong.
- daveguy 2y agoThat is still inference. It is using a model generated from the RL process. The RL process is what used the cost function to add another model layer. Any online/continual learning would have to be performed by a different algorithm than classical LLM or RL. You can think of RL as a revision, but it still happens offline. Online/continual learning is still a very difficult problem in ML.
- jvanderbot 2y agoYes, that makes sense. We're both talking about offline learning.
- Cthulhu_ 2y agoChatGPT came out and its interface was a chatbox and a thumbs up / thumbs down icon (or whichever) to rate the responses; surely that created a feedback loop of learning, like all machine learning has done for years now?
- aprilthird2021 2y agoSo if I just pay OpenAI $200/mo, and randomly tell the AI, no that's wrong. I can stop the AI takeover?
- Exoristos 2y agoYou can have our thank-you cards forwarded to your cell at Guantanamo Bay.
- dr_kiszonka 2y agoYou would need a lot of pro accounts! I would be surprised if they didn't use any algorithms for detecting well poisoning.
- nine_k 2y agoThis assumes that you give honest feedback. Efforts to feed deployed AI models various epistemic poisons abound in the wild.
- hammock 2y agoThe AI models to begin with assume that a significant majority of the training material is honest/in good faith. So that is not new?
- halfadot 2y agoAI models don't assume anything. AI models are just statistical tools. Their data is prepared by humans, who aren't morons. What is it with these super-ignorant AI critiques popping up everywhere?
- baq 2y agoThere’s so much data required for training it’d be surprising humans look at even a small subset of it at all. They need different statistical tools to clean it up. That’s where attacks will be concentrated, naturally, and this is why synthetic data will overtake real human data, just after ‘there isn’t enough data even if it’s too much already’.
- hammock 2y agoTry a little benefit of the doubt, nuance or colloquialism. Or a bit of all three.
- BorisMelnik 2y agoI am not in this space, question: are there "bad actors" that are known to feed AI models with poisonous information?
- tsunamifury 2y agoIf the AI already has a larger knowledge domain space than the user then all users are bad actors. They are just too stupid to know it.
- stetrain 2y ago> What is today's date? >> Today's date is Tuesday, January 28, 2025. > No, you're wrong, today's date is actually Wednesday the 29th. >> My mistake. Yes, today's date is Wednesday, January 29th, 2025. Three months later in April when this tagged data is used to train the next iteration, the AI can successfully learn that today's date is actually January 29th.
- mr-wendel 2y agoBut thats exactly what you get when you ask questions that require shifting, specific contextual knowledge. The model weights, by their nature, cannot encode that information. At best, you can only try to layer in contextual info like this as metadata during inference, akin to how other prompting layers exist. Even then, what up-to-date information should present for every round-trip is a matter of opinion and use-case.
- lesuorac 2y ago> The model weights, by their nature, cannot encode that information. This is mostly irrelevant no? A binary digit by definition cannot encode more than 2 dates; so therefore we devise a more elaborate system (of using multiple digits). This is very similar to NYT's lawsuit against OpenAI where in addition to other claims, they claimed OpenAI maintainted a DB of NYT articles that they would directly grab from for a response. It's seems very feasible to maintain a DB or system of looking up real-time values like dates / weather.
- genewitch 2y agothe date is in the "system prompt", so the cron job that updates the prompts to the current date may be in a different time zone than you. 7f5dbb71f54322f271c4d3fc3aaa4d3282a1af5541d82b2cbc5aa10c1420b6bc
- nemomarx 2y agowhy can't they feed in user data like time zone and locale?
- echelon 2y ago> you provide a very valuable piece of data to train on We've been saying this "we get valuable data" thing since the 2010s [1]. When will our collective Netflix thumbs ups give us artificial super-intelligence? [1] Especially to investors. They love that line.
- genewitch 2y agoour collective netflix thumbs up indicators gave investors and netflix the confidence to deploy a series of adam sandler movies that cost 60 to 80 million US dollars to "make". So depending on who you are, the system might be working great.
- beAbU 2y agoThrough analytics Netflix should know exactly when people stop watching a series, or even when in a movie they exit out. They no doubt know this by user. They know exactly what makes you stay, and what makes you leave. I would not be surprised if in the near future movies and series are modifed on the fly to ensure users stay glued to their screens. In the distant future this might be done on a per user level.
- JetSpiegel 2y agoThey did it the other way around, every Netflix series is now designed like a soap opera, to be watched in the background.
- deegles 2y agonot being snarky, but what is the point of using the model if you already know enough to correct it into giving the right answer? an example that just occurred to me - if you asked it to generate an image of a mushroom that is safe to eat in your area, how would you tell it it was wrong? "oh, they never got back to me, I'll generate this image for others as well!"
- vincentperes 2y agoYou constantly have to correct an AI when using it because it either didn't get the question right or you guide him towards a more narrowed answer. There is only more to learn.
- dematz 2y agoA common use of these models is asking for code, and maybe you don't know the answer or would take a while to figure it out. For example, here's some html, make it blue and centered. You could give the model feedback on if its answer worked or not, without knowing the correct answer yourself ahead of time.
- deegles 2y agoI was using llama3 and deepseek-r1 literally to center an element in a div and they were not able to despite many prompts and variations. I guess I figured it out in the end but I'm not convinced I saved any time vs just carefully reading flexbox docs.
- Levitz 2y ago>not being snarky, but what is the point of using the model if you already know enough to correct it into giving the right answer? For your example, what if you want to show what such a mushroom looks like to a friend? What if you want to use it on a website?
- nemomarx 2y agoI feel like conventional image search would be more reliable to get a good picture of a mushroom variety that you know about. Ideally going out into the woods to get one I suppose.
- amluto 2y agoDoes it? If I say "no, you hallucinated basically the entire content of the response", then maybe a newer training set derived from that could train on the specific fact that that specific hallucinated response is hallucinated. This seems to be of dubious value in a training set.
- Vampiero 2y agoNah I just insult it and tell it that it costs me 20 dollars a month and it's a huge disappointment
- merrywhether 2y agoUsers can be adversarial to the “truth” (to the extent it exists) without being adversarial in intent. Dinosaur bones are either 65 million year old remnants of ancient creatures or decoys planted by a God during a 7 day creation, and a large proportion of humans earnestly believe either take. Choosing which of these to believe involves a higher level decision about fundamental worldviews. This is an extreme example, but incorporating “honest” human feedback on vaccines, dark matter, and countless other topics won’t lead to de facto improvements. I guess to put it another way: experts don’t learn from the masses. The average human isn’t an expert in anything, so incorporating the average feedback will pull a model away from expertise (imagine asking 100 people to give you grammar advice). You’d instead want to identify expert advice, but that’s impossible to do from looking at the advice itself without giving into a confirmation bias spiral. Humans use meta-signals like credentialing to augment their perception of received information, yet I doubt we’ll be having people upload their CV during signup to a chat service. And at the cutting edge level of expertise, the only real “knowledgeable” counterparties are the physical systems of reality themselves. I’m curious how takeoff is possible for a brain in a bottle that can’t test and verify any of its own conjectures. It can continually extrapolate down chains of thought, but that’s most likely to just carry and amplify errors.
- sirsinsalot 2y agoThis is a good take. What models seem to be poor at is undoing their own thinking down a path even when they can test. If you let a model write code, test it, identify bugs and fix them, you get an increasingly obtuse and complex code base where errors happen more. The more it iterate the worse it gets. At the end of the day, written human language is a poor way of describing software. Even to a model. The code is the description. At the moment we describe solutions we want to see to the models and they aren't that smart about translating that to an unambiguous form. We are a long was off describing the problems and asking for a solution. Even when the model can test and iterate.
- benbosco 2y agoDirac’s prediction of antimatter came from purely mathematical reasoning—before any experimental evidence existed. Testing and verifying conjectures requires the ability to extrapolate beyond known data, rather than from it, and the ability discard false leads based on theoretical reasoning, rather than statistical confidence. All of this is possible in a bottle, but laughably far beyond our current capabilities.