6 ms·

> But now with reasoning systems and verifiers, we can create brand new legitimate data to train on. This can either be done offline where the developer pays to

> But now with reasoning systems and verifiers, we can create brand new legitimate data to train on. This can either be done offline where the developer pays to create the data or at inference time where the end user pays! > This is a fascinating shift in economics and suggests there could be a runaway power concentrating moment for AI system developers who have the largest number of paying customers. Those customers are footing the bill to create new high quality data … which improves the model … which becomes better and more preferred by users … you get the idea. While I think this is an interesting hypothesis, I'm skeptical. You might be lowering the cost of your training corpus by a few million dollars, but I highly doubt you are getting novel, high quality data. We are currently in a world where SOTA base model seems to be capped at around GPT4o levels. I have no doubt that in 2-3 years our base models will compete with o1 or even o3... just it remains to be seen what innovations/optimizations get us there. The most promising idea is to use reasoning models to generate data, and then train our non-reasoning models with the reasoning-embedded data. But... it remains to be seen how much of the chain of thought reasoning you can really capture into model weights. I'm guessing some, but I wonder if there is a cap to multi-head attention architecture. If reasoning can be transferred from reasoning models to base models, OpenAI should have already trained a new model with o3 training data, right? Another thought is maybe we don't need to improve our base models much. It's sufficient to have them be generalists, and to improve reasoning models (lowering price, improving quality) going forward.

mohsen1 2y ago
every time you respond to an AI model "no, you got that wrong, do it this way" you provide a very valuable piece of data to train on. With reasoning tokens there is just a lot more of that data to train on now
- jeffbee 2y ago
  If such labels are collected and used to retrain the model then yes. But these models are not learning online.
  - jvanderbot 2y ago
    Really? Isn't that the point of RL used in the way R1 did? Provide a cost function (vs labels) and have it argue itself to greatness as measured by that cost function? I believe that's what GP meant by "respond", not telling GPT they were wrong.
    - daveguy 2y ago
      That is still inference. It is using a model generated from the RL process. The RL process is what used the cost function to add another model layer. Any online/continual learning would have to be performed by a different algorithm than classical LLM or RL. You can think of RL as a revision, but it still happens offline. Online/continual learning is still a very difficult problem in ML.
      - jvanderbot 2y ago
        Yes, that makes sense. We're both talking about offline learning.
  - Cthulhu_ 2y ago
    ChatGPT came out and its interface was a chatbox and a thumbs up / thumbs down icon (or whichever) to rate the responses; surely that created a feedback loop of learning, like all machine learning has done for years now?
- aprilthird2021 2y ago
  So if I just pay OpenAI $200/mo, and randomly tell the AI, no that's wrong. I can stop the AI takeover?
  - Exoristos 2y ago
    You can have our thank-you cards forwarded to your cell at Guantanamo Bay.
  - dr_kiszonka 2y ago
    You would need a lot of pro accounts! I would be surprised if they didn't use any algorithms for detecting well poisoning.
- nine_k 2y ago
  This assumes that you give honest feedback. Efforts to feed deployed AI models various epistemic poisons abound in the wild.
  - hammock 2y ago
    The AI models to begin with assume that a significant majority of the training material is honest/in good faith. So that is not new?
    - halfadot 2y ago
      AI models don't assume anything. AI models are just statistical tools. Their data is prepared by humans, who aren't morons. What is it with these super-ignorant AI critiques popping up everywhere?
      - baq 2y ago
        There’s so much data required for training it’d be surprising humans look at even a small subset of it at all. They need different statistical tools to clean it up. That’s where attacks will be concentrated, naturally, and this is why synthetic data will overtake real human data, just after ‘there isn’t enough data even if it’s too much already’.
      - hammock 2y ago
        Try a little benefit of the doubt, nuance or colloquialism. Or a bit of all three.
  - BorisMelnik 2y ago
    I am not in this space, question: are there "bad actors" that are known to feed AI models with poisonous information?
    - tsunamifury 2y ago
      If the AI already has a larger knowledge domain space than the user then all users are bad actors. They are just too stupid to know it.
- stetrain 2y ago
  > What is today's date? >> Today's date is Tuesday, January 28, 2025. > No, you're wrong, today's date is actually Wednesday the 29th. >> My mistake. Yes, today's date is Wednesday, January 29th, 2025. Three months later in April when this tagged data is used to train the next iteration, the AI can successfully learn that today's date is actually January 29th.
  - mr-wendel 2y ago
    But thats exactly what you get when you ask questions that require shifting, specific contextual knowledge. The model weights, by their nature, cannot encode that information. At best, you can only try to layer in contextual info like this as metadata during inference, akin to how other prompting layers exist. Even then, what up-to-date information should present for every round-trip is a matter of opinion and use-case.
    - lesuorac 2y ago
      > The model weights, by their nature, cannot encode that information. This is mostly irrelevant no? A binary digit by definition cannot encode more than 2 dates; so therefore we devise a more elaborate system (of using multiple digits). This is very similar to NYT's lawsuit against OpenAI where in addition to other claims, they claimed OpenAI maintainted a DB of NYT articles that they would directly grab from for a response. It's seems very feasible to maintain a DB or system of looking up real-time values like dates / weather.
  - genewitch 2y ago
    the date is in the "system prompt", so the cron job that updates the prompts to the current date may be in a different time zone than you. 7f5dbb71f54322f271c4d3fc3aaa4d3282a1af5541d82b2cbc5aa10c1420b6bc
    - nemomarx 2y ago
      why can't they feed in user data like time zone and locale?
- echelon 2y ago
  > you provide a very valuable piece of data to train on We've been saying this "we get valuable data" thing since the 2010s [1]. When will our collective Netflix thumbs ups give us artificial super-intelligence? [1] Especially to investors. They love that line.
  - genewitch 2y ago
    our collective netflix thumbs up indicators gave investors and netflix the confidence to deploy a series of adam sandler movies that cost 60 to 80 million US dollars to "make". So depending on who you are, the system might be working great.
    - beAbU 2y ago
      Through analytics Netflix should know exactly when people stop watching a series, or even when in a movie they exit out. They no doubt know this by user. They know exactly what makes you stay, and what makes you leave. I would not be surprised if in the near future movies and series are modifed on the fly to ensure users stay glued to their screens. In the distant future this might be done on a per user level.
      - JetSpiegel 2y ago
        They did it the other way around, every Netflix series is now designed like a soap opera, to be watched in the background.
- deegles 2y ago
  not being snarky, but what is the point of using the model if you already know enough to correct it into giving the right answer? an example that just occurred to me - if you asked it to generate an image of a mushroom that is safe to eat in your area, how would you tell it it was wrong? "oh, they never got back to me, I'll generate this image for others as well!"
  - vincentperes 2y ago
    You constantly have to correct an AI when using it because it either didn't get the question right or you guide him towards a more narrowed answer. There is only more to learn.
  - dematz 2y ago
    A common use of these models is asking for code, and maybe you don't know the answer or would take a while to figure it out. For example, here's some html, make it blue and centered. You could give the model feedback on if its answer worked or not, without knowing the correct answer yourself ahead of time.
    - deegles 2y ago
      I was using llama3 and deepseek-r1 literally to center an element in a div and they were not able to despite many prompts and variations. I guess I figured it out in the end but I'm not convinced I saved any time vs just carefully reading flexbox docs.
  - Levitz 2y ago
    >not being snarky, but what is the point of using the model if you already know enough to correct it into giving the right answer? For your example, what if you want to show what such a mushroom looks like to a friend? What if you want to use it on a website?
    - nemomarx 2y ago
      I feel like conventional image search would be more reliable to get a good picture of a mushroom variety that you know about. Ideally going out into the woods to get one I suppose.
- amluto 2y ago
  Does it? If I say "no, you hallucinated basically the entire content of the response", then maybe a newer training set derived from that could train on the specific fact that that specific hallucinated response is hallucinated. This seems to be of dubious value in a training set.
  - Vampiero 2y ago
    Nah I just insult it and tell it that it costs me 20 dollars a month and it's a huge disappointment
- merrywhether 2y ago
  Users can be adversarial to the “truth” (to the extent it exists) without being adversarial in intent. Dinosaur bones are either 65 million year old remnants of ancient creatures or decoys planted by a God during a 7 day creation, and a large proportion of humans earnestly believe either take. Choosing which of these to believe involves a higher level decision about fundamental worldviews. This is an extreme example, but incorporating “honest” human feedback on vaccines, dark matter, and countless other topics won’t lead to de facto improvements. I guess to put it another way: experts don’t learn from the masses. The average human isn’t an expert in anything, so incorporating the average feedback will pull a model away from expertise (imagine asking 100 people to give you grammar advice). You’d instead want to identify expert advice, but that’s impossible to do from looking at the advice itself without giving into a confirmation bias spiral. Humans use meta-signals like credentialing to augment their perception of received information, yet I doubt we’ll be having people upload their CV during signup to a chat service. And at the cutting edge level of expertise, the only real “knowledgeable” counterparties are the physical systems of reality themselves. I’m curious how takeoff is possible for a brain in a bottle that can’t test and verify any of its own conjectures. It can continually extrapolate down chains of thought, but that’s most likely to just carry and amplify errors.
  - sirsinsalot 2y ago
    This is a good take. What models seem to be poor at is undoing their own thinking down a path even when they can test. If you let a model write code, test it, identify bugs and fix them, you get an increasingly obtuse and complex code base where errors happen more. The more it iterate the worse it gets. At the end of the day, written human language is a poor way of describing software. Even to a model. The code is the description. At the moment we describe solutions we want to see to the models and they aren't that smart about translating that to an unambiguous form. We are a long was off describing the problems and asking for a solution. Even when the model can test and iterate.
  - benbosco 2y ago
    Dirac’s prediction of antimatter came from purely mathematical reasoning—before any experimental evidence existed. Testing and verifying conjectures requires the ability to extrapolate beyond known data, rather than from it, and the ability discard false leads based on theoretical reasoning, rather than statistical confidence. All of this is possible in a bottle, but laughably far beyond our current capabilities.
ClumsyPilot 2y ago
> The most promising idea is to use reasoning models to generate data, and then train our non-reasoning models with the reasoning-embedded data. Why is it promising, aren’t you potentially amplifying AI biases and errors?
- pas 2y ago
  it seems to work and seems very scalable, "reasoning" helps to counter biases (answers become longer, ie. the system uses more tokens which means more time to answer a question -- likely longer answers allow better differentiation of answers from each other in the "answer space") https://newsletter.languagemodels.co/i/155812052/large-scale-reasoning-oriented-reinforcement-learning-r-zero https://newsletter.languagemodels.co/i/155812052/large-scale... also from the posted article """ The R1-Zero training process is capable of creating its own internal domain specific language (“DSL”) in token space via RL optimization. This makes intuitive sense, as language itself is effectively a reasoning DSL. """
jcims 2y ago
>You might be lowering the cost of your training corpus by a few million dollars, but I highly doubt you are getting novel, high quality data. The large foundational models don't really need more empirical data about the world. ChatGPT already 'knows' way more than I do, probably by many orders of magnitude. Yet it's still spewing nonsense at me regularly because it doesn't know how to think like a human or interact with me in a human-like way. To that end, the ability for a company like OpenAI to collect novel data from interacting with real humans is a material advantage over their competition.
- visarga 2y ago
  > the ability for a company like OpenAI to collect novel data from interacting with real humans is a material advantage over their competition It's different kind of data from the R1 reasoning chains. When LLMs have human in the loop, the human provides help based off their personal experience and real world validation. Sometimes users take an idea from the LLM and try it in real life. Then come back later and discuss the outcomes. This is a real world testing loop. In order to judge if an AI response was useful, you can look at the following messages with a judge LLM. Using hindsight helps a lot here. Maybe it doesn't pan out and the user tries another approach, or maybe some innocuous idea was key to success later. It's hard to tell in the moment, but easy when you see what followed after that. This scales well - OpenAI has 300M users, I estimate up to 1 Trillion interactive tokens/day. The user base is very diverse, problems are diverse, and feedback comes from user experience and actual testing. They form an experience flywheel, the more problem solving they do, the smarter it gets, attracting more users.
fizx 2y ago
You're not getting new high-quality textual data for pre-training from your chat service. But you are potentially getting a lot of RL feedback on ambiguous problems.
deleted 2y ago
[deleted]
baq 2y ago
It doesn't need much. 1 good lucky answer in a 1000 or maybe 10k queries gives you the little exponential kick you need to improve. This is how the hockey stick take off looks like and we're already here - OpenAI has it, now deepseek has it, too. You can be sure others also have it; Anthropic at the very least, they just never announced it officially, but go read what their CEO has been speaking and writing about.
- spyckie2 2y ago
  I have looked a bit into Anthropic CEOs writings but if you can point in the right direction would be helpful!
sheepscreek 2y ago
> The most promising idea is to use reasoning models to generate data, and then train our non-reasoning models with the reasoning-embedded data. DeepSeek did precisely this with their LLama fine-tunes. You can try the 70B one here (might have to sign up): https://groq.com/groqcloud-makes-deepseek-r1-distill-llama-70b-available/ https://groq.com/groqcloud-makes-deepseek-r1-distill-llama-7...
- spyckie2 2y ago
  Yes, but I meant it slightly differently than the distills. The idea is to create the next gen SOTA non reasoning model with synthetic reasoning training data.
  - entropicdrifter 2y ago
    So you mean something like, "what if the baseline, off-the-cuff response for the next-gen models was tuned based on the results of the reasoning model excluding the reasoning itself?"
    - spyckie2 2y ago
      Exactly, albeit it may need the reasoning later to form the proper foundational logic in the weights.
vagabund 2y ago
> I highly doubt you are getting novel, high quality data. Why wouldn't you? Presumably the end user would try their use case on the existing model, and if it performs well, wouldn't bother with the expense of setting up an RL environment specific to their task. If it doesn't perform well, they do bother, and they have all the incentive in the world to get the verifier right -- which is not an extraordinarily sophisticated task if you're only using rules-based outcome rewards (as R1 and R1-Zero do)
fsndz 2y ago
I think we will have to move with pre-training and post-training efforts in parallel. What DeepSeek showed is that you first need to have a strong enough pretrained model. For that, we have to continue the acquisition of high quality, multilingual datasets. Then, when we have a stronger pretrained model, we can apply pure RL to get a reasoning model that we use only to generate synthetic reasoning data. We then use those synthetic reasoning data to fine-tune the original pretrained model and make it even stronger. https://transitions.substack.com/p/the-laymans-introduction-to-deepseek https://transitions.substack.com/p/the-laymans-introduction-...
visarga 2y ago
> I wonder if there is a cap to multi head attention architecture I don't think there is a cap other than having good data. The model learns all languages in the world, it has capacity. A simple model like AlphaZero beats humans at board games. As long as you have data, the model is not an obstacle. A LLM like AlphaProof is ranked silver medal at IMO.
godelski 2y ago
> I highly doubt you are getting novel, high quality data. That's not the point. The point is you reject low quality data, aka noise
- hassleblad23 2y ago
  And how would that work at inference time?
  - bobxmax 2y ago
    Why would it need to work at inference time?
gbasin 2y ago
the main bottleneck will be model depth... you can only do so much with N layers, and recurrence has proven to be way less efficient (for now)
verisimi 2y ago
*SOTA state of the art
varispeed 2y ago
My non technical cousin is a heavy paying user of ChatGPT, once she discovered that she can type incoherent stuff, with typos and whatnot and ChatGPT still will get the gist and produce satisfying answers, she will just type in tons of nonsense (to me) keep long chat sessions, complain it is getting slow and then get mad when I remind her to open new chat each time she has something new to ask that is not related to the previous chat. I have my doubt many users will provide valuable training data.
danbala 2y ago
shouldn't the whole idea be: get away from needing data at all? if a model can really reason, it should be able to figure things out on its own.
- FuriouslyAdrift 2y ago
  An LLM is just a really good parser connected to a lossy compressed corpus of data. They need to be open ended and self training to be truly useful. Reasoning is way far away...

mohsen1 2y ago

every time you respond to an AI model "no, you got that wrong, do it this way" you provide a very valuable piece of data to train on. With reasoning tokens there is just a lot more of that data to train on now

jeffbee 2y ago
If such labels are collected and used to retrain the model then yes. But these models are not learning online.
- jvanderbot 2y ago
  Really? Isn't that the point of RL used in the way R1 did? Provide a cost function (vs labels) and have it argue itself to greatness as measured by that cost function? I believe that's what GP meant by "respond", not telling GPT they were wrong.
  - daveguy 2y ago
    That is still inference. It is using a model generated from the RL process. The RL process is what used the cost function to add another model layer. Any online/continual learning would have to be performed by a different algorithm than classical LLM or RL. You can think of RL as a revision, but it still happens offline. Online/continual learning is still a very difficult problem in ML.
    - jvanderbot 2y ago
      Yes, that makes sense. We're both talking about offline learning.
- Cthulhu_ 2y ago
  ChatGPT came out and its interface was a chatbox and a thumbs up / thumbs down icon (or whichever) to rate the responses; surely that created a feedback loop of learning, like all machine learning has done for years now?
aprilthird2021 2y ago
So if I just pay OpenAI $200/mo, and randomly tell the AI, no that's wrong. I can stop the AI takeover?
- Exoristos 2y ago
  You can have our thank-you cards forwarded to your cell at Guantanamo Bay.
- dr_kiszonka 2y ago
  You would need a lot of pro accounts! I would be surprised if they didn't use any algorithms for detecting well poisoning.
nine_k 2y ago
This assumes that you give honest feedback. Efforts to feed deployed AI models various epistemic poisons abound in the wild.
- hammock 2y ago
  The AI models to begin with assume that a significant majority of the training material is honest/in good faith. So that is not new?
  - halfadot 2y ago
    AI models don't assume anything. AI models are just statistical tools. Their data is prepared by humans, who aren't morons. What is it with these super-ignorant AI critiques popping up everywhere?
    - baq 2y ago
      There’s so much data required for training it’d be surprising humans look at even a small subset of it at all. They need different statistical tools to clean it up. That’s where attacks will be concentrated, naturally, and this is why synthetic data will overtake real human data, just after ‘there isn’t enough data even if it’s too much already’.
    - hammock 2y ago
      Try a little benefit of the doubt, nuance or colloquialism. Or a bit of all three.
- BorisMelnik 2y ago
  I am not in this space, question: are there "bad actors" that are known to feed AI models with poisonous information?
  - tsunamifury 2y ago
    If the AI already has a larger knowledge domain space than the user then all users are bad actors. They are just too stupid to know it.
stetrain 2y ago
> What is today's date? >> Today's date is Tuesday, January 28, 2025. > No, you're wrong, today's date is actually Wednesday the 29th. >> My mistake. Yes, today's date is Wednesday, January 29th, 2025. Three months later in April when this tagged data is used to train the next iteration, the AI can successfully learn that today's date is actually January 29th.
- mr-wendel 2y ago
  But thats exactly what you get when you ask questions that require shifting, specific contextual knowledge. The model weights, by their nature, cannot encode that information. At best, you can only try to layer in contextual info like this as metadata during inference, akin to how other prompting layers exist. Even then, what up-to-date information should present for every round-trip is a matter of opinion and use-case.
  - lesuorac 2y ago
    > The model weights, by their nature, cannot encode that information. This is mostly irrelevant no? A binary digit by definition cannot encode more than 2 dates; so therefore we devise a more elaborate system (of using multiple digits). This is very similar to NYT's lawsuit against OpenAI where in addition to other claims, they claimed OpenAI maintainted a DB of NYT articles that they would directly grab from for a response. It's seems very feasible to maintain a DB or system of looking up real-time values like dates / weather.
- genewitch 2y ago
  the date is in the "system prompt", so the cron job that updates the prompts to the current date may be in a different time zone than you. 7f5dbb71f54322f271c4d3fc3aaa4d3282a1af5541d82b2cbc5aa10c1420b6bc
  - nemomarx 2y ago
    why can't they feed in user data like time zone and locale?
echelon 2y ago
> you provide a very valuable piece of data to train on We've been saying this "we get valuable data" thing since the 2010s [1]. When will our collective Netflix thumbs ups give us artificial super-intelligence? [1] Especially to investors. They love that line.
- genewitch 2y ago
  our collective netflix thumbs up indicators gave investors and netflix the confidence to deploy a series of adam sandler movies that cost 60 to 80 million US dollars to "make". So depending on who you are, the system might be working great.
  - beAbU 2y ago
    Through analytics Netflix should know exactly when people stop watching a series, or even when in a movie they exit out. They no doubt know this by user. They know exactly what makes you stay, and what makes you leave. I would not be surprised if in the near future movies and series are modifed on the fly to ensure users stay glued to their screens. In the distant future this might be done on a per user level.
    - JetSpiegel 2y ago
      They did it the other way around, every Netflix series is now designed like a soap opera, to be watched in the background.
deegles 2y ago
not being snarky, but what is the point of using the model if you already know enough to correct it into giving the right answer? an example that just occurred to me - if you asked it to generate an image of a mushroom that is safe to eat in your area, how would you tell it it was wrong? "oh, they never got back to me, I'll generate this image for others as well!"
- vincentperes 2y ago
  You constantly have to correct an AI when using it because it either didn't get the question right or you guide him towards a more narrowed answer. There is only more to learn.
- dematz 2y ago
  A common use of these models is asking for code, and maybe you don't know the answer or would take a while to figure it out. For example, here's some html, make it blue and centered. You could give the model feedback on if its answer worked or not, without knowing the correct answer yourself ahead of time.
  - deegles 2y ago
    I was using llama3 and deepseek-r1 literally to center an element in a div and they were not able to despite many prompts and variations. I guess I figured it out in the end but I'm not convinced I saved any time vs just carefully reading flexbox docs.
- Levitz 2y ago
  >not being snarky, but what is the point of using the model if you already know enough to correct it into giving the right answer? For your example, what if you want to show what such a mushroom looks like to a friend? What if you want to use it on a website?
  - nemomarx 2y ago
    I feel like conventional image search would be more reliable to get a good picture of a mushroom variety that you know about. Ideally going out into the woods to get one I suppose.
amluto 2y ago
Does it? If I say "no, you hallucinated basically the entire content of the response", then maybe a newer training set derived from that could train on the specific fact that that specific hallucinated response is hallucinated. This seems to be of dubious value in a training set.
- Vampiero 2y ago
  Nah I just insult it and tell it that it costs me 20 dollars a month and it's a huge disappointment
merrywhether 2y ago
Users can be adversarial to the “truth” (to the extent it exists) without being adversarial in intent. Dinosaur bones are either 65 million year old remnants of ancient creatures or decoys planted by a God during a 7 day creation, and a large proportion of humans earnestly believe either take. Choosing which of these to believe involves a higher level decision about fundamental worldviews. This is an extreme example, but incorporating “honest” human feedback on vaccines, dark matter, and countless other topics won’t lead to de facto improvements. I guess to put it another way: experts don’t learn from the masses. The average human isn’t an expert in anything, so incorporating the average feedback will pull a model away from expertise (imagine asking 100 people to give you grammar advice). You’d instead want to identify expert advice, but that’s impossible to do from looking at the advice itself without giving into a confirmation bias spiral. Humans use meta-signals like credentialing to augment their perception of received information, yet I doubt we’ll be having people upload their CV during signup to a chat service. And at the cutting edge level of expertise, the only real “knowledgeable” counterparties are the physical systems of reality themselves. I’m curious how takeoff is possible for a brain in a bottle that can’t test and verify any of its own conjectures. It can continually extrapolate down chains of thought, but that’s most likely to just carry and amplify errors.
- sirsinsalot 2y ago
  This is a good take. What models seem to be poor at is undoing their own thinking down a path even when they can test. If you let a model write code, test it, identify bugs and fix them, you get an increasingly obtuse and complex code base where errors happen more. The more it iterate the worse it gets. At the end of the day, written human language is a poor way of describing software. Even to a model. The code is the description. At the moment we describe solutions we want to see to the models and they aren't that smart about translating that to an unambiguous form. We are a long was off describing the problems and asking for a solution. Even when the model can test and iterate.
- benbosco 2y ago
  Dirac’s prediction of antimatter came from purely mathematical reasoning—before any experimental evidence existed. Testing and verifying conjectures requires the ability to extrapolate beyond known data, rather than from it, and the ability discard false leads based on theoretical reasoning, rather than statistical confidence. All of this is possible in a bottle, but laughably far beyond our current capabilities.

ClumsyPilot 2y ago

> The most promising idea is to use reasoning models to generate data, and then train our non-reasoning models with the reasoning-embedded data. Why is it promising, aren’t you potentially amplifying AI biases and errors?

pas 2y ago
it seems to work and seems very scalable, "reasoning" helps to counter biases (answers become longer, ie. the system uses more tokens which means more time to answer a question -- likely longer answers allow better differentiation of answers from each other in the "answer space") https://newsletter.languagemodels.co/i/155812052/large-scale-reasoning-oriented-reinforcement-learning-r-zero https://newsletter.languagemodels.co/i/155812052/large-scale... also from the posted article """ The R1-Zero training process is capable of creating its own internal domain specific language (“DSL”) in token space via RL optimization. This makes intuitive sense, as language itself is effectively a reasoning DSL. """

jcims 2y ago

>You might be lowering the cost of your training corpus by a few million dollars, but I highly doubt you are getting novel, high quality data. The large foundational models don't really need more empirical data about the world. ChatGPT already 'knows' way more than I do, probably by many orders of magnitude. Yet it's still spewing nonsense at me regularly because it doesn't know how to think like a human or interact with me in a human-like way. To that end, the ability for a company like OpenAI to collect novel data from interacting with real humans is a material advantage over their competition.

visarga 2y ago
> the ability for a company like OpenAI to collect novel data from interacting with real humans is a material advantage over their competition It's different kind of data from the R1 reasoning chains. When LLMs have human in the loop, the human provides help based off their personal experience and real world validation. Sometimes users take an idea from the LLM and try it in real life. Then come back later and discuss the outcomes. This is a real world testing loop. In order to judge if an AI response was useful, you can look at the following messages with a judge LLM. Using hindsight helps a lot here. Maybe it doesn't pan out and the user tries another approach, or maybe some innocuous idea was key to success later. It's hard to tell in the moment, but easy when you see what followed after that. This scales well - OpenAI has 300M users, I estimate up to 1 Trillion interactive tokens/day. The user base is very diverse, problems are diverse, and feedback comes from user experience and actual testing. They form an experience flywheel, the more problem solving they do, the smarter it gets, attracting more users.

fizx 2y ago

You're not getting new high-quality textual data for pre-training from your chat service. But you are potentially getting a lot of RL feedback on ambiguous problems.

deleted 2y ago

[deleted]

baq 2y ago

It doesn't need much. 1 good lucky answer in a 1000 or maybe 10k queries gives you the little exponential kick you need to improve. This is how the hockey stick take off looks like and we're already here - OpenAI has it, now deepseek has it, too. You can be sure others also have it; Anthropic at the very least, they just never announced it officially, but go read what their CEO has been speaking and writing about.

spyckie2 2y ago
I have looked a bit into Anthropic CEOs writings but if you can point in the right direction would be helpful!

sheepscreek 2y ago

> The most promising idea is to use reasoning models to generate data, and then train our non-reasoning models with the reasoning-embedded data. DeepSeek did precisely this with their LLama fine-tunes. You can try the 70B one here (might have to sign up): https://groq.com/groqcloud-makes-deepseek-r1-distill-llama-70b-available/ https://groq.com/groqcloud-makes-deepseek-r1-distill-llama-7...

spyckie2 2y ago
Yes, but I meant it slightly differently than the distills. The idea is to create the next gen SOTA non reasoning model with synthetic reasoning training data.
- entropicdrifter 2y ago
  So you mean something like, "what if the baseline, off-the-cuff response for the next-gen models was tuned based on the results of the reasoning model excluding the reasoning itself?"
  - spyckie2 2y ago
    Exactly, albeit it may need the reasoning later to form the proper foundational logic in the weights.

vagabund 2y ago

> I highly doubt you are getting novel, high quality data. Why wouldn't you? Presumably the end user would try their use case on the existing model, and if it performs well, wouldn't bother with the expense of setting up an RL environment specific to their task. If it doesn't perform well, they do bother, and they have all the incentive in the world to get the verifier right -- which is not an extraordinarily sophisticated task if you're only using rules-based outcome rewards (as R1 and R1-Zero do)

fsndz 2y ago

I think we will have to move with pre-training and post-training efforts in parallel. What DeepSeek showed is that you first need to have a strong enough pretrained model. For that, we have to continue the acquisition of high quality, multilingual datasets. Then, when we have a stronger pretrained model, we can apply pure RL to get a reasoning model that we use only to generate synthetic reasoning data. We then use those synthetic reasoning data to fine-tune the original pretrained model and make it even stronger. https://transitions.substack.com/p/the-laymans-introduction-to-deepseek https://transitions.substack.com/p/the-laymans-introduction-...

visarga 2y ago

> I wonder if there is a cap to multi head attention architecture I don't think there is a cap other than having good data. The model learns all languages in the world, it has capacity. A simple model like AlphaZero beats humans at board games. As long as you have data, the model is not an obstacle. A LLM like AlphaProof is ranked silver medal at IMO.

godelski 2y ago

> I highly doubt you are getting novel, high quality data. That's not the point. The point is you reject low quality data, aka noise

hassleblad23 2y ago
And how would that work at inference time?
- bobxmax 2y ago
  Why would it need to work at inference time?

gbasin 2y ago

the main bottleneck will be model depth... you can only do so much with N layers, and recurrence has proven to be way less efficient (for now)

verisimi 2y ago

*SOTA state of the art

varispeed 2y ago

My non technical cousin is a heavy paying user of ChatGPT, once she discovered that she can type incoherent stuff, with typos and whatnot and ChatGPT still will get the gist and produce satisfying answers, she will just type in tons of nonsense (to me) keep long chat sessions, complain it is getting slow and then get mad when I remind her to open new chat each time she has something new to ask that is not related to the previous chat. I have my doubt many users will provide valuable training data.

danbala 2y ago

shouldn't the whole idea be: get away from needing data at all? if a model can really reason, it should be able to figure things out on its own.

FuriouslyAdrift 2y ago
An LLM is just a really good parser connected to a lossy compressed corpus of data. They need to be open ended and self training to be truly useful. Reasoning is way far away...