5 ms·

Yip. It's pretty obvious this 'innovation' is just based off training data collected from chain-of-thought prompting by people, ie., the 'big leap forward' is j

Yip. It's pretty obvious this 'innovation' is just based off training data collected from chain-of-thought prompting by people, ie., the 'big leap forward' is just another dataset of people repairing chatgpt's lack of reasoning capabilities. No wonder then, that many of the benchmarks they've tested on would be no doubt, in that very training dataset, repaired expertly by people running those benchmarks on chatgpt. There's nothing really to 'expose' here.

bugglebeetle 2y ago
> the 'big leap forward' is just another dataset Yeah, that’s called machine learning.
- mjburgess 2y ago
  You may want to file a complaint with OpenAI then, in their latest interface they call sampling from these prior conversations they've recorded, "thinking".
  - accountnum 2y ago
    They're not sampling from prior conversations. The model constructs abstracted representations of the domain-specific reasoning traces. Then it applies these reasoning traces in various combinations to solve unseen problems. If you want to call that sampling, then you might as well call everything sampling.
    - mjburgess 2y ago
      They're generative models. By definition, they are sampling from a joint distribution of text tokens fit by approximation to an empirical distribution.
      - accountnum 2y ago
        Again, you're stretching definitions into meaninglessness. The way you are using "sampling" and "distribution" here applies to any system processing any information. Yes, humans as well. I can trivially define the entirety of all nerve impulses reaching and exiting your brain as a "distribution" in your usage of the term. And then all possible actions and experiences are just "sampling" that "distribution" as well. But that definition is meaningless.
        mjburgess 2y ago
        No, causation isnt distribution sampling. And there's a difference between, say, an extrinsic description of a system and it's essential properties. Eg., you can describe a coin flip as a sampling from the space, {H,T} -- but insofar as we're talking about an actual coin, there's a causal mechanism -- and this description fails (eg., one can design a coin flipper to deterministically flip to heads). In the case of a transformer model, and all generative statistical models, these are actually learning distributions. The model is essentially constituted by a fit to a prior distribution. And when computing a model output, it is sampling from this fit distribution. ie., the relevant state of the graphics card which computes an output token is fully described by an equation which is a sampling from an empirical distribution (of prior text tokens). Your nervous system is a causal mechanism which is not fully described by sampling from this outcome space. There is no where in your body that stores all possible bodily states in an outcome space: this space would require more atoms in the universe to store. So this isn't the case for any causal mechanism. Reality itself comprises essential properties which interact with each other in ways that cannot be reduced to sampling. Statistical models are therefore never models of reality essentially, but basically circumstantial approximations. I'm not stretching definitions into meaninglessness, these are the ones given by AI researchers, of which I am one.
        accountnum 2y ago
        I'm going to simply address what I think are your main points here. There is nowhere that an LLM stores all possible outputs. Causality can trivially be represented by sampling by including the ordering of events, which you also implicitly did for LLMs. The coin is an arbitrary distinction, you are never just modeling a coin, just as an LLM is never just modeling a word. You are also modeling an environment, and that model would capture whatever you used to influence the coin toss. You are fundamentally misunderstanding probability and randomness, and then using that misunderstanding to arbitrarily imply simplicity in the system you want to diminish, while failing to apply the same reasoning to any other. If you are indeed an AI researcher, which I highly doubt without you providing actual credentials, then you would know that you are being imprecise and using that imprecision to sneak in unfounded assumptions.
DiscourseFan 2y ago
It seems like the best AI models are increasingly just combinations of writings of various people thrown together. Like they hired a few hundred professors, journalists and writers to work with the model and create material for it, so you just get various combinations of their contributions. It's very telling that this model, for instance, is extraordinarily good at STEM related queries, but much worse (and worse even in comparison to GPT4) than English composition, probably because the former is where the money is to be made, in automating away essentially almost all engineering jobs.
- GaggiX 2y ago
  Do you have a source about OpenAI hiring a few hundred professors, journalists and writers? Because I honestly doubt.
  - mattkrause 2y ago
    A few recruiters have contacted me (a scientist) about doing RLHF and annotation on biomedical tasks. I don’t know if the eventual client was OpenAI or some other LLM provider but they seemed to have money to burn.
    - vidarh 2y ago
      I fill in gaps in my contracting with one of these providers, and I know who the ultimate client is, and if you were to list 4-5 options they'd be in there. I've also done work for another company doing work in this space that had at least 4-5 different clients in that space that I can't be sure about. So, yes, while I can't confirm if OpenAI does this, I know one of the big players do, and it's likely most of the other clients are among the top ones...
  - tough 2y ago
    Just all their chatgpt customers
  - COAGULOPATH 2y ago
    I've heard rumors that GPT4's training data included "a custom dataset of college textbooks", curated by hand. Nothing beyond that. https://www.reddit.com/r/mlscaling/comments/14wcy7m/comment/jrir37r/ https://www.reddit.com/r/mlscaling/comments/14wcy7m/comment/...
GaggiX 2y ago
>the 'big leap forward' is just another dataset of people repairing chatgpt's lack of reasoning capabilities. I think there is a really strong reinforcement learning component with the training of this model and how it has learned to perform the chain of thought.
- HarHarVeryFunny 2y ago
  Yes, but I suspect that the goals of the RL (in order to reason, we need to be able to "break down tricky steps into simpler ones", etc) were hand chosen, then a training set demonstrating these reasoning capabilities/components was constructed to match.
mlsu 2y ago
I would be dying to know how they square these product decisions against their corporate charter internally. From the charter: > We will actively cooperate with other research and policy institutions; we seek to create a global community working together to address AGI’s global challenges. > We are committed to providing public goods that help society navigate the path to AGI. Today this includes publishing most of our AI research, but we expect that safety and security concerns will reduce our traditional publishing in the future, while increasing the importance of sharing safety, policy, and standards research. It's obvious to everyone in the room what they actually are, because their largest competitor actually does what they say their mission is here -- but most for-profit capitalist enterprises definitely do not have stuff like this in their mission statement. I'm not even mad or sad, the ship sailed long ago. I just really want to know what things are like in there. If you're the manager who is making this decision, what mental gymnastics are you doing to justify this to yourself and your colleagues? Is there any resistance left on the inside or did they all leave with Ilya?
tivert 2y ago
> Yip. It's pretty obvious this 'innovation' is just based off training data collected from chain-of-thought prompting by people, ie., the 'big leap forward' is just another dataset of people repairing chatgpt's lack of reasoning capabilities. Which would be ChatGPT chat logs, correct? It would be interesting if people started feeding ChatGPT deliberately bad repairs due it's "lack of reasoning capabilities" (e.g. get a local LLM setup with some response delays to simulate a human and just let it talk and talk and talk to ChatGPT), and see how it affects its behavior over the long run.
- exe34 2y ago
  i suspect they can detect that in a similar way to capchas and "verify you're human by clicking the box".
  - tivert 2y ago
    > i suspect they can detect that in a similar way to capchas and "verify you're human by clicking the box". I'm not so sure. IIRC, capchas are pretty much a solved problem, if you don't mind the cost of a little bit of human interaction (e.g. your interface pops up a captcha solver box when necessary, and is solved either by the bot's operator or some professional captcha-solver in a low-wage country).
    - immibis 2y ago
      there are services that solve captchas automatically with above-human success rates
- vidarh 2y ago
  These logs get manually reviewed by humans, sometimes annotated by automated systems first. The setups for manual reviews typically involve half a dozen steps with different people reviewing, comparing reviews, revising comparisons, and overseeing the revisions (source: I've done contract work at every stage of that process, have half a dozen internal documents for a company providing this service open right now). A lot of money is being pumped into automating parts of this, but a lot of money still also flows into manually reviewing and quality-assuring the whole process. Any logs showing significant quality declines would get picked up and filtered out pretty quickly.
exe34 2y ago
i think it's funny, every time you implement a clever solution to call gpt and get a decent answer, they get to use your idea in their product. what other project gets to crowdsource ideas and take credit for them like this? ps: actually maybe Amazon marketplace. probably others too.
- solveit 2y ago
  Most projects with an active user-created mods community are heavily influenced by them.
- egypturnash 2y ago
  "sherlocking" has been a thing since 2002, when Apple incorporated a bunch of third-party ideas for extending their "Sherlock" search tool into the official release. https://thehustle.co/sherlocking-explained https://thehustle.co/sherlocking-explained
- immibis 2y ago
  Amazon definitely does this, to great effect.
janalsncm 2y ago
Do people really expect anything different? There is a ton of cross-pollination in Silicon Valley. Keeping these innovations completely under wraps would be akin to a massive conspiracy. A peacetime Manhattan Project where everyone has a smartphone, a Twitter presence, and sleeps in their own bed. Frankly I am even skeptical of US-China separation at the moment. If Chinese scientists at e.g. Huawei somehow came up with the secret sauce to AGI tomorrow, no research group is so far behind that they couldn’t catch up pretty quickly. We saw this with ChatGPT/Claude/Gemini before, none of which are light years ahead of another. Of course this could change in the future. This is actually among the best case scenarios for research. It means that a preemptive strike on data centers is still off the table for now. (Sorry Eleazar)
sandspar 2y ago
It's been out for 24 hours and you make an extremely confident and dismissive claim. If you had to make a dollar bet that you precisely understand what's happening under the hood, exactly how much money would you bet?
golol 2y ago
What are you basing this one? The one thing that is very clearly stated up front is that this innovation is based on reinforcement learning. You dok't even have a good idea what the CoT looks like because those little summary snippets that the ChatGPT UI gives you are nothing substantial.
- mjburgess 2y ago
  People repairing chatgpt replies with additional prompts is reinforcement learning training data. "Reinforcement learning", just like any term used by AI researchers, is an extremely flexible, pseudo-psychological reskin of some pretty trivial stuff.

bugglebeetle 2y ago

> the 'big leap forward' is just another dataset Yeah, that’s called machine learning.

mjburgess 2y ago
You may want to file a complaint with OpenAI then, in their latest interface they call sampling from these prior conversations they've recorded, "thinking".
- accountnum 2y ago
  They're not sampling from prior conversations. The model constructs abstracted representations of the domain-specific reasoning traces. Then it applies these reasoning traces in various combinations to solve unseen problems. If you want to call that sampling, then you might as well call everything sampling.
  - mjburgess 2y ago
    They're generative models. By definition, they are sampling from a joint distribution of text tokens fit by approximation to an empirical distribution.
    - accountnum 2y ago
      Again, you're stretching definitions into meaninglessness. The way you are using "sampling" and "distribution" here applies to any system processing any information. Yes, humans as well. I can trivially define the entirety of all nerve impulses reaching and exiting your brain as a "distribution" in your usage of the term. And then all possible actions and experiences are just "sampling" that "distribution" as well. But that definition is meaningless.
      - mjburgess 2y ago
        No, causation isnt distribution sampling. And there's a difference between, say, an extrinsic description of a system and it's essential properties. Eg., you can describe a coin flip as a sampling from the space, {H,T} -- but insofar as we're talking about an actual coin, there's a causal mechanism -- and this description fails (eg., one can design a coin flipper to deterministically flip to heads). In the case of a transformer model, and all generative statistical models, these are actually learning distributions. The model is essentially constituted by a fit to a prior distribution. And when computing a model output, it is sampling from this fit distribution. ie., the relevant state of the graphics card which computes an output token is fully described by an equation which is a sampling from an empirical distribution (of prior text tokens). Your nervous system is a causal mechanism which is not fully described by sampling from this outcome space. There is no where in your body that stores all possible bodily states in an outcome space: this space would require more atoms in the universe to store. So this isn't the case for any causal mechanism. Reality itself comprises essential properties which interact with each other in ways that cannot be reduced to sampling. Statistical models are therefore never models of reality essentially, but basically circumstantial approximations. I'm not stretching definitions into meaninglessness, these are the ones given by AI researchers, of which I am one.
        accountnum 2y ago
        I'm going to simply address what I think are your main points here. There is nowhere that an LLM stores all possible outputs. Causality can trivially be represented by sampling by including the ordering of events, which you also implicitly did for LLMs. The coin is an arbitrary distinction, you are never just modeling a coin, just as an LLM is never just modeling a word. You are also modeling an environment, and that model would capture whatever you used to influence the coin toss. You are fundamentally misunderstanding probability and randomness, and then using that misunderstanding to arbitrarily imply simplicity in the system you want to diminish, while failing to apply the same reasoning to any other. If you are indeed an AI researcher, which I highly doubt without you providing actual credentials, then you would know that you are being imprecise and using that imprecision to sneak in unfounded assumptions.

DiscourseFan 2y ago

It seems like the best AI models are increasingly just combinations of writings of various people thrown together. Like they hired a few hundred professors, journalists and writers to work with the model and create material for it, so you just get various combinations of their contributions. It's very telling that this model, for instance, is extraordinarily good at STEM related queries, but much worse (and worse even in comparison to GPT4) than English composition, probably because the former is where the money is to be made, in automating away essentially almost all engineering jobs.

GaggiX 2y ago
Do you have a source about OpenAI hiring a few hundred professors, journalists and writers? Because I honestly doubt.
- mattkrause 2y ago
  A few recruiters have contacted me (a scientist) about doing RLHF and annotation on biomedical tasks. I don’t know if the eventual client was OpenAI or some other LLM provider but they seemed to have money to burn.
  - vidarh 2y ago
    I fill in gaps in my contracting with one of these providers, and I know who the ultimate client is, and if you were to list 4-5 options they'd be in there. I've also done work for another company doing work in this space that had at least 4-5 different clients in that space that I can't be sure about. So, yes, while I can't confirm if OpenAI does this, I know one of the big players do, and it's likely most of the other clients are among the top ones...
- tough 2y ago
  Just all their chatgpt customers
- COAGULOPATH 2y ago
  I've heard rumors that GPT4's training data included "a custom dataset of college textbooks", curated by hand. Nothing beyond that. https://www.reddit.com/r/mlscaling/comments/14wcy7m/comment/jrir37r/ https://www.reddit.com/r/mlscaling/comments/14wcy7m/comment/...

GaggiX 2y ago

>the 'big leap forward' is just another dataset of people repairing chatgpt's lack of reasoning capabilities. I think there is a really strong reinforcement learning component with the training of this model and how it has learned to perform the chain of thought.

HarHarVeryFunny 2y ago
Yes, but I suspect that the goals of the RL (in order to reason, we need to be able to "break down tricky steps into simpler ones", etc) were hand chosen, then a training set demonstrating these reasoning capabilities/components was constructed to match.

mlsu 2y ago

I would be dying to know how they square these product decisions against their corporate charter internally. From the charter: > We will actively cooperate with other research and policy institutions; we seek to create a global community working together to address AGI’s global challenges. > We are committed to providing public goods that help society navigate the path to AGI. Today this includes publishing most of our AI research, but we expect that safety and security concerns will reduce our traditional publishing in the future, while increasing the importance of sharing safety, policy, and standards research. It's obvious to everyone in the room what they actually are, because their largest competitor actually does what they say their mission is here -- but most for-profit capitalist enterprises definitely do not have stuff like this in their mission statement. I'm not even mad or sad, the ship sailed long ago. I just really want to know what things are like in there. If you're the manager who is making this decision, what mental gymnastics are you doing to justify this to yourself and your colleagues? Is there any resistance left on the inside or did they all leave with Ilya?

tivert 2y ago

> Yip. It's pretty obvious this 'innovation' is just based off training data collected from chain-of-thought prompting by people, ie., the 'big leap forward' is just another dataset of people repairing chatgpt's lack of reasoning capabilities. Which would be ChatGPT chat logs, correct? It would be interesting if people started feeding ChatGPT deliberately bad repairs due it's "lack of reasoning capabilities" (e.g. get a local LLM setup with some response delays to simulate a human and just let it talk and talk and talk to ChatGPT), and see how it affects its behavior over the long run.

exe34 2y ago
i suspect they can detect that in a similar way to capchas and "verify you're human by clicking the box".
- tivert 2y ago
  > i suspect they can detect that in a similar way to capchas and "verify you're human by clicking the box". I'm not so sure. IIRC, capchas are pretty much a solved problem, if you don't mind the cost of a little bit of human interaction (e.g. your interface pops up a captcha solver box when necessary, and is solved either by the bot's operator or some professional captcha-solver in a low-wage country).
  - immibis 2y ago
    there are services that solve captchas automatically with above-human success rates
vidarh 2y ago
These logs get manually reviewed by humans, sometimes annotated by automated systems first. The setups for manual reviews typically involve half a dozen steps with different people reviewing, comparing reviews, revising comparisons, and overseeing the revisions (source: I've done contract work at every stage of that process, have half a dozen internal documents for a company providing this service open right now). A lot of money is being pumped into automating parts of this, but a lot of money still also flows into manually reviewing and quality-assuring the whole process. Any logs showing significant quality declines would get picked up and filtered out pretty quickly.

exe34 2y ago

i think it's funny, every time you implement a clever solution to call gpt and get a decent answer, they get to use your idea in their product. what other project gets to crowdsource ideas and take credit for them like this? ps: actually maybe Amazon marketplace. probably others too.

solveit 2y ago
Most projects with an active user-created mods community are heavily influenced by them.
egypturnash 2y ago
"sherlocking" has been a thing since 2002, when Apple incorporated a bunch of third-party ideas for extending their "Sherlock" search tool into the official release. https://thehustle.co/sherlocking-explained https://thehustle.co/sherlocking-explained
immibis 2y ago
Amazon definitely does this, to great effect.

janalsncm 2y ago

Do people really expect anything different? There is a ton of cross-pollination in Silicon Valley. Keeping these innovations completely under wraps would be akin to a massive conspiracy. A peacetime Manhattan Project where everyone has a smartphone, a Twitter presence, and sleeps in their own bed. Frankly I am even skeptical of US-China separation at the moment. If Chinese scientists at e.g. Huawei somehow came up with the secret sauce to AGI tomorrow, no research group is so far behind that they couldn’t catch up pretty quickly. We saw this with ChatGPT/Claude/Gemini before, none of which are light years ahead of another. Of course this could change in the future. This is actually among the best case scenarios for research. It means that a preemptive strike on data centers is still off the table for now. (Sorry Eleazar)

sandspar 2y ago

It's been out for 24 hours and you make an extremely confident and dismissive claim. If you had to make a dollar bet that you precisely understand what's happening under the hood, exactly how much money would you bet?

golol 2y ago

What are you basing this one? The one thing that is very clearly stated up front is that this innovation is based on reinforcement learning. You dok't even have a good idea what the CoT looks like because those little summary snippets that the ChatGPT UI gives you are nothing substantial.

mjburgess 2y ago
People repairing chatgpt replies with additional prompts is reinforcement learning training data. "Reinforcement learning", just like any term used by AI researchers, is an extremely flexible, pseudo-psychological reskin of some pretty trivial stuff.