4 ms·

If this is for real, in some ways it shows how small OpenAIs moat is. Once someone knows something is possible and the rough idea, the community can replicate

If this is for real, in some ways it shows how small OpenAIs moat is. Once someone knows something is possible and the rough idea, the community can replicate it in 4 weeks.

deleted 2y ago
[deleted]
jsheard 2y ago
Isn't Reflection supposed to be based on CoT like o1? It was originally released a week before o1 was, so if it was the real deal all along then OpenAI were outright beaten to the punch rather than replicated after the fact.
- ipsum2 2y ago
  No. CoT has been around for several years (Jan 2022) https://arxiv.org/abs/2201.11903 https://arxiv.org/abs/2201.11903. And so has Reflection (March 2023) https://arxiv.org/abs/2303.11366 https://arxiv.org/abs/2303.11366. This approach that was taken by Reflection is nothing new.
- nisten 2y ago
  yes and the massive increase to GPQA scores from o1 was attributed to this technique, so there is something there, despite the hard feelings of unproductive reddit users
- thorum 2y ago
  CoT moderately improves model performance, but all non-o1 models suck at actually thinking step by step effectively. If the task is not straightforward, they make obvious mistakes or default to guessing. OpenAI trained o1 to pick better steps in its chain of thought. (The moat is the dataset they used to do that.)
- bastawhiz 2y ago
  Maybe, if it wasn't an outright fraud. Arguably they didn't beat anyone to anything.
  - refulgentis 2y ago
    > Maybe, if it wasn't an outright fraud. I mean, it obviously wasn't, did you read the thing we're commenting on? n.b. At this point, you have all you need to replicate it. Far shy of "outright fraud", though, I'm sure there's a bailey for that motte.
    - bastawhiz 2y ago
      It's indisputably fraud. Multiple people after the original launch showed strong evidence that the hosted model they produced was simply proxying Claude and prompted it to censor its own name. It genuinely doesn't matter what they say, they committed blatant fraud, went out of their way to hide it, and now they're pretending like that didn't happen. The results might be perfectly reproducible, but their reputation is completely burned. This is not how you launch your company. Even if you don't care they didn't release anything at all of substance before the o1 launch. They didn't release usable weights, they didn't ship a working hosted model of their own. So no, they didn't beat OpenAI to anything.
      - refulgentis 2y ago
        > showed strong evidence that the hosted model they produced was simply proxying Claude and prompted it to censor its own name. s/strong/extremely weak from my perspective, also, see article > They didn't release usable weights, Yes they did. They just weren't benchmarking the same as the initial claim. > they didn't ship a working hosted model of their own. Yes they did. They just weren't benchmarking the same as the initial claim. > So no, they didn't beat OpenAI to anything. Not sure where the idea they "beat OpenAI" is coming from, certainly not from me. I agree they did not. > It's indisputably fraud. This is indisputably incorrect, as I am disputing it. Happy to talk it out, don't take my shortness as being disagreeable. In general, people handwave about tokenizer[1] or "Claude" missing in a response[2]. I honestly expected the HN thread here to be far more insightful, instead, I'm seeing that its indisputable it was fraud, based on repeating a couple observations gooner teens made last week, then made vast conclusions on. Which were obviously wrong if you looked at it as an engineer. [1] no one can get the actual tokens out of an API. gooner local LLM stans were setting max tokens to some number <= 10, asking the same question of both, and seeing answers of similar length. This is mundane and expected, especially in english, at such a short length. I expected technical observers to, at least even if they don't grok tokenization, to note they weren't able to get the same responses with temperature = 0.0. [2] covered in article

If this is for real, in some ways it shows how small OpenAIs moat is. Once someone knows something is possible and the rough idea, the community can replicate it in 4 weeks.

deleted 2y ago

[deleted]

jsheard 2y ago

Isn't Reflection supposed to be based on CoT like o1? It was originally released a week before o1 was, so if it was the real deal all along then OpenAI were outright beaten to the punch rather than replicated after the fact.

ipsum2 2y ago
No. CoT has been around for several years (Jan 2022) https://arxiv.org/abs/2201.11903 https://arxiv.org/abs/2201.11903. And so has Reflection (March 2023) https://arxiv.org/abs/2303.11366 https://arxiv.org/abs/2303.11366. This approach that was taken by Reflection is nothing new.
nisten 2y ago
yes and the massive increase to GPQA scores from o1 was attributed to this technique, so there is something there, despite the hard feelings of unproductive reddit users
thorum 2y ago
CoT moderately improves model performance, but all non-o1 models suck at actually thinking step by step effectively. If the task is not straightforward, they make obvious mistakes or default to guessing. OpenAI trained o1 to pick better steps in its chain of thought. (The moat is the dataset they used to do that.)
bastawhiz 2y ago
Maybe, if it wasn't an outright fraud. Arguably they didn't beat anyone to anything.
- refulgentis 2y ago
  > Maybe, if it wasn't an outright fraud. I mean, it obviously wasn't, did you read the thing we're commenting on? n.b. At this point, you have all you need to replicate it. Far shy of "outright fraud", though, I'm sure there's a bailey for that motte.
  - bastawhiz 2y ago
    It's indisputably fraud. Multiple people after the original launch showed strong evidence that the hosted model they produced was simply proxying Claude and prompted it to censor its own name. It genuinely doesn't matter what they say, they committed blatant fraud, went out of their way to hide it, and now they're pretending like that didn't happen. The results might be perfectly reproducible, but their reputation is completely burned. This is not how you launch your company. Even if you don't care they didn't release anything at all of substance before the o1 launch. They didn't release usable weights, they didn't ship a working hosted model of their own. So no, they didn't beat OpenAI to anything.
    - refulgentis 2y ago
      > showed strong evidence that the hosted model they produced was simply proxying Claude and prompted it to censor its own name. s/strong/extremely weak from my perspective, also, see article > They didn't release usable weights, Yes they did. They just weren't benchmarking the same as the initial claim. > they didn't ship a working hosted model of their own. Yes they did. They just weren't benchmarking the same as the initial claim. > So no, they didn't beat OpenAI to anything. Not sure where the idea they "beat OpenAI" is coming from, certainly not from me. I agree they did not. > It's indisputably fraud. This is indisputably incorrect, as I am disputing it. Happy to talk it out, don't take my shortness as being disagreeable. In general, people handwave about tokenizer[1] or "Claude" missing in a response[2]. I honestly expected the HN thread here to be far more insightful, instead, I'm seeing that its indisputable it was fraud, based on repeating a couple observations gooner teens made last week, then made vast conclusions on. Which were obviously wrong if you looked at it as an engineer. [1] no one can get the actual tokens out of an API. gooner local LLM stans were setting max tokens to some number <= 10, asking the same question of both, and seeing answers of similar length. This is mundane and expected, especially in english, at such a short length. I expected technical observers to, at least even if they don't grok tokenization, to note they weren't able to get the same responses with temperature = 0.0. [2] covered in article