3 ms·

It's indisputably fraud. Multiple people after the original launch showed strong evidence that the hosted model they produced was simply proxying Claude and pro

It's indisputably fraud. Multiple people after the original launch showed strong evidence that the hosted model they produced was simply proxying Claude and prompted it to censor its own name. It genuinely doesn't matter what they say, they committed blatant fraud, went out of their way to hide it, and now they're pretending like that didn't happen. The results might be perfectly reproducible, but their reputation is completely burned. This is not how you launch your company. Even if you don't care they didn't release anything at all of substance before the o1 launch. They didn't release usable weights, they didn't ship a working hosted model of their own. So no, they didn't beat OpenAI to anything.

refulgentis 2y ago
> showed strong evidence that the hosted model they produced was simply proxying Claude and prompted it to censor its own name. s/strong/extremely weak from my perspective, also, see article > They didn't release usable weights, Yes they did. They just weren't benchmarking the same as the initial claim. > they didn't ship a working hosted model of their own. Yes they did. They just weren't benchmarking the same as the initial claim. > So no, they didn't beat OpenAI to anything. Not sure where the idea they "beat OpenAI" is coming from, certainly not from me. I agree they did not. > It's indisputably fraud. This is indisputably incorrect, as I am disputing it. Happy to talk it out, don't take my shortness as being disagreeable. In general, people handwave about tokenizer[1] or "Claude" missing in a response[2]. I honestly expected the HN thread here to be far more insightful, instead, I'm seeing that its indisputable it was fraud, based on repeating a couple observations gooner teens made last week, then made vast conclusions on. Which were obviously wrong if you looked at it as an engineer. [1] no one can get the actual tokens out of an API. gooner local LLM stans were setting max tokens to some number <= 10, asking the same question of both, and seeing answers of similar length. This is mundane and expected, especially in english, at such a short length. I expected technical observers to, at least even if they don't grok tokenization, to note they weren't able to get the same responses with temperature = 0.0. [2] covered in article
- bastawhiz 2y ago
  I'm really not going to argue with you, because when faced with lots of little bits of compelling evidence from a whole bunch of sources (showing their work) versus the word of some guy on the Internet, I'll believe the evidence. Sahil didn't actually refute any of the concerns around the hosted model, he just acknowledged that it was weird and said he didn't know why. Great. That's useless. So much for "looking at it as an engineer". But what's great about the passage of time is that people can actually take what's presented in the article and try to replicate the benchmarks. And now that it's Friday, October 4th, and we've got this gem: https://x.com/mattshumer_/status/1842313328166907995 https://x.com/mattshumer_/status/1842313328166907995 So frankly it's still a fraud, now because even after the postmortem the results are still not reproducible. That's the whole point, right? That it does what it says on the tin. And it doesn't. This whole process could be a shell script that downloads and runs. They've had more time than they should need. Now, it's gone from a shell game to plain old academic dishonesty. If this was a published paper, it would be ripe for retraction.

refulgentis 2y ago

> showed strong evidence that the hosted model they produced was simply proxying Claude and prompted it to censor its own name. s/strong/extremely weak from my perspective, also, see article > They didn't release usable weights, Yes they did. They just weren't benchmarking the same as the initial claim. > they didn't ship a working hosted model of their own. Yes they did. They just weren't benchmarking the same as the initial claim. > So no, they didn't beat OpenAI to anything. Not sure where the idea they "beat OpenAI" is coming from, certainly not from me. I agree they did not. > It's indisputably fraud. This is indisputably incorrect, as I am disputing it. Happy to talk it out, don't take my shortness as being disagreeable. In general, people handwave about tokenizer[1] or "Claude" missing in a response[2]. I honestly expected the HN thread here to be far more insightful, instead, I'm seeing that its indisputable it was fraud, based on repeating a couple observations gooner teens made last week, then made vast conclusions on. Which were obviously wrong if you looked at it as an engineer. [1] no one can get the actual tokens out of an API. gooner local LLM stans were setting max tokens to some number <= 10, asking the same question of both, and seeing answers of similar length. This is mundane and expected, especially in english, at such a short length. I expected technical observers to, at least even if they don't grok tokenization, to note they weren't able to get the same responses with temperature = 0.0. [2] covered in article

bastawhiz 2y ago
I'm really not going to argue with you, because when faced with lots of little bits of compelling evidence from a whole bunch of sources (showing their work) versus the word of some guy on the Internet, I'll believe the evidence. Sahil didn't actually refute any of the concerns around the hosted model, he just acknowledged that it was weird and said he didn't know why. Great. That's useless. So much for "looking at it as an engineer". But what's great about the passage of time is that people can actually take what's presented in the article and try to replicate the benchmarks. And now that it's Friday, October 4th, and we've got this gem: https://x.com/mattshumer_/status/1842313328166907995 https://x.com/mattshumer_/status/1842313328166907995 So frankly it's still a fraud, now because even after the postmortem the results are still not reproducible. That's the whole point, right? That it does what it says on the tin. And it doesn't. This whole process could be a shell script that downloads and runs. They've had more time than they should need. Now, it's gone from a shell game to plain old academic dishonesty. If this was a published paper, it would be ripe for retraction.