5 ms·

Sahil Chaudhary of GlaiveAI perpetuated fraud, where he replaced the model that he "trained" with other backend ML providers. He still has not given a reason wh

by ipsum2 2y ago

Sahil Chaudhary of GlaiveAI perpetuated fraud, where he replaced the model that he "trained" with other backend ML providers. He still has not given a reason why "Claude" the string would be missing, just magically happened, despite the base model, Llama3.1-70B having no issues producing the text "Claude" nor the dataset missing the string "Claude"! Note that there was additional proof, besides missing the string "Claude", by matching the max number of tokens the model was able to produce. This is more technical, but chatGPT, Claude, Llama all have different tokenizers, so words are be broken up into different sections. The API consistently did NOT match the base model tokenizer (Llama), instead, producing the same number of tokens as Claude. Companies and individuals should probably avoid GlaiveAI and Matt Shumer less they get scammed too.

nisten 2y ago
they're 140Gb folders, for each checkpoint, yes file corruption happens and as for the fraud part...it was an opensource model release that did not meet the claimed benchmarks when people tried to replicate it
- alsodumb 2y ago
  Are you telling me someone trained a huge model, and served it for hours to tons of users, and had only one instance of the checkpoint? I call BS. The model being open-source doesn't mean what they could have gotten away with, or tried to, isn't fraud.
  - bhouston 2y ago
    He served tons of people from his personal laptop? How is that possible? A 70B LLM is pretty taxing even to serve a single user let along the crush of users that tried out this new hyped model no? What am I missing?
    - deleted 2y ago
      [deleted]
- bastawhiz 2y ago
  The fraud part was multiple independent sources producing fairly indisputable evidence that their "hosted" version of the model was just running GPT and Claude. That alone is enough to completely discredit absolutely everything about this work. As for corruption, I don't believe the excuse "yes file corruption happens". They're model weights. If this was trained (in real life) it was done on some serious hardware with disks with error correction. They weren't storing the checkpoints on microSD cards. It's certainly possible that there was really unfortunate luck and there was corruption, but I don't find that excuse to be plausible. Especially when this is your business (and launch!)
- ipsum2 2y ago
  Definition of fraud, from Google: * wrongful or criminal deception intended to result in financial or personal gain. * a person or thing intended to deceive others, typically by unjustifiably claiming or being credited with accomplishments or qualities. Since they were advertising GlaiveAI as this magical source of data where they trained a model that performed better than Claude and chatGPT, I think this firmly falls into that camp! Your definitions may be different than mine.
  - nisten 2y ago
    it was a free opensource model release, the api was not for sale, there are literally over a million FREE models on huggingface.
    - alsodumb 2y ago
      Who cares if the model was free? No one said they were trying to commit fraud by releasing that model, they were trying to commit fraud by subtly advertising that their companies/products had the secret sauce to make state-of-the-art models which they obviously didn't.
lolinder 2y ago
Sorry, I'm having trouble finding more information about this—what is the significance of the model being unable to produce the string "Claude"? Was this some sort of half-hearted censorship to prevent it from disclosing its name? Where can I read more?
- ipsum2 2y ago
  https://x.com/shinboson/status/1832933747529834747 https://x.com/shinboson/status/1832933747529834747 is the best summary. if you don't have a twitter account: https://threadreaderapp.com/thread/1832933768031588622.html https://threadreaderapp.com/thread/1832933768031588622.html
  - lolinder 2y ago
    Thanks, this is helpful!
- 0cf8612b2e1e 2y ago
  Here was a HN post which links to a Reddit thread about this. Multiple users find evidence that the model was a lie. The most damning to me was that the tokenizer was not Llama’s, which cannot be explained away. https://news.ycombinator.com/item?id=41484981 https://news.ycombinator.com/item?id=41484981

by ipsum2 2y ago

nisten 2y ago

they're 140Gb folders, for each checkpoint, yes file corruption happens and as for the fraud part...it was an opensource model release that did not meet the claimed benchmarks when people tried to replicate it

alsodumb 2y ago
Are you telling me someone trained a huge model, and served it for hours to tons of users, and had only one instance of the checkpoint? I call BS. The model being open-source doesn't mean what they could have gotten away with, or tried to, isn't fraud.
- bhouston 2y ago
  He served tons of people from his personal laptop? How is that possible? A 70B LLM is pretty taxing even to serve a single user let along the crush of users that tried out this new hyped model no? What am I missing?
  - deleted 2y ago
    [deleted]
bastawhiz 2y ago
The fraud part was multiple independent sources producing fairly indisputable evidence that their "hosted" version of the model was just running GPT and Claude. That alone is enough to completely discredit absolutely everything about this work. As for corruption, I don't believe the excuse "yes file corruption happens". They're model weights. If this was trained (in real life) it was done on some serious hardware with disks with error correction. They weren't storing the checkpoints on microSD cards. It's certainly possible that there was really unfortunate luck and there was corruption, but I don't find that excuse to be plausible. Especially when this is your business (and launch!)
ipsum2 2y ago
Definition of fraud, from Google: * wrongful or criminal deception intended to result in financial or personal gain. * a person or thing intended to deceive others, typically by unjustifiably claiming or being credited with accomplishments or qualities. Since they were advertising GlaiveAI as this magical source of data where they trained a model that performed better than Claude and chatGPT, I think this firmly falls into that camp! Your definitions may be different than mine.
- nisten 2y ago
  it was a free opensource model release, the api was not for sale, there are literally over a million FREE models on huggingface.
  - alsodumb 2y ago
    Who cares if the model was free? No one said they were trying to commit fraud by releasing that model, they were trying to commit fraud by subtly advertising that their companies/products had the secret sauce to make state-of-the-art models which they obviously didn't.

lolinder 2y ago

Sorry, I'm having trouble finding more information about this—what is the significance of the model being unable to produce the string "Claude"? Was this some sort of half-hearted censorship to prevent it from disclosing its name? Where can I read more?

ipsum2 2y ago
https://x.com/shinboson/status/1832933747529834747 https://x.com/shinboson/status/1832933747529834747 is the best summary. if you don't have a twitter account: https://threadreaderapp.com/thread/1832933768031588622.html https://threadreaderapp.com/thread/1832933768031588622.html
- lolinder 2y ago
  Thanks, this is helpful!
0cf8612b2e1e 2y ago
Here was a HN post which links to a Reddit thread about this. Multiple users find evidence that the model was a lie. The most damning to me was that the tokenizer was not Llama’s, which cannot be explained away. https://news.ycombinator.com/item?id=41484981 https://news.ycombinator.com/item?id=41484981