4 ms·

OpenAI has been irrelevant for a while now. All of the new and exciting developments on AI are coming from other places. ClosedAI is no longer the driver of c

by buyucu 2y ago

OpenAI has been irrelevant for a while now. All of the new and exciting developments on AI are coming from other places. ClosedAI is no longer the driver of change and innovation.

nickthegreek 2y ago
The other models are literally distilling OpenAI’s models into theirs.
- demosthanos 2y ago
  So it's been claimed, but has it been proven yet? I'm not even sure what is being alleged there—o1's reasoning tokens are kept secret precisely to avoid the kind of distillation that's being alleged. How can you distill a reasoning process given only the final output?
  - ipaddr 2y ago
    The outputting that they are chatGPT from deepseek is a big clue.
    - TheKGB 2y ago
      [dead]
  - Nevermark 2y ago
    You really can distill a model to a smaller one just by training on the first model's behavior. The first model has a much harder problem. It must figure out a way to iron out the differences between all the different models (i.e. multiple humans, or noise in the dataset) it is trained on, to come up with its single viewpoint of relationships. It must then be tuned to optimally fit its purpose, instead of simply operating like a (composite) rando human. The second model then has these advantages: 1. Trained on a single model, whose behavior comes out of consistent learned relationships, as apposed to many inconsistent models with noise (lots of different humans operating differently at any given time for reasons beyond what the data can show). 2. Trained on post-tuned model behavior directly. One-step training is a considerably simpler problem than two-step. It doesn't learning anything it will need to unlearn or modify. 3. These simplifications both reduce the necessary size of the second model. And with these combined advantages (a simpler more consistent single-system to model, smaller size, and fewer training steps), it can be trained much faster. There, of course, can be disadvantages. The distilled model may not be as versatile, if the data used to prompt the original model isn't as diverse as the first models data. So prompt data for training still matters.
    - demosthanos 2y ago
      A bit late to reply, but none of this actually answers my question. I don't doubt distillation from behavior is possible, I doubt that it's possible when 90% of o1's behavior is never returned from the API. If the chain of thought process is what improves the results, then distillation without the chain of thought to train on should not produce comparable results.
- orbital-decay 2y ago
  Do they? Why doesn't this happen to Claude then? I've been hearing this for a while, but never saw any evidence beyond the contamination of the dataset with GPT slop that is all over the web. Just by sending anything to the competitors you're giving up a large part of your know how before you even finish your product, that's a big incentive against doing that.
  - nickthegreek 2y ago
    Who said it isn’t happening to Claude? Companies are 100% using these big players to generate synthetic data. Distillation is extremely powerful. How is this even in question?
    - nullc 2y ago
      OpenAI conceals probabilities so how is anyone distilling from it?
- kossTKR 2y ago
  And OpenAI based their tech on a Google paper again building on years of public academic research so what's the point exactly here? OpenAI was just first out of the gates, there'll always be some company that's first, essence is how they handle their leadership, and they've sadly been absolutely terrible and scummy. Actually i think Google was a pretty good example of the exact opposite, decades of "actually not being evil", while openAI switched up 1 second after launch.
  - nickthegreek 2y ago
    > so what’s the point exactly here. What is your point? OpenAI wasn’t the first out of that gate as your own argument cites Google prior. All these companies are predatory, who is arguing against that? OP said OpenAi was irrelevant. That’s just dumb. They are not. Feel free to advance an argument in favor of that narrative if you wish as I was just trying to provide a single example that shows that some of these lightweight models are building directly off the backs of giants spending the big money. I find nothing wrong with distillation and am excited about companies like DeepSeek.
  - ipaddr 2y ago
    Google wasn't the first search engine but they were the best marketing google = search. That's where we are with openai. Google search was a better product at the time and chatGPT 3.5 was a breakthrough the public used. Fast forward and some will say Google isn't the best search engine anymore (kagi, duckduckgo, yandex offer different experiences) but people still think of google=search. Same with chatGPT. Claude may be better for coding or gemini better are searching or Deepseek cheaper but equal but chatGPT is a verb and will live on like Intel inside long after it's actual value has declined.
    - williamcotton 2y ago
      Google was so much better than AltaVista that I just can’t buy that it was marketing that pushed them to the forefront of search.
      - 2y ago
- TheKGB 2y ago
  [dead]
- buyucu 2y ago
  I don't know about OpenAI, but I'm generating training data from Deepseek 671b to train smaller models.
mrcwinn 2y ago
That’s quite a world you’ve constructed!
jug 2y ago
I think OpenAI is currently in this position where they are still industry standard, but also not leading. Deepseek R1 beat o1 on perf/cost with similar perf at a fraction of the cost. o3-mini is judged as ”weird” and quite hit and miss on coding (basically the sole reason for its existence) with a sky high SimpleQA hallucination rate due to its limited scope, probably beat by Sonnet 3.7 by a fairly large margin. Still, being early with a product and still often ”good enough” still takes them a long way. I think GPT-5 and where their competition will be then will be quite important for OpenAI though. I think the signs on the horizon is that everyone will close up on each other as we hit the diminishing returns, so the underlying business model, integrations, enterprise reach, marketing and market share will probably be king rather than the underlying LLM in 2026. Since GPT-5 is meant to select the best model behind the scenes, one issue might be that users won’t have the same confidence in the model, feeling like it’s deciding for them or OpenAI tuning it to err on the side of being cheap.
TheKGB 2y ago
[dead]

by buyucu 2y ago

OpenAI has been irrelevant for a while now. All of the new and exciting developments on AI are coming from other places. ClosedAI is no longer the driver of change and innovation.

nickthegreek 2y ago

The other models are literally distilling OpenAI’s models into theirs.

demosthanos 2y ago
So it's been claimed, but has it been proven yet? I'm not even sure what is being alleged there—o1's reasoning tokens are kept secret precisely to avoid the kind of distillation that's being alleged. How can you distill a reasoning process given only the final output?
- ipaddr 2y ago
  The outputting that they are chatGPT from deepseek is a big clue.
  - TheKGB 2y ago
    [dead]
- Nevermark 2y ago
  You really can distill a model to a smaller one just by training on the first model's behavior. The first model has a much harder problem. It must figure out a way to iron out the differences between all the different models (i.e. multiple humans, or noise in the dataset) it is trained on, to come up with its single viewpoint of relationships. It must then be tuned to optimally fit its purpose, instead of simply operating like a (composite) rando human. The second model then has these advantages: 1. Trained on a single model, whose behavior comes out of consistent learned relationships, as apposed to many inconsistent models with noise (lots of different humans operating differently at any given time for reasons beyond what the data can show). 2. Trained on post-tuned model behavior directly. One-step training is a considerably simpler problem than two-step. It doesn't learning anything it will need to unlearn or modify. 3. These simplifications both reduce the necessary size of the second model. And with these combined advantages (a simpler more consistent single-system to model, smaller size, and fewer training steps), it can be trained much faster. There, of course, can be disadvantages. The distilled model may not be as versatile, if the data used to prompt the original model isn't as diverse as the first models data. So prompt data for training still matters.
  - demosthanos 2y ago
    A bit late to reply, but none of this actually answers my question. I don't doubt distillation from behavior is possible, I doubt that it's possible when 90% of o1's behavior is never returned from the API. If the chain of thought process is what improves the results, then distillation without the chain of thought to train on should not produce comparable results.
orbital-decay 2y ago
Do they? Why doesn't this happen to Claude then? I've been hearing this for a while, but never saw any evidence beyond the contamination of the dataset with GPT slop that is all over the web. Just by sending anything to the competitors you're giving up a large part of your know how before you even finish your product, that's a big incentive against doing that.
- nickthegreek 2y ago
  Who said it isn’t happening to Claude? Companies are 100% using these big players to generate synthetic data. Distillation is extremely powerful. How is this even in question?
  - nullc 2y ago
    OpenAI conceals probabilities so how is anyone distilling from it?
kossTKR 2y ago
And OpenAI based their tech on a Google paper again building on years of public academic research so what's the point exactly here? OpenAI was just first out of the gates, there'll always be some company that's first, essence is how they handle their leadership, and they've sadly been absolutely terrible and scummy. Actually i think Google was a pretty good example of the exact opposite, decades of "actually not being evil", while openAI switched up 1 second after launch.
- nickthegreek 2y ago
  > so what’s the point exactly here. What is your point? OpenAI wasn’t the first out of that gate as your own argument cites Google prior. All these companies are predatory, who is arguing against that? OP said OpenAi was irrelevant. That’s just dumb. They are not. Feel free to advance an argument in favor of that narrative if you wish as I was just trying to provide a single example that shows that some of these lightweight models are building directly off the backs of giants spending the big money. I find nothing wrong with distillation and am excited about companies like DeepSeek.
- ipaddr 2y ago
  Google wasn't the first search engine but they were the best marketing google = search. That's where we are with openai. Google search was a better product at the time and chatGPT 3.5 was a breakthrough the public used. Fast forward and some will say Google isn't the best search engine anymore (kagi, duckduckgo, yandex offer different experiences) but people still think of google=search. Same with chatGPT. Claude may be better for coding or gemini better are searching or Deepseek cheaper but equal but chatGPT is a verb and will live on like Intel inside long after it's actual value has declined.
  - williamcotton 2y ago
    Google was so much better than AltaVista that I just can’t buy that it was marketing that pushed them to the forefront of search.
    - 2y ago
TheKGB 2y ago
[dead]
buyucu 2y ago
I don't know about OpenAI, but I'm generating training data from Deepseek 671b to train smaller models.

mrcwinn 2y ago

That’s quite a world you’ve constructed!

jug 2y ago

I think OpenAI is currently in this position where they are still industry standard, but also not leading. Deepseek R1 beat o1 on perf/cost with similar perf at a fraction of the cost. o3-mini is judged as ”weird” and quite hit and miss on coding (basically the sole reason for its existence) with a sky high SimpleQA hallucination rate due to its limited scope, probably beat by Sonnet 3.7 by a fairly large margin. Still, being early with a product and still often ”good enough” still takes them a long way. I think GPT-5 and where their competition will be then will be quite important for OpenAI though. I think the signs on the horizon is that everyone will close up on each other as we hit the diminishing returns, so the underlying business model, integrations, enterprise reach, marketing and market share will probably be king rather than the underlying LLM in 2026. Since GPT-5 is meant to select the best model behind the scenes, one issue might be that users won’t have the same confidence in the model, feeling like it’s deciding for them or OpenAI tuning it to err on the side of being cheap.

TheKGB 2y ago

[dead]