4 ms·

> “It is (relatively) easy to copy something that you know works,” Altman tweeted. “It is extremely hard to do something new, risky, and difficult when you don’

by thorum 2y ago

> “It is (relatively) easy to copy something that you know works,” Altman tweeted. “It is extremely hard to do something new, risky, and difficult when you don’t know if it will work.” The humor/hypocrisy of the situation aside, it does seem to be true that OpenAI is consistently the one coming up with new ideas first (GPT 4, o1, 4o-style multimodality, voice chat, DALL-E, …) and then other companies reproduce their work, and get more credit because they actually publish the research. Unfortunately for them it’s challenging to profit in the long term from being first in this space and the time it takes for each new idea to be reproduced is getting shorter.

spencerflem 2y ago
Fortunately, OpenAI doesn't need to make money because they are a nonprofit dedicated to the safe and transparent advancement of AI for all of humanity
- mjburgess 2y ago
  ...somewhere a yacht salesman cried out in terror
turtlesdown11 2y ago
> other companies reproduce their work, and get more credit because they actually publish the research. I don't understand, you mean OpenAI isn't releasing open models and openly publishing their research?
- Tostino 2y ago
  Are you being sarcastic (honestly, it's hard to tell after reading as many uninformed takes in the past week as I have). No, they aren't (other than whisper). Their "papers" are closer to marketing materials. Very intentionally leaving out tons of technical information.
  - KolmogorovComp 2y ago
    They are being sarcastic.
  - sota_pop 2y ago
    /s
actuallyalys 2y ago
There’s some truth in that, but isn’t making a radically cheaper version also a new idea that deepseek didn’t know whether it would work? I mean, there was already research into distillation, but there was already research into some of (most of?) OpenAI’s ideas.
- janalsncm 2y ago
  Yes, for people who look into the research Deepseek released, there are a good number of novelties which enabled much cheaper R&D. For example, improvements to Mixture of Experts modules and Multi-head Latent Attention. If you have infinite money, you don’t need to innovate there, but DeepSeek didn’t.
weego 2y ago
Boy who stole test papers complains about child copying his answers.
- FridgeSeal 2y ago
  No you don’t understand, AI is “dangerous” and only him and his uber rich billionaire mates should get to control it!
joe_the_user 2y ago
The humor/hypocrisy of the situation aside, it does seem to be true that OpenAI is consistently the one coming up with new ideas first (GPT 4, o1, 4o-style multimodality, voice chat, DALL-E, …) and then other companies reproduce their work, and get more credit because they actually publish the research I claim one just can't put the humor/hypocrisy aside that easily. What OpenAI did with the release of ChatGPT is productize research that was open and ongoing with Deepmind and other leading at least as much. And everything after that was an extension of the basic approach - improved, expanded but ultimately the same sort of beast. One might even say the situation of OpenAI to DeepMind was like Apple to Xerox. Productizing is nothing to sneeze at - it requires creativity and work to productize basic research. But naturally get end-users who consider the productizers the "fountain heads", who overestimate the productizers because products are all they see.
- Davidzheng 2y ago
  They RLHF'd first no?
mistercheph 2y ago
Not really, they just put their eye to where everyone knows the ball is going and publish fake / cherrypicked results and then pretend like they got there first (o1, gpt voice, sora)
Hatchback7599 2y ago
Reminds me of the Bill Gates quote when Steve Jobs accused him of stealing the ideas of Windows from Mac: Well, Steve... I think it’s more like we both had this rich neighbor named Xerox and I broke into his house to steal the TV set and found out that you had already stolen it. Xerox could be seen as Google, whose researchers produced the landmark Attention Is All You Need paper, and the general public, who provided all of the training data to make these models possible.
namuol 2y ago
The eye-watering funding numbers proposed by Altman in the past and more recently with “Stargate” suggests a publicly-funded research pivot is not out of the question. Could see a big defense department grant being given. Sigh.
- WhyNotHugo 2y ago
  I don't see any reason to assume that "publicly funded" will imply that the research is public. Although I'd be more than happy to be wrong on this one.
rndphs 2y ago
> OpenAI is consistently the one coming up with new ideas first (GPT 4, o1, 4o-style multimodality, voice chat, DALL-E, …) As far as I can tell o1 was based on Q-star, which could likely be Quiet-STaR, a CoT RL technique developed at Stanford that OpenAI may have learned about before it got published. Presumably that's why they never used the Q-Star name even though it had garnered mystique and would have been good for building hype. This is just speculation, but since OpenAI haven't published their technique then we can't know if it really was their innovation.
sota_pop 2y ago
I may be wrong, but to my knowledge OpenAI: - did not invent transformer architecture - did not invent diffusion architecture - did not come up with the idea for multi-modality - did not invent the notion/architecture of the latest “agentic” models They simply were the first to aggressively pursue scaling the transformer to the extent that is normal for the industry today. Although this has proven to produce interesting results, “simply adding scale” is, in my view, the least interesting development in modern ML. Giving credit where it’s due, they MAY have popularized the RLHF methodology, but I don’t recall them inventing that either? (feel free to point out any of the above that I falsely attributed to NOT OpenAI.) Additionally I seem to remember in an interview with Altman circa late ‘21 where he explains that the spirit of “OpenAI” and how their only goal is pursuing AGI, and “should someone else come up with a more promising path to get there, we would stop what we’re doing and help them”. I couldn’t find a reference to this interview, but anyone else, please feel free to share (I think it was a youtube link). - fast forward to 2025 and now “OpenAI” is the least open large contributor and indiscernible from your run-of-the-mill AI/ML valley startup insofar as they’re referring to others as “competitors” as opposed to collaborators.. interesting times…
- crackalamoo 2y ago
  As far as I can tell, OpenAI really did invent RLHF in its current form. https://arxiv.org/pdf/1909.08593 https://arxiv.org/pdf/1909.08593

by thorum 2y ago

spencerflem 2y ago

Fortunately, OpenAI doesn't need to make money because they are a nonprofit dedicated to the safe and transparent advancement of AI for all of humanity

mjburgess 2y ago
...somewhere a yacht salesman cried out in terror

turtlesdown11 2y ago

> other companies reproduce their work, and get more credit because they actually publish the research. I don't understand, you mean OpenAI isn't releasing open models and openly publishing their research?

Tostino 2y ago
Are you being sarcastic (honestly, it's hard to tell after reading as many uninformed takes in the past week as I have). No, they aren't (other than whisper). Their "papers" are closer to marketing materials. Very intentionally leaving out tons of technical information.
- KolmogorovComp 2y ago
  They are being sarcastic.
- sota_pop 2y ago
  /s

actuallyalys 2y ago

There’s some truth in that, but isn’t making a radically cheaper version also a new idea that deepseek didn’t know whether it would work? I mean, there was already research into distillation, but there was already research into some of (most of?) OpenAI’s ideas.

janalsncm 2y ago
Yes, for people who look into the research Deepseek released, there are a good number of novelties which enabled much cheaper R&D. For example, improvements to Mixture of Experts modules and Multi-head Latent Attention. If you have infinite money, you don’t need to innovate there, but DeepSeek didn’t.

weego 2y ago

Boy who stole test papers complains about child copying his answers.

FridgeSeal 2y ago
No you don’t understand, AI is “dangerous” and only him and his uber rich billionaire mates should get to control it!

joe_the_user 2y ago

The humor/hypocrisy of the situation aside, it does seem to be true that OpenAI is consistently the one coming up with new ideas first (GPT 4, o1, 4o-style multimodality, voice chat, DALL-E, …) and then other companies reproduce their work, and get more credit because they actually publish the research I claim one just can't put the humor/hypocrisy aside that easily. What OpenAI did with the release of ChatGPT is productize research that was open and ongoing with Deepmind and other leading at least as much. And everything after that was an extension of the basic approach - improved, expanded but ultimately the same sort of beast. One might even say the situation of OpenAI to DeepMind was like Apple to Xerox. Productizing is nothing to sneeze at - it requires creativity and work to productize basic research. But naturally get end-users who consider the productizers the "fountain heads", who overestimate the productizers because products are all they see.

Davidzheng 2y ago
They RLHF'd first no?

mistercheph 2y ago

Not really, they just put their eye to where everyone knows the ball is going and publish fake / cherrypicked results and then pretend like they got there first (o1, gpt voice, sora)

Hatchback7599 2y ago

Reminds me of the Bill Gates quote when Steve Jobs accused him of stealing the ideas of Windows from Mac: Well, Steve... I think it’s more like we both had this rich neighbor named Xerox and I broke into his house to steal the TV set and found out that you had already stolen it. Xerox could be seen as Google, whose researchers produced the landmark Attention Is All You Need paper, and the general public, who provided all of the training data to make these models possible.

namuol 2y ago

The eye-watering funding numbers proposed by Altman in the past and more recently with “Stargate” suggests a publicly-funded research pivot is not out of the question. Could see a big defense department grant being given. Sigh.

WhyNotHugo 2y ago
I don't see any reason to assume that "publicly funded" will imply that the research is public. Although I'd be more than happy to be wrong on this one.

rndphs 2y ago

> OpenAI is consistently the one coming up with new ideas first (GPT 4, o1, 4o-style multimodality, voice chat, DALL-E, …) As far as I can tell o1 was based on Q-star, which could likely be Quiet-STaR, a CoT RL technique developed at Stanford that OpenAI may have learned about before it got published. Presumably that's why they never used the Q-Star name even though it had garnered mystique and would have been good for building hype. This is just speculation, but since OpenAI haven't published their technique then we can't know if it really was their innovation.

sota_pop 2y ago

I may be wrong, but to my knowledge OpenAI: - did not invent transformer architecture - did not invent diffusion architecture - did not come up with the idea for multi-modality - did not invent the notion/architecture of the latest “agentic” models They simply were the first to aggressively pursue scaling the transformer to the extent that is normal for the industry today. Although this has proven to produce interesting results, “simply adding scale” is, in my view, the least interesting development in modern ML. Giving credit where it’s due, they MAY have popularized the RLHF methodology, but I don’t recall them inventing that either? (feel free to point out any of the above that I falsely attributed to NOT OpenAI.) Additionally I seem to remember in an interview with Altman circa late ‘21 where he explains that the spirit of “OpenAI” and how their only goal is pursuing AGI, and “should someone else come up with a more promising path to get there, we would stop what we’re doing and help them”. I couldn’t find a reference to this interview, but anyone else, please feel free to share (I think it was a youtube link). - fast forward to 2025 and now “OpenAI” is the least open large contributor and indiscernible from your run-of-the-mill AI/ML valley startup insofar as they’re referring to others as “competitors” as opposed to collaborators.. interesting times…

crackalamoo 2y ago
As far as I can tell, OpenAI really did invent RLHF in its current form. https://arxiv.org/pdf/1909.08593 https://arxiv.org/pdf/1909.08593