3 ms·

I get that he didn’t work on these things day to day, but ultimately who was responsible for the innovations that made these programs possible in the first plac

I get that he didn’t work on these things day to day, but ultimately who was responsible for the innovations that made these programs possible in the first place? For example, was anyone at OpenAI involved in the creation of transformers?

sashank_1509 3y ago
Even Ilya was not involved in the creation of transformers… Honestly by your logic, NVIDIA are the real innovators, without their hardware none of this would be possible (including the very first Deep Learning Win, known as AlexNet)
- kmlevitt 3y ago
  I'm not proposing any logic. I'm just trying to figure out who if anyone at open AI actually pioneered the development of this technology, rather than simply worked toward its implementation at scale. The reason it matters is because if they are just building all this stuff based on ideas taken from other outside researchers' papers and throwing money toward making it happen, OpenAI's moat is smaller, and it's easier for a bright group of competitors with lots of funding to catch up.
  - sashank_1509 3y ago
    OpenAI has not invented much of any new ideas. None of them worked on transformers, or diffusion or a host of other technologies that power their models. (though one of the leads at OpenAI built PPO while in his PhD). But they’ve perfected the art of training and deploying models, which I think is their moat. It still shocks me that their speech recognition model is miles ahead of Siri (that struggles heavily with my accent) and even Alexa. How is it that 2 companies that have invested billions of dollars into these products, end up with worse results than a small team in startup whose main focus isn’t speech recognition? Almost like how Toyota didn’t invent the car, but perfected the art of building reliable cars, I see OpenAI as perfecting the art of the building ML models that just work in the real world
    - kmlevitt 3y ago
      OK thanks, that makes sense. Who at Open AI would you say was most responsible for perfecting that training?
      - sashank_1509 3y ago
        I’m not sure, it’s hard to say unless you’re in OpenAI. Also it’s almost likely a team, not 1 person

sashank_1509 3y ago

Even Ilya was not involved in the creation of transformers… Honestly by your logic, NVIDIA are the real innovators, without their hardware none of this would be possible (including the very first Deep Learning Win, known as AlexNet)

kmlevitt 3y ago
I'm not proposing any logic. I'm just trying to figure out who if anyone at open AI actually pioneered the development of this technology, rather than simply worked toward its implementation at scale. The reason it matters is because if they are just building all this stuff based on ideas taken from other outside researchers' papers and throwing money toward making it happen, OpenAI's moat is smaller, and it's easier for a bright group of competitors with lots of funding to catch up.
- sashank_1509 3y ago
  OpenAI has not invented much of any new ideas. None of them worked on transformers, or diffusion or a host of other technologies that power their models. (though one of the leads at OpenAI built PPO while in his PhD). But they’ve perfected the art of training and deploying models, which I think is their moat. It still shocks me that their speech recognition model is miles ahead of Siri (that struggles heavily with my accent) and even Alexa. How is it that 2 companies that have invested billions of dollars into these products, end up with worse results than a small team in startup whose main focus isn’t speech recognition? Almost like how Toyota didn’t invent the car, but perfected the art of building reliable cars, I see OpenAI as perfecting the art of the building ML models that just work in the real world
  - kmlevitt 3y ago
    OK thanks, that makes sense. Who at Open AI would you say was most responsible for perfecting that training?
    - sashank_1509 3y ago
      I’m not sure, it’s hard to say unless you’re in OpenAI. Also it’s almost likely a team, not 1 person