4 ms·

All these research science bureaucrats at Big Tech could have released LLM models or tried to develop what OpenAI did. But none of them did. We should applaud O

All these research science bureaucrats at Big Tech could have released LLM models or tried to develop what OpenAI did. But none of them did. We should applaud OpenAI for the innovation and let them do as they please.

deleted 4y ago
[deleted]
sacrosancty 4y ago
[dead]
redox99 4y ago
Google (and others) may not have released model weights, but they've published papers, which is ultimately what makes the field advance. OpenAI not only did not publish any GPT4 paper, they haven't even said how many parameters it has.
- buildbot 4y ago
  Then what is this? 99 pages of bullshit? https://arxiv.org/pdf/2303.08774.pdf https://arxiv.org/pdf/2303.08774.pdf
  - redox99 4y ago
    > Given both the competitive landscape and the safety implications of large-scale models like GPT-4, this report contains no further details about the architecture (including model size), hardware, training compute, dataset construction, training method, or similar. It's 99 pages of marketing material
    - buildbot 4y ago
      Personally I disagree, there are lot of interesting tidbits in this paper. More than marketing would need at least.
      - whatshisface 4y ago
        What good bits did you find? (I'm not sure how fruitful the "OpenAI is a Microsoft department" debate is given that they are almost one and everybody knows it, but I am curious if anyone has found anything good in those many pages.)
        buildbot 4y ago
        I think the most interesting thing is the their ability to predict performance from loss and on a wide range of tasks using a much smaller model - this lets them fine tune their architecture and hypers, then run a single large training run to get full scale gpt4 - from the paper it sounds like they only trained the large model once, then did a Reinforcement learning with human feedback finetune. Disclaimer - I work at Microsoft, in AI, and have no internal knowledge about gpt4.
        tempusalaria 4y ago
        This isn’t that interesting imo. This is the basic outcome of the scaling laws from Kaplan, Chinchilla papers pushed to a larger final model delta. They likely did extensive small model building on the gpt-4 architecture to establish hyperparameter scaling laws and then did a predicted build in exactly the same way chinchilla did.
        buildbot 4y ago
        I guess, but its actually not simple to do that, in my experience. There’s another paper on that: https://arxiv.org/abs/2203.03466 https://arxiv.org/abs/2203.03466 Why isn’t chinchilla running google AI chat or whatever then?
- thatsadude 4y ago
  Google published papers but has anybody be able to replicate their results?
  - redox99 4y ago
    Yes? Attention is all you need and Alpha Zero are the first that come to mind, but there are thousands.
- mach1ne 4y ago
  Indeed Google came up with Transformers and decided to gift the model to humanity. By broad strokes it was luck that OpenAI chose the seemingly right path of AI. Closest competitor DeepMind played games, which is intuitively closer to what humans do, but its relevance given aspects of deep learning is questionable.
  - teruakohatu 4y ago
    > DeepMind played games, which is intuitively closer to what humans do, but its relevance given aspects of deep learning is questionable. Reinforcement Learning is part of what OpenAI is doing. I don't think Google went down the wrong path. If anything they should have run down the path they were on.

deleted 4y ago

[deleted]

sacrosancty 4y ago

[dead]

redox99 4y ago

Google (and others) may not have released model weights, but they've published papers, which is ultimately what makes the field advance. OpenAI not only did not publish any GPT4 paper, they haven't even said how many parameters it has.

buildbot 4y ago
Then what is this? 99 pages of bullshit? https://arxiv.org/pdf/2303.08774.pdf https://arxiv.org/pdf/2303.08774.pdf
- redox99 4y ago
  > Given both the competitive landscape and the safety implications of large-scale models like GPT-4, this report contains no further details about the architecture (including model size), hardware, training compute, dataset construction, training method, or similar. It's 99 pages of marketing material
  - buildbot 4y ago
    Personally I disagree, there are lot of interesting tidbits in this paper. More than marketing would need at least.
    - whatshisface 4y ago
      What good bits did you find? (I'm not sure how fruitful the "OpenAI is a Microsoft department" debate is given that they are almost one and everybody knows it, but I am curious if anyone has found anything good in those many pages.)
      - buildbot 4y ago
        I think the most interesting thing is the their ability to predict performance from loss and on a wide range of tasks using a much smaller model - this lets them fine tune their architecture and hypers, then run a single large training run to get full scale gpt4 - from the paper it sounds like they only trained the large model once, then did a Reinforcement learning with human feedback finetune. Disclaimer - I work at Microsoft, in AI, and have no internal knowledge about gpt4.
        tempusalaria 4y ago
        This isn’t that interesting imo. This is the basic outcome of the scaling laws from Kaplan, Chinchilla papers pushed to a larger final model delta. They likely did extensive small model building on the gpt-4 architecture to establish hyperparameter scaling laws and then did a predicted build in exactly the same way chinchilla did.
        buildbot 4y ago
        I guess, but its actually not simple to do that, in my experience. There’s another paper on that: https://arxiv.org/abs/2203.03466 https://arxiv.org/abs/2203.03466 Why isn’t chinchilla running google AI chat or whatever then?
thatsadude 4y ago
Google published papers but has anybody be able to replicate their results?
- redox99 4y ago
  Yes? Attention is all you need and Alpha Zero are the first that come to mind, but there are thousands.
mach1ne 4y ago
Indeed Google came up with Transformers and decided to gift the model to humanity. By broad strokes it was luck that OpenAI chose the seemingly right path of AI. Closest competitor DeepMind played games, which is intuitively closer to what humans do, but its relevance given aspects of deep learning is questionable.
- teruakohatu 4y ago
  > DeepMind played games, which is intuitively closer to what humans do, but its relevance given aspects of deep learning is questionable. Reinforcement Learning is part of what OpenAI is doing. I don't think Google went down the wrong path. If anything they should have run down the path they were on.