7 ms·

There has actually been research that found that there are strong diminishing returns in terms of at least expanding parameter sizes. While I think there are st

by jug 3y ago

There has actually been research that found that there are strong diminishing returns in terms of at least expanding parameter sizes. While I think there are still breakthroughs to be made in terms of window sizes and workarounds like Mixture of Experts, I'm not sure how much farther we will get here in the long term in terms of raw performance of the LLM itself. FWIW, Sam Altman agrees and has a surprisingly similar quote as found here "I think we’re at the end of the era where it’s gonna be these giant models, and we’ll make them better in other ways": https://techcrunch.com/2023/04/14/sam-altman-size-of-llms-wont-matter-as-much-moving-forward https://techcrunch.com/2023/04/14/sam-altman-size-of-llms-wo...

brandall10 3y ago
GPT-4 is already based on a MOE approach. Just noting as this was placed in the "breakthroughs to be made".
FrustratedMonky 3y ago
How can research predict this wave is coming to an end, when research also didn't think this wave would happen either. It seems like there are always people saying 'it can't be done'. Then it happens. If there was a way to predict the future, then wouldn't that research need to know how something would be implemented, in order to know it can't be?
- StackOverlord 3y ago
  For the first time with GPT4, OpenAI as been able to predict model progress with accuracy: > A large focus of the GPT-4 project has been building a deep learning stack that scales predictably. The primary reason is that, for very large training runs like GPT-4, it is not feasible to do extensive model-specific tuning. We developed infrastructure and optimization that have very predictable behavior across multiple scales. To verify this scalability, we accurately predicted in advance GPT-4’s final loss on our internal codebase (not part of the training set) by extrapolating from models trained using the same methodology but using 10,000x less compute: > Now that we can accurately predict the metric we optimize during training (loss), we’re starting to develop methodology to predict more interpretable metrics. For example, we successfully predicted the pass rate on a subset of the HumanEval dataset, extrapolating from models with 1,000x less compute: > We believe that accurately predicting future machine learning capabilities is an important part of safety that doesn’t get nearly enough attention relative to its potential impact (though we’ve been encouraged by efforts across several institutions). We are scaling up our efforts to develop methods that provide society with better guidance about what to expect from future systems, and we hope this becomes a common goal in the field. Source: https://openai.com/research/gpt-4 https://openai.com/research/gpt-4
  - sidewndr46 3y ago
    Isn't this all based off self-attestation? There is no comprehensive audit of their research data and finances I am aware of. If I was OpenAI and blew millions of dollars training models that showed exponentially worse performance for incrementally more resources expended training the model, my next step would not be to publish about it.
  - FrustratedMonky 3y ago
    There was an entire Post/Discussion on HN a few weeks ago discussing whether or not this Post from OpenAI was for: "just trying to temper investor enthusiasm" "trying to downplay AI threats to calm down regulators" etc.... etc.... But it is not some 'Proof', that LLM's have reached a limit. It is a self reported note along the lines of : "nothing to see here, we're at our limit, it's all good, stop probing us".
- wredue 3y ago
  Blockchain was a complete bust, and middle managers need buzzwords to sell senior leadership. That’s my personal take on the current wave.
  - ActionHank 3y ago
    I've seen this as well as the dying down of the hype once the new cycle dropped off on AI hype.
  - earthboundkid 3y ago
    I am a mild LLM skeptic. But I find the response of "oh, it's all just post-crypto scamming" really weird. Crypto was a total scam. There was never a concrete, non-criminal (important caveat) application where crypto was easier than just using PayPal or whatever. LLMs are very imperfect and still have a lot of work to do, but they do actually do some job related tasks today. If I have JS snippet and I wish it were in Go or vice versa, I can ask an LLM and get a good translation. If I have a document, I can ask the LLM if there's something I should add or simplify. Are these world changing capabilities? No, not yet. But they do exist and they are real and they are new. So yes, of course investors are interested in a new technology whose future limits we don't know yet.
    - jedbrown 3y ago
      The LM industry valuation would be way smaller if they were not laundering behavior that would be illegal if a human did it. If "AI" were required to practice clean-room design (https://en.wikipedia.org/wiki/Clean_room_design https://en.wikipedia.org/wiki/Clean_room_design) to avoid infringing copyright, we would laugh at the ineptitude. If people believed the FTC-CFPB-DOJ-EEOC joint statement was going to lead to successful prosecutions, the industry valuation would collapse. https://www.ftc.gov/system/files/ftc_gov/pdf/EEOC-CRT-FTC-CFPB-AI-Joint-Statement%28final%29.pdf https://www.ftc.gov/system/files/ftc_gov/pdf/EEOC-CRT-FTC-CF...
      - unusualmonkey 3y ago
        Humans aren't required to use clean room design. Using copyrighted materials as inspiration/reference is not uncommon or illegal.
- coryfklein 3y ago
  "Research" is not some monolithic single concept. One might also ask, "How can research produce ChatGPT when for decades research failed to produce ChatGPT?"
  - FrustratedMonky 3y ago
    Exactly. So why now are we trusting 'Research' that is trying to predict the future of other 'Research'. The linked article is just some estimates on the error built into the current LLM model. How can we extrapolate that to be "well, gosh darn, these LLM's are already played out, guess we're all done"
gondolinion 3y ago
[dead]
stuckinhell 3y ago
The open source community has breakthrough after breakthrough lately. its absolutely stunning how fast it's advancing. I can run art models and llms on cpu/gpu now. I've tested out opensource models with quality better than chatgpt3.5turbo, and I can even fine tune them on my notes and books for better results. It's all so easy with so many one click installers now too! My husbands D&D group uses some AI for their games now (koboldcpp?). It's staggering how fast it's moving.
- jprete 3y ago
  > I can run art models and llms on cpu/gpu now. I've tested out opensource models with quality better than chatgpt3.5turbo, and I can even fine tune them on my notes and books for better results. It's all so easy with so many one click installers now too! Do you have a good resource to find this stuff? I’m behind the times. (Also, to be a pedant, most of that is inference and not training. But I can’t say much about fine tuning so I’m not really trying to argue against your point.)
  - stuckinhell 3y ago
    Honestly no good single resource, I spend 8 hours a week just taking notes from hackernews,discord, hackernoon, slack ai groups, reddit, github issues (probably the most useful) on various open source ai libraries. It's real work to keep up on everything.

by jug 3y ago

brandall10 3y ago

GPT-4 is already based on a MOE approach. Just noting as this was placed in the "breakthroughs to be made".

FrustratedMonky 3y ago

How can research predict this wave is coming to an end, when research also didn't think this wave would happen either. It seems like there are always people saying 'it can't be done'. Then it happens. If there was a way to predict the future, then wouldn't that research need to know how something would be implemented, in order to know it can't be?

StackOverlord 3y ago
For the first time with GPT4, OpenAI as been able to predict model progress with accuracy: > A large focus of the GPT-4 project has been building a deep learning stack that scales predictably. The primary reason is that, for very large training runs like GPT-4, it is not feasible to do extensive model-specific tuning. We developed infrastructure and optimization that have very predictable behavior across multiple scales. To verify this scalability, we accurately predicted in advance GPT-4’s final loss on our internal codebase (not part of the training set) by extrapolating from models trained using the same methodology but using 10,000x less compute: > Now that we can accurately predict the metric we optimize during training (loss), we’re starting to develop methodology to predict more interpretable metrics. For example, we successfully predicted the pass rate on a subset of the HumanEval dataset, extrapolating from models with 1,000x less compute: > We believe that accurately predicting future machine learning capabilities is an important part of safety that doesn’t get nearly enough attention relative to its potential impact (though we’ve been encouraged by efforts across several institutions). We are scaling up our efforts to develop methods that provide society with better guidance about what to expect from future systems, and we hope this becomes a common goal in the field. Source: https://openai.com/research/gpt-4 https://openai.com/research/gpt-4
- sidewndr46 3y ago
  Isn't this all based off self-attestation? There is no comprehensive audit of their research data and finances I am aware of. If I was OpenAI and blew millions of dollars training models that showed exponentially worse performance for incrementally more resources expended training the model, my next step would not be to publish about it.
- FrustratedMonky 3y ago
  There was an entire Post/Discussion on HN a few weeks ago discussing whether or not this Post from OpenAI was for: "just trying to temper investor enthusiasm" "trying to downplay AI threats to calm down regulators" etc.... etc.... But it is not some 'Proof', that LLM's have reached a limit. It is a self reported note along the lines of : "nothing to see here, we're at our limit, it's all good, stop probing us".
wredue 3y ago
Blockchain was a complete bust, and middle managers need buzzwords to sell senior leadership. That’s my personal take on the current wave.
- ActionHank 3y ago
  I've seen this as well as the dying down of the hype once the new cycle dropped off on AI hype.
- earthboundkid 3y ago
  I am a mild LLM skeptic. But I find the response of "oh, it's all just post-crypto scamming" really weird. Crypto was a total scam. There was never a concrete, non-criminal (important caveat) application where crypto was easier than just using PayPal or whatever. LLMs are very imperfect and still have a lot of work to do, but they do actually do some job related tasks today. If I have JS snippet and I wish it were in Go or vice versa, I can ask an LLM and get a good translation. If I have a document, I can ask the LLM if there's something I should add or simplify. Are these world changing capabilities? No, not yet. But they do exist and they are real and they are new. So yes, of course investors are interested in a new technology whose future limits we don't know yet.
  - jedbrown 3y ago
    The LM industry valuation would be way smaller if they were not laundering behavior that would be illegal if a human did it. If "AI" were required to practice clean-room design (https://en.wikipedia.org/wiki/Clean_room_design https://en.wikipedia.org/wiki/Clean_room_design) to avoid infringing copyright, we would laugh at the ineptitude. If people believed the FTC-CFPB-DOJ-EEOC joint statement was going to lead to successful prosecutions, the industry valuation would collapse. https://www.ftc.gov/system/files/ftc_gov/pdf/EEOC-CRT-FTC-CFPB-AI-Joint-Statement%28final%29.pdf https://www.ftc.gov/system/files/ftc_gov/pdf/EEOC-CRT-FTC-CF...
    - unusualmonkey 3y ago
      Humans aren't required to use clean room design. Using copyrighted materials as inspiration/reference is not uncommon or illegal.
coryfklein 3y ago
"Research" is not some monolithic single concept. One might also ask, "How can research produce ChatGPT when for decades research failed to produce ChatGPT?"
- FrustratedMonky 3y ago
  Exactly. So why now are we trusting 'Research' that is trying to predict the future of other 'Research'. The linked article is just some estimates on the error built into the current LLM model. How can we extrapolate that to be "well, gosh darn, these LLM's are already played out, guess we're all done"

gondolinion 3y ago

[dead]

stuckinhell 3y ago

The open source community has breakthrough after breakthrough lately. its absolutely stunning how fast it's advancing. I can run art models and llms on cpu/gpu now. I've tested out opensource models with quality better than chatgpt3.5turbo, and I can even fine tune them on my notes and books for better results. It's all so easy with so many one click installers now too! My husbands D&D group uses some AI for their games now (koboldcpp?). It's staggering how fast it's moving.

jprete 3y ago
> I can run art models and llms on cpu/gpu now. I've tested out opensource models with quality better than chatgpt3.5turbo, and I can even fine tune them on my notes and books for better results. It's all so easy with so many one click installers now too! Do you have a good resource to find this stuff? I’m behind the times. (Also, to be a pedant, most of that is inference and not training. But I can’t say much about fine tuning so I’m not really trying to argue against your point.)
- stuckinhell 3y ago
  Honestly no good single resource, I spend 8 hours a week just taking notes from hackernews,discord, hackernoon, slack ai groups, reddit, github issues (probably the most useful) on various open source ai libraries. It's real work to keep up on everything.