25 ms·

Goldman on Generative AI: doesn't justify costs or solve complex problems [pdf]

93 points by edzitron 2y ago

deleted 2y ago
[deleted]
nabla9 2y ago
Great report.
great_psy 2y ago
Is there more than the cover to this ? On mobile I only see one page of the PDF.
- dekhn 2y ago
  Yes, here's the original URL which should download a full multi-page PDF: https://www.goldmansachs.com/intelligence/pages/gs-research/gen-ai-too-much-spend-too-little-benefit/report.pdf https://www.goldmansachs.com/intelligence/pages/gs-research/... It's mostly speculative narrative with a fair number of data-driven charts. I wouldn't spend much time on it unless you like financial analysis with hand-waving.
- geor9e 2y ago
  The PDF has 31 pages.
dekhn 2y ago
Except for a short window around the release of GPT-4 (especially the inflated claims around beating expert trained humans at legal and math tests, as well as "replacing google"), I think people have more or less right-sized their expectations for large language models and generative AI. Clearly it can do interesting and impressive things but it's not superintelligence, and the folks predicting we're just around the corner have been recognized once again as shysters, hucksters, and charlatans. It doesn't help that state of the art ML researches have gotten so good at over-hyping the actual abilities of their technology. However, I do think we'll continue to see impressive advances in the areas of media consumption and production, with complex reasoning on hard problems being a likely area of improvement in the near (1 decade) future. While I once never expected to see something like HAL in my lifetime, I feel that many aspects of HAL (voice recognition, ship automation, and chess-playing) have been achieved, if not fully integrated into a single agent. We can expect most applications to be banal- the giants who have the largest data piles will train models that continue to optimize the addictivity of social media, and click-thru rates of ads. I am also quite impressed at the recall of information by language models for highly factual and well-supported things (computer reviews in particular).
- karaterobot 2y ago
  I agree with what you say above, but my perception is that most people still view the current crop of models as a step or two away from superintelligence. That superintelligence, or AGI, is a matter of continued improvement along the current lines, rather than along entirely different lines.
  - torginus 2y ago
    I like to think of the 'car factory' analogy - it's populated by robots that are in some respects far superior to humans, and are doing 90% of the labor. Some ancient futurist, not having seen one before, could correctly predict that 9 out of 10 jobs will be done by robots, and arrive at the incorrect conclusion that robots have rendered humans obsolete. In actuality, humans are still needed for the 10% the robots can't do well, or serve to enhance the productivity of humans. I predict AI is like this and going to be for a while - it can clearly do some stuff well and sometimes better than humans, but humans will have their niches for a while.
    - dekhn 2y ago
      I call this the "filter changing problem". No matter how complex you make the technology, somebody still has to change the oil filter (or do whatever other maintainence is required to keep the system running). Sort of like ML-SRE, for those who are familiar with the concept.
      - sseagull 2y ago
        This is related to Moravec’s Paradox: https://en.wikipedia.org/wiki/Moravec%27s_paradox https://en.wikipedia.org/wiki/Moravec%27s_paradox “it is comparatively easy to make computers exhibit adult level performance on intelligence tests or playing checkers, and difficult or impossible to give them the skills of a one-year-old when it comes to perception and mobility”
  - deleted 2y ago
    [deleted]
xmichael909 2y ago
This interview by Adam Conover, really is a wonderful discussion on the topic https://www.youtube.com/watch?v=T8ByoAt5gCA https://www.youtube.com/watch?v=T8ByoAt5gCA I was pretty amazed with GPT when it came up, but increasingly find it makes to many mistakes. I full use it as a tool in writing code, but it needs to be treated as Intellisense plus, or something to that affect, not something that will handle complex tasks. GPT and Claude make many mistakes and unless they can solve it from completely making up stuff (which I don't think they can) will not advance much more beyond waht they currently are.
- pixl97 2y ago
  I take this view, a correct one as "Thank goodness". I don't think humanity is ready for a 'correct intelligence' yet, especially one that if existed at the level of human intelligence would likely rapidly go into the realm of superintelligence. Even if it didn't get out of human control, the humans that controlled said AI would gain an immense amount of power which would present a great destabilization risk.
- kaast202 2y ago
  they seem pissed off at capitalism and big tech. Thats all that interview was .
zzzbra 2y ago
Heartbreaking: The Worst People You Know Just Made A Great Point
- echelon 2y ago
  The music, film, and game industries are about to be completely disrupted. LLMs and AGI might be hogwash, but processing multimedia is where Gen AI and especially diffusion models shine. Furthermore text-to-{whatever} models might produce slop, but Gen AI "exoskeletons" (spatial domain, temporal domain editors) are Photoshop and Blender from next century. These turbocharge creatives. Hearing and vision are simple operations relative to reasoning. They're naturally occurring physical signals that the animal kingdom has evolved, on several different occasions, to process. This is likely why they're such a low hanging fruit to replicate with Gen AI.
  - deleted 2y ago
    [deleted]
  - beachandbytes 2y ago
    Id agree with you on the creative industry, but disagree that that the generative AI's aren't going to do the same to just about every other industry. We were at a point where we had extremely specialized models that were useful, and now we have general models that are EXTREMELY useful in almost all contexts. Text, Audio, Video, Data Processing, etc. At least in my eyes we are at the same point with LLMs as we were with computing when you had a large part of the population that was just "not into them". As if it was like choosing any other hobby. I'm sure tons of people aren't getting much utility out of the space now, but it's not because the utility isn't there.
  - mistrial9 2y ago
    ordinary surveillance applications with some fine or billing attached.. pure marketing where public facing materials have to be consistent but not much more than that.. and famously, anything in journalism from video creation to writing to narration.. are all also ground central in a "vocation crisis" too
weweweoo 2y ago
Generative AI appears fantastic aid for many smaller tasks where there's enough training data, and correctness of the answer is subjective (like art), or easily verifiable by a human in the loop (small snippets of code, checking that summary of an article matches the contents of the original). Generally it helps with the tedious parts, but not with the hard parts of my job. I don't have much belief in fully autonomous generative AI agents performing more complex tasks any time soon. It's a significant productivity boost for some jobs, but not a total replacement for humans who do more than read from a script, or write clickbait articles for media.
- harrisoned 2y ago
  I agree with that. At work, we are about to implement a decent LLM and ditch Dialogflow for our chatbot. But not to talk directly to the client (it's asking for a disaster), just to recognize intentions, pretty much like Dialogflow but better. Right now there are many small but decent models available for free, and cheap to use. If it wasn't for the hype, it would never have reached that level of optimization. Now we can make decent home assistants, text parsers and a bunch of other stuff you already mentioned. But someone paid for that. The companies who believed this would be revolutionary will eventually have a really hard reality check. Not that they won't try and use it for critical stuff, but once they do and it fails spectacularly they will realize a lot of money went down the drain.
  - elforce002 2y ago
    And we'll thank them for their service.
- alecco 2y ago
  > where there's enough training data The newer models are 10x faster and cheaper, therefore synthetic data is 10x cheaper to make now. If the ARC challenge makes an impact, there's a good chance the next generation AI will need a lot less data.
ChrisArchitect 2y ago
[dupe] Please don't post wayback links unnecessarily. Content still fresh and available. Discussion here: https://news.ycombinator.com/item?id=40856329 https://news.ycombinator.com/item?id=40856329
anu7df 2y ago
The only question I have is whether Goldman is shorting NVIDIA..
- random3 2y ago
  Ha! +1 Although, I'd be shorting 80% of everyone else spending money with NVidia without a clear path to recover. However, given that most are (likely?) not listed, there isn't that much to short?
deleted 2y ago
[deleted]
russellbeattie 2y ago
Pretty sure I read that Goldman itself is currently creating its own internal models using its proprietary data to help its analysts, IT and investors.
- rsynnott 2y ago
  The likes of Goldman have been doing ML stuff (which is generally marketed as ‘AI’ as of a few years ago) for decades, but it’s generally not generative AI.
cpursley 2y ago
Ironically, AI still sucks at accurately parsing PDFs.
bluelightning2k 2y ago
There is a paradox. To build the future requires irrational belief. And to sell that vision. Perhaps the difference between insanity and visionary, "scam" and genius is simply the outcome. When someone like Sam Altman declares optimistically that we will get AGI and talks about what kind of society we will need to build... It's kind of hard to tell what mix of those 4 is at work. But certainly it will be perceived differently based upon the outcome not the sincerity of the effort.
- crabmusket 2y ago
  > There is a paradox. To build the future requires irrational belief. I'm not convinced this is true. What is irrational about the possibility of e.g. scientific progress, inventing new products, or creating a viable business? Irrational belief may be one way to motivate yourself to try those things. But it's not the only way. Calculated risk-taking isn't irrational, is it?
mgh2 2y ago
Original: https://www.goldmansachs.com/intelligence/pages/gs-research/gen-ai-too-much-spend-too-little-benefit/report.pdf https://www.goldmansachs.com/intelligence/pages/gs-research/...

93 points by edzitron 2y ago

deleted 2y ago

[deleted]

nabla9 2y ago

Great report.

great_psy 2y ago

Is there more than the cover to this ? On mobile I only see one page of the PDF.

dekhn 2y ago
Yes, here's the original URL which should download a full multi-page PDF: https://www.goldmansachs.com/intelligence/pages/gs-research/gen-ai-too-much-spend-too-little-benefit/report.pdf https://www.goldmansachs.com/intelligence/pages/gs-research/... It's mostly speculative narrative with a fair number of data-driven charts. I wouldn't spend much time on it unless you like financial analysis with hand-waving.
geor9e 2y ago
The PDF has 31 pages.

dekhn 2y ago

Except for a short window around the release of GPT-4 (especially the inflated claims around beating expert trained humans at legal and math tests, as well as "replacing google"), I think people have more or less right-sized their expectations for large language models and generative AI. Clearly it can do interesting and impressive things but it's not superintelligence, and the folks predicting we're just around the corner have been recognized once again as shysters, hucksters, and charlatans. It doesn't help that state of the art ML researches have gotten so good at over-hyping the actual abilities of their technology. However, I do think we'll continue to see impressive advances in the areas of media consumption and production, with complex reasoning on hard problems being a likely area of improvement in the near (1 decade) future. While I once never expected to see something like HAL in my lifetime, I feel that many aspects of HAL (voice recognition, ship automation, and chess-playing) have been achieved, if not fully integrated into a single agent. We can expect most applications to be banal- the giants who have the largest data piles will train models that continue to optimize the addictivity of social media, and click-thru rates of ads. I am also quite impressed at the recall of information by language models for highly factual and well-supported things (computer reviews in particular).

karaterobot 2y ago
I agree with what you say above, but my perception is that most people still view the current crop of models as a step or two away from superintelligence. That superintelligence, or AGI, is a matter of continued improvement along the current lines, rather than along entirely different lines.
- torginus 2y ago
  I like to think of the 'car factory' analogy - it's populated by robots that are in some respects far superior to humans, and are doing 90% of the labor. Some ancient futurist, not having seen one before, could correctly predict that 9 out of 10 jobs will be done by robots, and arrive at the incorrect conclusion that robots have rendered humans obsolete. In actuality, humans are still needed for the 10% the robots can't do well, or serve to enhance the productivity of humans. I predict AI is like this and going to be for a while - it can clearly do some stuff well and sometimes better than humans, but humans will have their niches for a while.
  - dekhn 2y ago
    I call this the "filter changing problem". No matter how complex you make the technology, somebody still has to change the oil filter (or do whatever other maintainence is required to keep the system running). Sort of like ML-SRE, for those who are familiar with the concept.
    - sseagull 2y ago
      This is related to Moravec’s Paradox: https://en.wikipedia.org/wiki/Moravec%27s_paradox https://en.wikipedia.org/wiki/Moravec%27s_paradox “it is comparatively easy to make computers exhibit adult level performance on intelligence tests or playing checkers, and difficult or impossible to give them the skills of a one-year-old when it comes to perception and mobility”
- deleted 2y ago
  [deleted]

xmichael909 2y ago

This interview by Adam Conover, really is a wonderful discussion on the topic https://www.youtube.com/watch?v=T8ByoAt5gCA https://www.youtube.com/watch?v=T8ByoAt5gCA I was pretty amazed with GPT when it came up, but increasingly find it makes to many mistakes. I full use it as a tool in writing code, but it needs to be treated as Intellisense plus, or something to that affect, not something that will handle complex tasks. GPT and Claude make many mistakes and unless they can solve it from completely making up stuff (which I don't think they can) will not advance much more beyond waht they currently are.

pixl97 2y ago
I take this view, a correct one as "Thank goodness". I don't think humanity is ready for a 'correct intelligence' yet, especially one that if existed at the level of human intelligence would likely rapidly go into the realm of superintelligence. Even if it didn't get out of human control, the humans that controlled said AI would gain an immense amount of power which would present a great destabilization risk.
kaast202 2y ago
they seem pissed off at capitalism and big tech. Thats all that interview was .

zzzbra 2y ago

Heartbreaking: The Worst People You Know Just Made A Great Point

echelon 2y ago
The music, film, and game industries are about to be completely disrupted. LLMs and AGI might be hogwash, but processing multimedia is where Gen AI and especially diffusion models shine. Furthermore text-to-{whatever} models might produce slop, but Gen AI "exoskeletons" (spatial domain, temporal domain editors) are Photoshop and Blender from next century. These turbocharge creatives. Hearing and vision are simple operations relative to reasoning. They're naturally occurring physical signals that the animal kingdom has evolved, on several different occasions, to process. This is likely why they're such a low hanging fruit to replicate with Gen AI.
- deleted 2y ago
  [deleted]
- beachandbytes 2y ago
  Id agree with you on the creative industry, but disagree that that the generative AI's aren't going to do the same to just about every other industry. We were at a point where we had extremely specialized models that were useful, and now we have general models that are EXTREMELY useful in almost all contexts. Text, Audio, Video, Data Processing, etc. At least in my eyes we are at the same point with LLMs as we were with computing when you had a large part of the population that was just "not into them". As if it was like choosing any other hobby. I'm sure tons of people aren't getting much utility out of the space now, but it's not because the utility isn't there.
- mistrial9 2y ago
  ordinary surveillance applications with some fine or billing attached.. pure marketing where public facing materials have to be consistent but not much more than that.. and famously, anything in journalism from video creation to writing to narration.. are all also ground central in a "vocation crisis" too

weweweoo 2y ago

Generative AI appears fantastic aid for many smaller tasks where there's enough training data, and correctness of the answer is subjective (like art), or easily verifiable by a human in the loop (small snippets of code, checking that summary of an article matches the contents of the original). Generally it helps with the tedious parts, but not with the hard parts of my job. I don't have much belief in fully autonomous generative AI agents performing more complex tasks any time soon. It's a significant productivity boost for some jobs, but not a total replacement for humans who do more than read from a script, or write clickbait articles for media.

harrisoned 2y ago
I agree with that. At work, we are about to implement a decent LLM and ditch Dialogflow for our chatbot. But not to talk directly to the client (it's asking for a disaster), just to recognize intentions, pretty much like Dialogflow but better. Right now there are many small but decent models available for free, and cheap to use. If it wasn't for the hype, it would never have reached that level of optimization. Now we can make decent home assistants, text parsers and a bunch of other stuff you already mentioned. But someone paid for that. The companies who believed this would be revolutionary will eventually have a really hard reality check. Not that they won't try and use it for critical stuff, but once they do and it fails spectacularly they will realize a lot of money went down the drain.
- elforce002 2y ago
  And we'll thank them for their service.
alecco 2y ago
> where there's enough training data The newer models are 10x faster and cheaper, therefore synthetic data is 10x cheaper to make now. If the ARC challenge makes an impact, there's a good chance the next generation AI will need a lot less data.

ChrisArchitect 2y ago

[dupe] Please don't post wayback links unnecessarily. Content still fresh and available. Discussion here: https://news.ycombinator.com/item?id=40856329 https://news.ycombinator.com/item?id=40856329

anu7df 2y ago

The only question I have is whether Goldman is shorting NVIDIA..

random3 2y ago
Ha! +1 Although, I'd be shorting 80% of everyone else spending money with NVidia without a clear path to recover. However, given that most are (likely?) not listed, there isn't that much to short?

deleted 2y ago

[deleted]

russellbeattie 2y ago

Pretty sure I read that Goldman itself is currently creating its own internal models using its proprietary data to help its analysts, IT and investors.

rsynnott 2y ago
The likes of Goldman have been doing ML stuff (which is generally marketed as ‘AI’ as of a few years ago) for decades, but it’s generally not generative AI.

cpursley 2y ago

Ironically, AI still sucks at accurately parsing PDFs.

bluelightning2k 2y ago

There is a paradox. To build the future requires irrational belief. And to sell that vision. Perhaps the difference between insanity and visionary, "scam" and genius is simply the outcome. When someone like Sam Altman declares optimistically that we will get AGI and talks about what kind of society we will need to build... It's kind of hard to tell what mix of those 4 is at work. But certainly it will be perceived differently based upon the outcome not the sincerity of the effort.

crabmusket 2y ago
> There is a paradox. To build the future requires irrational belief. I'm not convinced this is true. What is irrational about the possibility of e.g. scientific progress, inventing new products, or creating a viable business? Irrational belief may be one way to motivate yourself to try those things. But it's not the only way. Calculated risk-taking isn't irrational, is it?

mgh2 2y ago

Original: https://www.goldmansachs.com/intelligence/pages/gs-research/gen-ai-too-much-spend-too-little-benefit/report.pdf https://www.goldmansachs.com/intelligence/pages/gs-research/...