7 ms·

AI Is Smoke and Mirrors

71 points by cdme 2y ago

hiddencost 2y ago
Machine learning however works, and generates hundreds of billions of dollars a year. Both can be true!
- hulitu 2y ago
  Yes, but they are not related to each other.
  - whimsicalism 2y ago
    they are absolutely related to each other
furyofantares 2y ago
The usual pattern is to overestimate short-term change and underestimate long-term change. We're a year since this stuff really got popular, and so I think a thing that's happening right now is many people noticing the short-term changes were overestimated.
- Upvoter33 2y ago
  yes, exactly. There will be a lot of these types of articles in the next few years, some disappointments along the way, but deep and longlasting change in the tech sector and beyond is coming.
whimsicalism 2y ago
No matter how much the chattering class wants to portray this as a "fad" and not an inevitability, those of us who can type and read can see the reality for ourselves.
- mlyle 2y ago
  Yah-- what we have now is enough to be enormously disruptive as it is absorbed. The big question is how much better it's going to get. We may be plateauing in capability-- or at least slowing in capability growth-- or we may not be. But it doesn't need to get better for it to eat a lot of peoples' lunch.
  - lambdaba 2y ago
    My opinion shifts every other day it seems, recently, Roon, the "famous" anon X account that works at OpenAI, and presumably has access to some unreleased info, said, in response to "are we plateauing soon", "not even close". I honestly don't know what to think.
    - whimsicalism 2y ago
      I know his real identity and I doubt there is some key insider info there although he does work at OAI
    - mlyle 2y ago
      I feel like for LLM's at least, we've not seen a lot of progress on the high end for a year or so. GPT4-Turbo is a little better than GPT4, but mainly only for things that can benefit from huge contexts.
      - deleted 2y ago
        [deleted]
      - whimsicalism 2y ago
        We have seen progress, but mostly price progress.
        mlyle 2y ago
        Yes, and also not-always-bumping-into-capacity-limits (availability) progress.
simonbarker87 2y ago
Clearly AI (re-skinned ML) is following the standard Gartner hype cycle graph.
pantsforbirds 2y ago
I think the AI itself is working extremely well. The state-of-the-art LLMs are exceptional at many real-world tasks and are deployed in many production code bases successfully. The more recent AI failures (like the humane pin) are at least partially a hardware problem. These LLMs require huge amounts of compute to do nontrivial tasks, and a wearable just can't run locally with the current architectures.
- FrustratedMonky 2y ago
  Exactly. Yes, there will be failures along the way, but the AI tools we have now are still pretty amazing and really would have been called magical 5 years ago. And yes, the AI tech already available now is already causing disruption and will continue to disrupt. Anybody saying this is just a 'blip' or 'fad' are really not trying them out. I remember in the 90's someone telling me to not bother with Java (back with applets), because this 'Internet Thing' isn't going to pan out. That is how the anti-AI sentiment sounds to me. It's already amazing and continues to grow. There is not pulling it back now.
yawnxyz 2y ago
That's a lot of words from someone who doesn't seem to have tried using these tools yet? The Hype is smoke and mirrors, but there are tons of real use cases where things were MUCH easier than before. Things like text to speech or image recognition previously required a team of 10 ML engineers a couple of years to build now take a weekend. It's smoke and mirrors to the "business world" because none of these translate into bottom line numbers though, and I think that's all this guy is able to see
- hn_throwaway_99 2y ago
  I'm just kinda sad at how cynical so many of us have become. I mean, I see tons of comments "But it hallucinates!!!!!" Yeah, no shit. At the same time, you're having a perfectly grammatically correct, natural, sensical conversation with a computer. Just a couple years ago that was purely the realm of Star Trek.
  - whimsicalism 2y ago
    Luckily I don't think this is trending worse, HN has always been like this. If anything, it was worse a few years ago - especially during peak pandemic and the reddit issues. Nowadays people seem more positive.
  - dinobones 2y ago
    "But it hallucinates!!!!!" Yeah, and it's getting quantitatively better, all the time. There is a clear progress curve. Go back and look at the comical quality of GPT2 outputs, and the things people were getting excited about on Twitter. I wrote GPT off at the time because it just seemed like Markov model shenanigans to me, then I tried GPT3.5 and had my mind blown (but was still skeptical due to all the hallucinations), and GPT4 has finally convinced me that this is inevitable, and it's now just an iterative improvement game.
  - hermitcrab 2y ago
    I'm 58. It is hard not to be cynical when you've seen as many hype cycles as I have. Especially when you suspect the person hyping LLMs is the same person that was hyping blockchain and NFTs.
tinyhouse 2y ago
[flagged]
bigyikes 2y ago
This article, like so many AI hit pieces, fails to make any credible claim against AI. This one spent far too long propping up a tenuous analogy to smoke and mirrors. Is current-gen AI valuable? Obviously. Look at how many people say as much and willingly pay for the services. Is current-gen AI worth the hype? No, not really. Current models are impressive but highly limited, as any user can attest (and as any hater will focus on). What the naysayers always miss is exponential growth. They don’t look towards the future. Nvidia’s stock is high because of hype, yes, but also because the market is anticipating huge advancements in model ability. Current models are limited, but future models are world-changing, and we have every reason to believe models will continue to scale in this way. Just you wait for GPT-5.
- skybrian 2y ago
  The hype is the part where you start talking about the future. We have little idea what will be in GPT-5. OpenAI hasn't announced anything. Sam Altman does hype it up sometimes, in very vague terms. > future models are world-changing Possibly, but talking about what you imagine the future will be in the present tense is hype. That's not the future, it's speculation about one scenario. There are others. There's been a lot of progress making models that aren't quite as good as GPT-4, or are maybe comparable. Why is that? Have we hit a wall? I don't know, but that's a scenario, too.
- racional 2y ago
  Future models are world-changing, and we have every reason to believe models will continue to scale in this way. So how do you know this exactly? What the naysayers always miss is exponential growth. They don’t look towards the future. This is quite a pat response. There are plenty of critics within the AI/ML field of the unbounded growth hypothesis that you are effectively fronting here. I don't think you really believe that they "don't look towards the future", or that they aren't aware of the concept of exponential growth. I'd be interested in a cohesive response to their arguments. But in the above post at least, I'm not seeing one.
- foobarbaz321 2y ago
  [flagged]
GlenTheEskimo 2y ago
"Please write the first couple sections for an article based exclusively on the content below. Be direct and concise, but dive into details. Use a personal tone with “you” language. Write in the style of blogger Brian Dean. Use short paragraphs for easy readability. Change the wording so we don’t plagiarize the source material. Make the headlines somewhat catchy and intriguing while still concisely communicating what each section is about. - AI, also known as artificial intelligence, is overhyped - AI will never be able to replicate what a human can do - AI will never replace humans - AI is a trend that will die out - AI is not really artificial intelligence, just a trick"
ctoth 2y ago
> a bombshell scientific paper recently cast serious doubt on AI developers’ ability to continue to dramatically improve their models’ performance. This is the problem with this type of (AI isn't real, I promise!) writing. Each and every single time. This bombshell scientific result that he links to? Is Gary Marcus lamenting about a single paper, specific to a single architecture, which primarily shows how quickly algorithmic improvements are bringing down energy costs[0]. We get it. You don't like AI. But you don't have to lie about it, unless of course you are Gary Marcus and this is how you talk your book and earn your speaking fees. [0]: Algorithmic progress in language models. https://arxiv.org/abs/2403.05812 https://arxiv.org/abs/2403.05812
EVyesnoyesnoyes 2y ago
[flagged]
stuckinhell 2y ago
I can't agree anymore. After trying out udio.com my mind is completely blown at what I thought was possible.
rqtwteye 2y ago
There is the old quote “we overestimate the impact of technology in the short-term and underestimate the effect in the long run”. I watched this with the internet. First it was a toy (before maybe 1991}, then it was massively overhyped (1992-2001), then disillusionment kicked, in the end it transformed a lot of business, maybe not in the way predicted but the impact is undeniable. Same with AI, right now we are probably at the 1992 internet stage. There will be a lot of wrong paths but it’s pretty clear this will have a massive impact. People were right to be skeptical about blockchain and cryptic, but AI is on a totally different level of usefulness.
spaceguillotine 2y ago
I do think they are right about AI corrupting infrastructure of the internet, Google is pretty much useless if you aren't trying to buy a product and is full of SEO'd to hell AI generated content that is just terrible or completely lacking information. It has made job seeking a pain in the ass and is leading to tools of manipulation on every side, nothing is real, everything is a grift now seeking short term gains while playing scorched earth with everything long term.
deleted 2y ago
[deleted]
itvision 2y ago
[flagged]
spyckie2 2y ago
IMO, the struggles with AI in robotics exposes where the current AI bottlenecks are more clearly. Specifically, we don't have a training data corpus for any of our AI robots, enough to help them perform generalized task behaviors. So we have to go out and create them / use non training techniques, which crucially cannot leverage the existing learning / movements of humans accomplishing things. ChatAI's training data of "generalized text on the internet" mirrors exactly what it can do right now — know a bit about everything. But there lacks computer-readable corpuses for the things that are truly useful — 1) efforts that lead to a task accomplishment and 2) better-than human efforts that lead to better-than-human output. The key question is this: can AI reach beyond the human dataset that is currently powering it, crucially, at low(ish) cost? Or will AI always be fed training data? If we think the former, then we have to admit our algos aren't quite there yet and this iteration of AI is selling that dream. But even with our current data-fed AI algos, we have not yet put a lot of effort into the root of the challenge - how to quickly build large sets of high quality training data. I think there can be optimization here (over a period of 10-20 years) on a similar scale of going from million dollar computer mainframes to hundred dollar laptops. If high quality training data for specialized efforts were indeed available at a low cost, then the AI revolution will effectively deliver the dream it promises - just wait for Moore's law. The first step to solving that is an extremely polished LLM. You can see the proof of concept with SORA and other image-to-text efforts on how quality training data is created with the help of LLMs. But of course, "LLM generated great training data" doesn't draw headlines, so it may just look like nothing is happening for a long, long time, until you see AI task bots pop up everywhere.
constantcrying 2y ago
The Problem is that it isn't smoke and mirrors. People see legitimately novel and exiting new things computers can do. The Problems is that those things are extrapolated into the future based on insanely optimistic predictions. Up until now every single innovation has had times of rapid advances and adoption followed by a slowing down and stagnation. Believing that artificial neural networks aren't subject to that trend is, again, extremely optimistic. The greatest folly, I believe, is people who look at arbitrary problems and imagine AI solving them. This will not happen. Yet I see that thinking again and again. Artificial neural networks are good at certain things and bad at others. OpenAI has succeeded massive on making people believe that it can continually improve on basically every problem at an accelerating rate. If that is true, it is obviously the most valuable company in the world. But if artificial neural networks behave like any other innovation there will be a point at which implementation is impractical and improvements will be limited.
MP_1729 2y ago
It seems hard for people to take a nuanced approach that GPT-4 level models have in the present the potential to improve many people's lives and corporations' bottom lines while still being cautiously pessimists on the next generation of models. GPT-4, Claude 3 & Co. are simply too useful for certain coding tasks or to review a contract. Obviously, you need to understand you're dealing with a probabilistic being, and for many tasks it isn't the correct tool, but I use ChatGPT ~5 a day and the $20 are super well spent. Now, there's an ocean apart of me liking to use ChatGPT and the promises from Silicon Valley and Microsoft.
HarHarVeryFunny 2y ago
IMO it's not an entirely unreasonable take. It seems some companies adopting GenAI at this stage are doing so, at least in part, because they want to believe (that they can replace workers), rather than out of any sober analysis of what it can actually do. Maybe there's an element of FOMO too, and companies wanting to use GenAI because they hear everyone talking about it. No doubt LLMs will continue to improve, but what remains to be seen is if there will be a direct path from LLMs to AGI (which is where the real value gets unlocked), or if we'll just continue to see quantitative improvement in benchmark scores, hallucination reduction, etc, but not much qualitative change in the types of task they are capable of. I do think that AGI is inevitable, maybe not even that far off (but certainly not next 5 years, probably not next 10 years *), but there's a lot missing from LLMs to get there, and it seems that at least one critical piece, online learning, may require a different approach. * Note that it's already been 7 years since the transformer paper came out, and all we've really seen since then is a bunch of engineering work in making them more efficient and how best to train them. We haven't yet seen any advances in "cognitive architecture", or even any widespread recognition that there is a need to do so. If all people are doing for the foreseeable future is building pre-trained LLMs, then that is all we will get.
ryandvm 2y ago
Look, when I can dump a function into GPT4 and tell it to "refactor the following function with an emphasis on maintainability and testing" and it does a pretty damn good job of refactoring it, I have a hard time believing that this is "smoke and mirrors".
tim333 2y ago
>AI’s smoke and mirrors moment - Because of course, AI is not inevitable. Long term it seems pretty inevitable to me. Maybe the current LLMs are overhyped and there will be an investment slump before the next improved algorithm but long term it's here to stay. It's probably a bit like the early days of electical power. Some of the early gadgets were no doubt rubbish but electrical stuff was there to stay and become a normal part of the world.

71 points by cdme 2y ago

hiddencost 2y ago

Machine learning however works, and generates hundreds of billions of dollars a year. Both can be true!

hulitu 2y ago
Yes, but they are not related to each other.
- whimsicalism 2y ago
  they are absolutely related to each other

furyofantares 2y ago

The usual pattern is to overestimate short-term change and underestimate long-term change. We're a year since this stuff really got popular, and so I think a thing that's happening right now is many people noticing the short-term changes were overestimated.

Upvoter33 2y ago
yes, exactly. There will be a lot of these types of articles in the next few years, some disappointments along the way, but deep and longlasting change in the tech sector and beyond is coming.

whimsicalism 2y ago

No matter how much the chattering class wants to portray this as a "fad" and not an inevitability, those of us who can type and read can see the reality for ourselves.

mlyle 2y ago
Yah-- what we have now is enough to be enormously disruptive as it is absorbed. The big question is how much better it's going to get. We may be plateauing in capability-- or at least slowing in capability growth-- or we may not be. But it doesn't need to get better for it to eat a lot of peoples' lunch.
- lambdaba 2y ago
  My opinion shifts every other day it seems, recently, Roon, the "famous" anon X account that works at OpenAI, and presumably has access to some unreleased info, said, in response to "are we plateauing soon", "not even close". I honestly don't know what to think.
  - whimsicalism 2y ago
    I know his real identity and I doubt there is some key insider info there although he does work at OAI
  - mlyle 2y ago
    I feel like for LLM's at least, we've not seen a lot of progress on the high end for a year or so. GPT4-Turbo is a little better than GPT4, but mainly only for things that can benefit from huge contexts.
    - deleted 2y ago
      [deleted]
    - whimsicalism 2y ago
      We have seen progress, but mostly price progress.
      - mlyle 2y ago
        Yes, and also not-always-bumping-into-capacity-limits (availability) progress.

simonbarker87 2y ago

Clearly AI (re-skinned ML) is following the standard Gartner hype cycle graph.

pantsforbirds 2y ago

I think the AI itself is working extremely well. The state-of-the-art LLMs are exceptional at many real-world tasks and are deployed in many production code bases successfully. The more recent AI failures (like the humane pin) are at least partially a hardware problem. These LLMs require huge amounts of compute to do nontrivial tasks, and a wearable just can't run locally with the current architectures.

FrustratedMonky 2y ago
Exactly. Yes, there will be failures along the way, but the AI tools we have now are still pretty amazing and really would have been called magical 5 years ago. And yes, the AI tech already available now is already causing disruption and will continue to disrupt. Anybody saying this is just a 'blip' or 'fad' are really not trying them out. I remember in the 90's someone telling me to not bother with Java (back with applets), because this 'Internet Thing' isn't going to pan out. That is how the anti-AI sentiment sounds to me. It's already amazing and continues to grow. There is not pulling it back now.

yawnxyz 2y ago

That's a lot of words from someone who doesn't seem to have tried using these tools yet? The Hype is smoke and mirrors, but there are tons of real use cases where things were MUCH easier than before. Things like text to speech or image recognition previously required a team of 10 ML engineers a couple of years to build now take a weekend. It's smoke and mirrors to the "business world" because none of these translate into bottom line numbers though, and I think that's all this guy is able to see

hn_throwaway_99 2y ago
I'm just kinda sad at how cynical so many of us have become. I mean, I see tons of comments "But it hallucinates!!!!!" Yeah, no shit. At the same time, you're having a perfectly grammatically correct, natural, sensical conversation with a computer. Just a couple years ago that was purely the realm of Star Trek.
- whimsicalism 2y ago
  Luckily I don't think this is trending worse, HN has always been like this. If anything, it was worse a few years ago - especially during peak pandemic and the reddit issues. Nowadays people seem more positive.
- dinobones 2y ago
  "But it hallucinates!!!!!" Yeah, and it's getting quantitatively better, all the time. There is a clear progress curve. Go back and look at the comical quality of GPT2 outputs, and the things people were getting excited about on Twitter. I wrote GPT off at the time because it just seemed like Markov model shenanigans to me, then I tried GPT3.5 and had my mind blown (but was still skeptical due to all the hallucinations), and GPT4 has finally convinced me that this is inevitable, and it's now just an iterative improvement game.
- hermitcrab 2y ago
  I'm 58. It is hard not to be cynical when you've seen as many hype cycles as I have. Especially when you suspect the person hyping LLMs is the same person that was hyping blockchain and NFTs.

tinyhouse 2y ago

[flagged]

bigyikes 2y ago

This article, like so many AI hit pieces, fails to make any credible claim against AI. This one spent far too long propping up a tenuous analogy to smoke and mirrors. Is current-gen AI valuable? Obviously. Look at how many people say as much and willingly pay for the services. Is current-gen AI worth the hype? No, not really. Current models are impressive but highly limited, as any user can attest (and as any hater will focus on). What the naysayers always miss is exponential growth. They don’t look towards the future. Nvidia’s stock is high because of hype, yes, but also because the market is anticipating huge advancements in model ability. Current models are limited, but future models are world-changing, and we have every reason to believe models will continue to scale in this way. Just you wait for GPT-5.

skybrian 2y ago
The hype is the part where you start talking about the future. We have little idea what will be in GPT-5. OpenAI hasn't announced anything. Sam Altman does hype it up sometimes, in very vague terms. > future models are world-changing Possibly, but talking about what you imagine the future will be in the present tense is hype. That's not the future, it's speculation about one scenario. There are others. There's been a lot of progress making models that aren't quite as good as GPT-4, or are maybe comparable. Why is that? Have we hit a wall? I don't know, but that's a scenario, too.
racional 2y ago
Future models are world-changing, and we have every reason to believe models will continue to scale in this way. So how do you know this exactly? What the naysayers always miss is exponential growth. They don’t look towards the future. This is quite a pat response. There are plenty of critics within the AI/ML field of the unbounded growth hypothesis that you are effectively fronting here. I don't think you really believe that they "don't look towards the future", or that they aren't aware of the concept of exponential growth. I'd be interested in a cohesive response to their arguments. But in the above post at least, I'm not seeing one.
foobarbaz321 2y ago
[flagged]

GlenTheEskimo 2y ago

"Please write the first couple sections for an article based exclusively on the content below. Be direct and concise, but dive into details. Use a personal tone with “you” language. Write in the style of blogger Brian Dean. Use short paragraphs for easy readability. Change the wording so we don’t plagiarize the source material. Make the headlines somewhat catchy and intriguing while still concisely communicating what each section is about. - AI, also known as artificial intelligence, is overhyped - AI will never be able to replicate what a human can do - AI will never replace humans - AI is a trend that will die out - AI is not really artificial intelligence, just a trick"

ctoth 2y ago

> a bombshell scientific paper recently cast serious doubt on AI developers’ ability to continue to dramatically improve their models’ performance. This is the problem with this type of (AI isn't real, I promise!) writing. Each and every single time. This bombshell scientific result that he links to? Is Gary Marcus lamenting about a single paper, specific to a single architecture, which primarily shows how quickly algorithmic improvements are bringing down energy costs[0]. We get it. You don't like AI. But you don't have to lie about it, unless of course you are Gary Marcus and this is how you talk your book and earn your speaking fees. [0]: Algorithmic progress in language models. https://arxiv.org/abs/2403.05812 https://arxiv.org/abs/2403.05812

EVyesnoyesnoyes 2y ago

[flagged]

stuckinhell 2y ago

I can't agree anymore. After trying out udio.com my mind is completely blown at what I thought was possible.

rqtwteye 2y ago

There is the old quote “we overestimate the impact of technology in the short-term and underestimate the effect in the long run”. I watched this with the internet. First it was a toy (before maybe 1991}, then it was massively overhyped (1992-2001), then disillusionment kicked, in the end it transformed a lot of business, maybe not in the way predicted but the impact is undeniable. Same with AI, right now we are probably at the 1992 internet stage. There will be a lot of wrong paths but it’s pretty clear this will have a massive impact. People were right to be skeptical about blockchain and cryptic, but AI is on a totally different level of usefulness.

spaceguillotine 2y ago

I do think they are right about AI corrupting infrastructure of the internet, Google is pretty much useless if you aren't trying to buy a product and is full of SEO'd to hell AI generated content that is just terrible or completely lacking information. It has made job seeking a pain in the ass and is leading to tools of manipulation on every side, nothing is real, everything is a grift now seeking short term gains while playing scorched earth with everything long term.

deleted 2y ago

[deleted]

itvision 2y ago

[flagged]

spyckie2 2y ago

IMO, the struggles with AI in robotics exposes where the current AI bottlenecks are more clearly. Specifically, we don't have a training data corpus for any of our AI robots, enough to help them perform generalized task behaviors. So we have to go out and create them / use non training techniques, which crucially cannot leverage the existing learning / movements of humans accomplishing things. ChatAI's training data of "generalized text on the internet" mirrors exactly what it can do right now — know a bit about everything. But there lacks computer-readable corpuses for the things that are truly useful — 1) efforts that lead to a task accomplishment and 2) better-than human efforts that lead to better-than-human output. The key question is this: can AI reach beyond the human dataset that is currently powering it, crucially, at low(ish) cost? Or will AI always be fed training data? If we think the former, then we have to admit our algos aren't quite there yet and this iteration of AI is selling that dream. But even with our current data-fed AI algos, we have not yet put a lot of effort into the root of the challenge - how to quickly build large sets of high quality training data. I think there can be optimization here (over a period of 10-20 years) on a similar scale of going from million dollar computer mainframes to hundred dollar laptops. If high quality training data for specialized efforts were indeed available at a low cost, then the AI revolution will effectively deliver the dream it promises - just wait for Moore's law. The first step to solving that is an extremely polished LLM. You can see the proof of concept with SORA and other image-to-text efforts on how quality training data is created with the help of LLMs. But of course, "LLM generated great training data" doesn't draw headlines, so it may just look like nothing is happening for a long, long time, until you see AI task bots pop up everywhere.

constantcrying 2y ago

The Problem is that it isn't smoke and mirrors. People see legitimately novel and exiting new things computers can do. The Problems is that those things are extrapolated into the future based on insanely optimistic predictions. Up until now every single innovation has had times of rapid advances and adoption followed by a slowing down and stagnation. Believing that artificial neural networks aren't subject to that trend is, again, extremely optimistic. The greatest folly, I believe, is people who look at arbitrary problems and imagine AI solving them. This will not happen. Yet I see that thinking again and again. Artificial neural networks are good at certain things and bad at others. OpenAI has succeeded massive on making people believe that it can continually improve on basically every problem at an accelerating rate. If that is true, it is obviously the most valuable company in the world. But if artificial neural networks behave like any other innovation there will be a point at which implementation is impractical and improvements will be limited.

MP_1729 2y ago

It seems hard for people to take a nuanced approach that GPT-4 level models have in the present the potential to improve many people's lives and corporations' bottom lines while still being cautiously pessimists on the next generation of models. GPT-4, Claude 3 & Co. are simply too useful for certain coding tasks or to review a contract. Obviously, you need to understand you're dealing with a probabilistic being, and for many tasks it isn't the correct tool, but I use ChatGPT ~5 a day and the $20 are super well spent. Now, there's an ocean apart of me liking to use ChatGPT and the promises from Silicon Valley and Microsoft.

HarHarVeryFunny 2y ago

IMO it's not an entirely unreasonable take. It seems some companies adopting GenAI at this stage are doing so, at least in part, because they want to believe (that they can replace workers), rather than out of any sober analysis of what it can actually do. Maybe there's an element of FOMO too, and companies wanting to use GenAI because they hear everyone talking about it. No doubt LLMs will continue to improve, but what remains to be seen is if there will be a direct path from LLMs to AGI (which is where the real value gets unlocked), or if we'll just continue to see quantitative improvement in benchmark scores, hallucination reduction, etc, but not much qualitative change in the types of task they are capable of. I do think that AGI is inevitable, maybe not even that far off (but certainly not next 5 years, probably not next 10 years *), but there's a lot missing from LLMs to get there, and it seems that at least one critical piece, online learning, may require a different approach. * Note that it's already been 7 years since the transformer paper came out, and all we've really seen since then is a bunch of engineering work in making them more efficient and how best to train them. We haven't yet seen any advances in "cognitive architecture", or even any widespread recognition that there is a need to do so. If all people are doing for the foreseeable future is building pre-trained LLMs, then that is all we will get.

ryandvm 2y ago

Look, when I can dump a function into GPT4 and tell it to "refactor the following function with an emphasis on maintainability and testing" and it does a pretty damn good job of refactoring it, I have a hard time believing that this is "smoke and mirrors".

tim333 2y ago

>AI’s smoke and mirrors moment - Because of course, AI is not inevitable. Long term it seems pretty inevitable to me. Maybe the current LLMs are overhyped and there will be an investment slump before the next improved algorithm but long term it's here to stay. It's probably a bit like the early days of electical power. Some of the early gadgets were no doubt rubbish but electrical stuff was there to stay and become a normal part of the world.