4 ms·

The hype is not because what the status quo is today (think ARPANet), but because so many experts (not just investors) in the field believe that LLMs (and its m

The hype is not because what the status quo is today (think ARPANet), but because so many experts (not just investors) in the field believe that LLMs (and its multi modal versions) are on an exponential (quadratic?) improvement trajectory. Before today, access to AI was gated by firms like Google, who kept the implementation to themsevles as a competitive advantage (but did publish copious amounts of research). With these LLMs being so general purpose and made readily accessible to other software shops at affordable rates, things are in for a dramatic shift as far as utility AI is concerned. A vertical integrator in Apple just demonstrated how the tech is really coming together with its VisionPro launch. For an example closer to home, GitHub was right on the money with Copilot. Personally speaking, I've been using Copilot for some months now, and I can barely program without it. Not because I forgot how to program, but because it is so darn good that I'd rather not code without it. Surely, I can't be the only one that feels this way? If the experts are right about the exponentials, then the world is going to look very different by the time GPT-6 rolls out, especially if the models get cheaper (even as good as free bar computer costs, in the case of open source models) as they get more capable.

version_five 3y ago
The "experts" are experts in ML: stats and linear algebra and whatnot. Their hot takes about it coming alive or through some other means destroying us are just as irrelevant as any other lay person's. From what I can see, a lot of this is about delusions of self importance on the researchers' part. It's not a coincidence, though a bit ironic, that LeCun, who comes across as the most arrogant and contrarian is the least caught up in the "oh what have we done" faux hype and the most realistic about what the technology is.
- visarga 3y ago
  I think you should take into consideration the fact that AI researchers spend much more time analysing errors and studying failure modes, hypothesising about causes, because that is their f job. They might come out as pessimists because of it, but that's only because of so many times they got confirmation before. Laypeople spend a few hours playing with the models and think they already understood their limitations. At the present moment there is no AI model that can successfully execute critical tasks completely autonomously. That is a limitation that is confirmed over all the fields, modalities and applications. Many articles in the press simply ignore it, and extrapolate absurd timelines over the near future. The problem is that solving that last 1% gets exponentially harder.
- gmerc 3y ago
  Any argument that reduces itself to the credibility of people when the technology is freely available for testing and application is flawed. You can uze the capabilities today, no need to listen to experts.
- jhbadger 3y ago
  But it is interesting that (some) AI researchers have come to the conclusion that it can be world-ending. Yes, as you say, researchers do have a tendency to over exaggerate their field's importance, but I'm a biologist and it seems we spend a lot of our time with media reassuring people that despite what bad 1950s SF movies said, genetic research isn't going to create horrible monsters that will destroy the world.
  - version_five 3y ago
    That's an interesting point. I wonder if that would still be true if genetic research was "democratized" so that the barrier to entry was as low as writing a computer program? There's always some self interest involved. I'd guess in the current state of biology, researchers don't want additional limitations placed on them but are secure in their monopoly on the research. In the current state of ML, incumbents want limitations placed on others so they can have a monopoly. Add to that the desire to feel important, and you get the direction of the rhetoric.
ChatGTP 3y ago
Surely, I can't be the only one that feels this way? Not but there's a lot of people who don't feel that way too. So you're just one of many people who has their own preference. I use language servers and I find they're a game changer in a way that Copilot isn't for me. In fact I believe there was a study on here that showed Copilot only helps mediocre programmers improve the time it takes them to get something working. More experienced and skilled programmers received minimal benefit.
- ignoramous 3y ago
  Good thing that mediocre / barely programmers and script kiddies far outnumber experience and skilled ones. Copilot has the potential to go mass market.
  - ChatGTP 3y ago
    When I first saw co-pilot, this immediately popped into my head. CP is a script kiddies ultimate wet dream.
smoldesu 3y ago
> Before today, access to AI was gated by firms like Google Did you miss Talk-to-Transformer, GANbreeder and Stable Diffusion? Did you ignore GPT-J and the GPT-2 quantizations that existed long before LLaMA or ChatGPT were common parts of everyday discussion? I think a lot of people telling the history of AI completely miss the zeroth and first waves of AI generation. ChatGPT and GPT4 are asymptotes of that intelligent scaling - past this point, very little will impress the average person.
- ignoramous 3y ago
  Yes I did, because they weren't as good. The good stuff was all within the four walls at Google and probably one or two other companies? As a consumer, Google Search, Translate, Maps, Earth, Photos all have had impressive capabilities for a long time now; and I don't think any upstart could have hoped to compete with that. I, of course, think that is no longer the case with many researchers freeing themselves from the Google / BigTech Vortex and striking out on their own, having a very different strategy to enable other upstarts around them.
williamcotton 3y ago
I’ve been using GPT based coding tools for quite awhile. GPT-4 coupled with my extensive experience has enabled me to tackle just about every programming challenge I set for myself, from systems programming, neural networks and up to GUI programming on multiple kinds of devices and languages. I have been programming since the mid-90s so I have strong fundamentals. What I often lack is knowledge of syntax, APIs or libraries. Once presented with this information I am off to the races. Almost the entirety of this codebase was written by GPT: https://github.com/williamcotton/chordviz https://github.com/williamcotton/chordviz Yes, it required a background in LA and studying various ML techniques in order to understand how to guide the completions. It also frequently produced incorrect code but I was able to isolate the problems and reduce the complexity presented to GPT in order to get functional results. The majority of the time was spent labeling around 12,000 images using the labeling software that I guided GPT to construct. I now have a SwiftUI iOS application that takes a live camera feed and predicts guitar chords based on a PyTorch CNN model trained on images labeled with a React application.
- dinvlad 3y ago
  I think it’s fascinating one can get deep understanding of so many topics so quickly now!
  - williamcotton 3y ago
    In fairness, the deep understanding took many years of studying the fundamentals. I started playing around with LA techniques (SVD on images, k-means clustering) around 20 years ago. I had a copy of K&R in the mid 90s because before I found out about Perl I stumbled across a way to add comments and a visitor counter to my homepage using C in the cgi-bin of the shell account on my ISP. Where GPT has been the most valuable is the nitty-gritty of converting between various data types and image formats!
sampo 3y ago
> because so many experts ... indicating that LLMs ... are on an exponential ... improvement trajectory. The experts have consistently and grossly overestimated the progress in AI research since 1955. https://en.wikipedia.org/wiki/Dartmouth_workshop https://en.wikipedia.org/wiki/Dartmouth_workshop
- sebzim4500 3y ago
  I don't work in AI myself, but this is not consistent with my experience of talking to people who do. A lot of them were surprised by AlphaGo, GPT-3 and shocked by GPT-4. The fact that some people were ridiculously optimistic at the beginning should not take away from the fact that few people saw GPT-4 coming.
oh_sigh 3y ago
There's no such thing as improvement trajectory. The argument could sound like sci fi buffs in 1970 wringing their hands on what the governance of the alpha centauri colonies would look like. After all, the improvement trajectory from first flight, to flight being common, to first satellite, to first man in space, to walking on the moon shows a stunning improvement trajectory. There might be a huge breakthrough tomorrow, or we might hit a brick wall. Or what looked promising might have diminishing returns.
asjfka 3y ago
I expect people becoming much more conservative about technology from now on. People will want to stay with the thing were LLM can help because there are ample documentation to train them on. By example, it will be a lot harder for a new, even if it's more powerful, programming language to find its niche because LLM wont be able to help. The end result will be a higher barrier to innovation because now to innovate is not enough, you also need LLM training materials, thus a much more conservative society.

version_five 3y ago

The "experts" are experts in ML: stats and linear algebra and whatnot. Their hot takes about it coming alive or through some other means destroying us are just as irrelevant as any other lay person's. From what I can see, a lot of this is about delusions of self importance on the researchers' part. It's not a coincidence, though a bit ironic, that LeCun, who comes across as the most arrogant and contrarian is the least caught up in the "oh what have we done" faux hype and the most realistic about what the technology is.

visarga 3y ago
I think you should take into consideration the fact that AI researchers spend much more time analysing errors and studying failure modes, hypothesising about causes, because that is their f job. They might come out as pessimists because of it, but that's only because of so many times they got confirmation before. Laypeople spend a few hours playing with the models and think they already understood their limitations. At the present moment there is no AI model that can successfully execute critical tasks completely autonomously. That is a limitation that is confirmed over all the fields, modalities and applications. Many articles in the press simply ignore it, and extrapolate absurd timelines over the near future. The problem is that solving that last 1% gets exponentially harder.
gmerc 3y ago
Any argument that reduces itself to the credibility of people when the technology is freely available for testing and application is flawed. You can uze the capabilities today, no need to listen to experts.
jhbadger 3y ago
But it is interesting that (some) AI researchers have come to the conclusion that it can be world-ending. Yes, as you say, researchers do have a tendency to over exaggerate their field's importance, but I'm a biologist and it seems we spend a lot of our time with media reassuring people that despite what bad 1950s SF movies said, genetic research isn't going to create horrible monsters that will destroy the world.
- version_five 3y ago
  That's an interesting point. I wonder if that would still be true if genetic research was "democratized" so that the barrier to entry was as low as writing a computer program? There's always some self interest involved. I'd guess in the current state of biology, researchers don't want additional limitations placed on them but are secure in their monopoly on the research. In the current state of ML, incumbents want limitations placed on others so they can have a monopoly. Add to that the desire to feel important, and you get the direction of the rhetoric.

ChatGTP 3y ago

Surely, I can't be the only one that feels this way? Not but there's a lot of people who don't feel that way too. So you're just one of many people who has their own preference. I use language servers and I find they're a game changer in a way that Copilot isn't for me. In fact I believe there was a study on here that showed Copilot only helps mediocre programmers improve the time it takes them to get something working. More experienced and skilled programmers received minimal benefit.

ignoramous 3y ago
Good thing that mediocre / barely programmers and script kiddies far outnumber experience and skilled ones. Copilot has the potential to go mass market.
- ChatGTP 3y ago
  When I first saw co-pilot, this immediately popped into my head. CP is a script kiddies ultimate wet dream.

smoldesu 3y ago

> Before today, access to AI was gated by firms like Google Did you miss Talk-to-Transformer, GANbreeder and Stable Diffusion? Did you ignore GPT-J and the GPT-2 quantizations that existed long before LLaMA or ChatGPT were common parts of everyday discussion? I think a lot of people telling the history of AI completely miss the zeroth and first waves of AI generation. ChatGPT and GPT4 are asymptotes of that intelligent scaling - past this point, very little will impress the average person.

ignoramous 3y ago
Yes I did, because they weren't as good. The good stuff was all within the four walls at Google and probably one or two other companies? As a consumer, Google Search, Translate, Maps, Earth, Photos all have had impressive capabilities for a long time now; and I don't think any upstart could have hoped to compete with that. I, of course, think that is no longer the case with many researchers freeing themselves from the Google / BigTech Vortex and striking out on their own, having a very different strategy to enable other upstarts around them.

williamcotton 3y ago

I’ve been using GPT based coding tools for quite awhile. GPT-4 coupled with my extensive experience has enabled me to tackle just about every programming challenge I set for myself, from systems programming, neural networks and up to GUI programming on multiple kinds of devices and languages. I have been programming since the mid-90s so I have strong fundamentals. What I often lack is knowledge of syntax, APIs or libraries. Once presented with this information I am off to the races. Almost the entirety of this codebase was written by GPT: https://github.com/williamcotton/chordviz https://github.com/williamcotton/chordviz Yes, it required a background in LA and studying various ML techniques in order to understand how to guide the completions. It also frequently produced incorrect code but I was able to isolate the problems and reduce the complexity presented to GPT in order to get functional results. The majority of the time was spent labeling around 12,000 images using the labeling software that I guided GPT to construct. I now have a SwiftUI iOS application that takes a live camera feed and predicts guitar chords based on a PyTorch CNN model trained on images labeled with a React application.

dinvlad 3y ago
I think it’s fascinating one can get deep understanding of so many topics so quickly now!
- williamcotton 3y ago
  In fairness, the deep understanding took many years of studying the fundamentals. I started playing around with LA techniques (SVD on images, k-means clustering) around 20 years ago. I had a copy of K&R in the mid 90s because before I found out about Perl I stumbled across a way to add comments and a visitor counter to my homepage using C in the cgi-bin of the shell account on my ISP. Where GPT has been the most valuable is the nitty-gritty of converting between various data types and image formats!

sampo 3y ago

> because so many experts ... indicating that LLMs ... are on an exponential ... improvement trajectory. The experts have consistently and grossly overestimated the progress in AI research since 1955. https://en.wikipedia.org/wiki/Dartmouth_workshop https://en.wikipedia.org/wiki/Dartmouth_workshop

sebzim4500 3y ago
I don't work in AI myself, but this is not consistent with my experience of talking to people who do. A lot of them were surprised by AlphaGo, GPT-3 and shocked by GPT-4. The fact that some people were ridiculously optimistic at the beginning should not take away from the fact that few people saw GPT-4 coming.

oh_sigh 3y ago

There's no such thing as improvement trajectory. The argument could sound like sci fi buffs in 1970 wringing their hands on what the governance of the alpha centauri colonies would look like. After all, the improvement trajectory from first flight, to flight being common, to first satellite, to first man in space, to walking on the moon shows a stunning improvement trajectory. There might be a huge breakthrough tomorrow, or we might hit a brick wall. Or what looked promising might have diminishing returns.

asjfka 3y ago

I expect people becoming much more conservative about technology from now on. People will want to stay with the thing were LLM can help because there are ample documentation to train them on. By example, it will be a lot harder for a new, even if it's more powerful, programming language to find its niche because LLM wont be able to help. The end result will be a higher barrier to innovation because now to innovate is not enough, you also need LLM training materials, thus a much more conservative society.