4 ms·

Where on the S-shaped curve are we? I am curious if one day the paradigm that brought forth the current chatbots has to be abandoned because it's reached his li

Where on the S-shaped curve are we? I am curious if one day the paradigm that brought forth the current chatbots has to be abandoned because it's reached his limits, and a new approach will replace it. But certainly it has inspired the field of AI massively and I expect many new developments which will lead is into a future that resembles sci-fi books and movies. I am also interested in what good sci-fi will look like in the future? It looks like "the singularity" is defined by the limits of our imagination. We cannot imagine what comes after it, so this means we cannot write sci-fi on it, right?

amelius 3y ago
> Where on the S-shaped curve are we? Why do you think it's an S-shape? As you say, we're heading for a singularity!
- CatWChainsaw 3y ago
  Yeah it sure would be convenient if this is the one time in all of nature that a curve that looked like it was going exponential before suddenly conforming to all past trends and becoming sigmoid was actually exponential after all because you really really want it to be.
- GuB-42 3y ago
  Because exponentials will eventually break the laws of physics. For instance there is a relationship between information and energy, so a system processing an exponential amount of information will need an exponential amount of energy, which the universe will eventually run out of.
  - amelius 3y ago
    At this point we know very little and we have to leave it to the AIs to determine if physics has any limits.
    - lewhoo 3y ago
      At this point we know very little and we have to leave it to the AIs to determine if physics has any limits. That's a tall order. Despite all the fancy news headlines, which I must admit gave me existential anxiety at some point, it seems that AI still can't truly make a discovery. Also I really don't know if we know little or a lot, how can we really tell ? It could be argued we actually know a lot because it is ridiculously hard to make an actual big breakthrough in physics.
foobarbecue 3y ago
Tangent: It has always annoyed me that a sigmoid curve does not look like a sigma, and it doesn't really look much like an S either. Just the middle part of an S. Am I missing something?
- roenxi 3y ago
  Old school handwriting, probably [0]. When people had to squiggle the glyphs on paper they had all sorts of weird and wonderful ways of scribing. Consider that the integral symbol is actually an S. [0] https://en.wikipedia.org/wiki/Long_s https://en.wikipedia.org/wiki/Long_s
  - foobarbecue 3y ago
    Oooh that makes sense, thanks!
- addandsubtract 3y ago
  Hyperbolic tangent :P
  - amelius 3y ago
    Ha, good one! :)
gerad 3y ago
For LLMs and generative AI we’re likely at the start of the curve. Things take longer than you expect in the short-term and less time than you expect in the long-term. After this hype cycle ends, things will probably bump along slowly for a decade or so, then suddenly you’ll look up and they’ll be everywhere, and you’ll wonder “when did that happen?” At least that’s the way it felt for other technologies to me. Internet, Bluetooth, cell phones, etc.
- janalsncm 3y ago
  > After this hype cycle ends, things will probably bump along slowly for a decade or so, then suddenly you’ll look up and they’ll be everywhere, and you’ll wonder “when did that happen?” The reason this is incorrect is that “AI” isn’t one capability. It’s a huge swath of capabilities under the umbrella of “automate X”. All of them have been enabled by software and hardware improvements over the last decades. The underlying technologies aren’t that new in many cases either, it’s just the scale that’s different. After we get bored of one huge new capability, another comes along. A year ago everyone was talking about automatic image generation. Now it’s automatic text generation. There are plenty of other things we’d like to automate. So the hype train is likely to continue for quite a while.
jiggawatts 3y ago
> Where on the S-shaped curve are we? No-one knows. We can however make educated guesses based on: - Extrapolating the last couple of years of progress. - Scientific studies of LLM quality scaling in proportion to input data size, parameter count, and total training compute ops. - How much more training data is available. - Reasonable budgets, especially the cost-efficiency of inference, which appears to be more limiting than training, especially for long-term usage and profitability. - Efficiencies such as 4-bit quantization, algorithmic improvements, etc... Based on the above, there is at least a factor of 10x scaling available in many directions within a year or two, but not likely all at once. E.g.: 10x context window size at the same intelligence level? Sure! 10x inference speed? Doable! 10x cost reduction? Coming soon! All at once? Not yet, and not for a while. Everyone is hardware constrained, and demand is pushing up prices and limiting the training scale maximums. Maximum intelligence is much harder to predict. The current generation of AIs are trained on human-authored text as their input data. They're trained to predict that text. That means that they're "blurry JPEGs of the Internet". More training might make them "sharper JPEGs", but not necessarily smarter because the Internet didn't get smarter. They'll better model humans, but they'll still be modelling humans, not superhumans. That can be fixed with self-training, etc... but that will take a lot longer. I'm guessing 5-10 years, which is in line with the predictions of AI futurists that seem to know what they're talking about. Many of our jobs will remain safe... for now.
- ChatGTP 3y ago
  It really stands to reason that at some point these systems will just be "too smart" and we won't find any value in something that much more intelligent. It's like a 4 year old trying to understand a seasoned professor explaining some concept, we just won't get it. Imagine a hypothetical scenario where some ultra-intelligent beings delivered a book with all the answers humans ever wanted, but we needed an IQ of 500 t understand it. This IMO is where we're heading. There might be some pretty sweet low hanging fruit to attain along the way, but at some stage, we'll hit a ceiling and the only way forward would be to augment our own intelligence.
  - throwaway22032 3y ago
AnimalMuppet 3y ago
That's kind of hard to say. You don't know how long the curve is until you're somewhat past halfway, unless you can very carefully measure progress. That's easy with something like transistor size. That's hard with AI, because we can't put a number on how good it is. (The model size is probably not a good number for measuring AI progress.) So nobody knows. We're all guessing.
janalsncm 3y ago
The S curve only makes sense for one technology at a time. AI isn't one technology. It’s a category of technologies.

amelius 3y ago

> Where on the S-shaped curve are we? Why do you think it's an S-shape? As you say, we're heading for a singularity!

CatWChainsaw 3y ago
Yeah it sure would be convenient if this is the one time in all of nature that a curve that looked like it was going exponential before suddenly conforming to all past trends and becoming sigmoid was actually exponential after all because you really really want it to be.
GuB-42 3y ago
Because exponentials will eventually break the laws of physics. For instance there is a relationship between information and energy, so a system processing an exponential amount of information will need an exponential amount of energy, which the universe will eventually run out of.
- amelius 3y ago
  At this point we know very little and we have to leave it to the AIs to determine if physics has any limits.
  - lewhoo 3y ago
    At this point we know very little and we have to leave it to the AIs to determine if physics has any limits. That's a tall order. Despite all the fancy news headlines, which I must admit gave me existential anxiety at some point, it seems that AI still can't truly make a discovery. Also I really don't know if we know little or a lot, how can we really tell ? It could be argued we actually know a lot because it is ridiculously hard to make an actual big breakthrough in physics.

foobarbecue 3y ago

Tangent: It has always annoyed me that a sigmoid curve does not look like a sigma, and it doesn't really look much like an S either. Just the middle part of an S. Am I missing something?

roenxi 3y ago
Old school handwriting, probably [0]. When people had to squiggle the glyphs on paper they had all sorts of weird and wonderful ways of scribing. Consider that the integral symbol is actually an S. [0] https://en.wikipedia.org/wiki/Long_s https://en.wikipedia.org/wiki/Long_s
- foobarbecue 3y ago
  Oooh that makes sense, thanks!
addandsubtract 3y ago
Hyperbolic tangent :P
- amelius 3y ago
  Ha, good one! :)

gerad 3y ago

For LLMs and generative AI we’re likely at the start of the curve. Things take longer than you expect in the short-term and less time than you expect in the long-term. After this hype cycle ends, things will probably bump along slowly for a decade or so, then suddenly you’ll look up and they’ll be everywhere, and you’ll wonder “when did that happen?” At least that’s the way it felt for other technologies to me. Internet, Bluetooth, cell phones, etc.

janalsncm 3y ago
> After this hype cycle ends, things will probably bump along slowly for a decade or so, then suddenly you’ll look up and they’ll be everywhere, and you’ll wonder “when did that happen?” The reason this is incorrect is that “AI” isn’t one capability. It’s a huge swath of capabilities under the umbrella of “automate X”. All of them have been enabled by software and hardware improvements over the last decades. The underlying technologies aren’t that new in many cases either, it’s just the scale that’s different. After we get bored of one huge new capability, another comes along. A year ago everyone was talking about automatic image generation. Now it’s automatic text generation. There are plenty of other things we’d like to automate. So the hype train is likely to continue for quite a while.

jiggawatts 3y ago

> Where on the S-shaped curve are we? No-one knows. We can however make educated guesses based on: - Extrapolating the last couple of years of progress. - Scientific studies of LLM quality scaling in proportion to input data size, parameter count, and total training compute ops. - How much more training data is available. - Reasonable budgets, especially the cost-efficiency of inference, which appears to be more limiting than training, especially for long-term usage and profitability. - Efficiencies such as 4-bit quantization, algorithmic improvements, etc... Based on the above, there is at least a factor of 10x scaling available in many directions within a year or two, but not likely all at once. E.g.: 10x context window size at the same intelligence level? Sure! 10x inference speed? Doable! 10x cost reduction? Coming soon! All at once? Not yet, and not for a while. Everyone is hardware constrained, and demand is pushing up prices and limiting the training scale maximums. Maximum intelligence is much harder to predict. The current generation of AIs are trained on human-authored text as their input data. They're trained to predict that text. That means that they're "blurry JPEGs of the Internet". More training might make them "sharper JPEGs", but not necessarily smarter because the Internet didn't get smarter. They'll better model humans, but they'll still be modelling humans, not superhumans. That can be fixed with self-training, etc... but that will take a lot longer. I'm guessing 5-10 years, which is in line with the predictions of AI futurists that seem to know what they're talking about. Many of our jobs will remain safe... for now.

ChatGTP 3y ago
It really stands to reason that at some point these systems will just be "too smart" and we won't find any value in something that much more intelligent. It's like a 4 year old trying to understand a seasoned professor explaining some concept, we just won't get it. Imagine a hypothetical scenario where some ultra-intelligent beings delivered a book with all the answers humans ever wanted, but we needed an IQ of 500 t understand it. This IMO is where we're heading. There might be some pretty sweet low hanging fruit to attain along the way, but at some stage, we'll hit a ceiling and the only way forward would be to augment our own intelligence.
- throwaway22032 3y ago

AnimalMuppet 3y ago

That's kind of hard to say. You don't know how long the curve is until you're somewhat past halfway, unless you can very carefully measure progress. That's easy with something like transistor size. That's hard with AI, because we can't put a number on how good it is. (The model size is probably not a good number for measuring AI progress.) So nobody knows. We're all guessing.

janalsncm 3y ago

The S curve only makes sense for one technology at a time. AI isn't one technology. It’s a category of technologies.