3 ms·

>Have we hit peak AI? AI is at the same point cinema was at when pioneers of the genre showed short films in "salons" designed to frighten people with simple t

>Have we hit peak AI? AI is at the same point cinema was at when pioneers of the genre showed short films in "salons" designed to frighten people with simple tricks. This is hardly the peak. We're just at the beginning.

orwin 3y ago
AI development is different than most CS fields, as you need a solid mathematical base to build a "new discovery", and a lot of computation to use it efficiently. I'm 100% certain it's not the AI peak, but i'm also confident that we are close to the local maxima, at least for transformer architecture. Maybe the following improvement will be in chaining architecture, or a brand new technique, or maybe progress will be made in parallel fields (even smaller processors!). Charitably, i'll decide that the question meant "Have we hit peak GPT" and in this case, this is a legitimate question.
- unshavedyak 3y ago
  I'm most interested in refined training sets, refined compression, etc. Where do you suspect the cap is there? I say that because i find GPT4 extremely useful. Even if GPT4 (in it's hypothetical un-nerfed state some would argue)* is the peak LLM i think we have a ton of room to grow even within that peak. Locally running larger context windows, multiple domain experts in parallel and hooked up to a local repository of available IO actions would give a very significant "advancement" within the same conceptual "peak LLM". Now will we be able to get there on consumer hardware? Not idea. However i don't think i need more than GPT4 to find the whole technology extremely useful. Moderately cheap to run large context windows[1] entirely locally is the "next big thing" for me. I get a fair bit of value from GPT4 and Phind. The technology has sold me already. [1]: I should amend, i'm speaking loosely about context windows. What i mean is the ability to feed it refined datasets locally to have it's "local knowledge" change, which may be less about context windows and more about refined training. I'll leave that to the experts to debate, i am assuredly nothing close.
  - orwin 3y ago
    > I say that because i find GPT4 extremely useful. Even if GPT4 (in it's hypothetical un-nerfed state some would argue)* is the peak LLM i think we have a ton of room to grow even within that peak. Yes. sorry my point wasn't well formulated (i'm not an expert either: i've trained LLMs on receipts in 2016, and quickly decided i hated that). I'll try doing it better by dividing the two idea i put in that post: - i think that GPT4 is probably close the peak generalist GPT, meaning that maybe Bard can surpass it (or a chatGPT4.5 or something) but the improvement won't be stellar. - If future improvement won't come from the transformer architecture itself (i absolutely can be wrong on that), it'll come from what we interface with it. Absolutely context windows will probably a great way to have "specialists" transformers, but alos linking it with different "kind" of AI might push it too (i've read a post from Wolfram a while ago that sold me on interlinking different AI models/languages/state machine). I'm sure specialist or even large users haves tons of ideas. I got some value from GPT3.5, enough for me to buy access to GPT4. with a caveat: i'm pretty sure that for work-related questions i lost a bit more time than i "gained" overall. because my mastery of the technologies we use and very specific needs that GPT cannot know. But in the last three weeks or so, i'm pretty sure it was overall a net benefit at work, as i know better when i should use it.
- CuriouslyC 3y ago
  I don't think we'll need to completely replace the transformer, we just need better, more annotated data sets and a really good solution to large context windows. Data set annotation has a huge impact on model output, you can see this for yourself by comparing SDXL to Stable Diffusion 1.5, the model is larger but the real juice comes from using aesthetics scores for images during training to bias output towards things that look good - compare it to other large models like DALL-E or Imagen and you'll see parameter size is not the most important thing. With larger context windows, we can use traditional search algorithms to cherry pick an educational context for the LLM for few shot learning. If that's paired with more, better and annotated data, that will take us quite far.
YeGoblynQueenne 3y ago
We're not at the beginning. AI research started in 1956 with the Dartmouth workshop [1]. We've already had several cycles of boom and bust and at least two AI winters already [2]. Btw, there was a sci-fi novel where the plot took place on a planet where winter came in periods too long for the natives of the planet to remember (they had short lifespans, or they had limited memories, I er, why, I can't remember!). I don't remember the title of the novel or the author and I've been trying to find it. I thought it was something by Ursula K. Le Guin, but it doesn't sound like it. And no, it's not Game of Thrones. I think GoT took the idea of long-coming winters from the earlier novel, or it's just a coincidence (or cryptomnesia). Anyway I'd appreciate if someone knows what I'm talking about and lets me know. ___________ [1] https://en.wikipedia.org/wiki/Dartmouth_workshop https://en.wikipedia.org/wiki/Dartmouth_workshop [2] https://en.wikipedia.org/wiki/AI_winter https://en.wikipedia.org/wiki/AI_winter
- moffkalast 3y ago
  We're not at the beginning, but we're not close to the end either. The fact that organic brains exist proves the baseline of what's fundamentally possible and at very low power levels too if optimized well. We haven't reached that yet, and it's impossible to say what the upper limit is. LLMs might not get there as an end-to-end solution but they'll will certainly be a significant part of one.
- dri_ft 3y ago
  >a planet where winter came in periods too long for the natives of the planet to remember Not Asimov's Nightfall?
  - YeGoblynQueenne 3y ago
    Nope. Winter, not night-time :)
- grumbel 3y ago
  BingChat says it's Helliconia[1] by Brian W. Aldiss. Nightfall and the movie Pitch Black also go in that direction. [1] https://en.wikipedia.org/wiki/Helliconia https://en.wikipedia.org/wiki/Helliconia
rootusrootus 3y ago
I don't know, maybe this is the peak of this kind of AI. It seems like a lot of what drives this boom in LLMs isn't fundamental improvements in the technology, but rather the easy availability of enough computing power to do them. It's plausible that we're going to hit a wall abruptly and that the next step towards true AI will be down a different path.

orwin 3y ago

AI development is different than most CS fields, as you need a solid mathematical base to build a "new discovery", and a lot of computation to use it efficiently. I'm 100% certain it's not the AI peak, but i'm also confident that we are close to the local maxima, at least for transformer architecture. Maybe the following improvement will be in chaining architecture, or a brand new technique, or maybe progress will be made in parallel fields (even smaller processors!). Charitably, i'll decide that the question meant "Have we hit peak GPT" and in this case, this is a legitimate question.

unshavedyak 3y ago
I'm most interested in refined training sets, refined compression, etc. Where do you suspect the cap is there? I say that because i find GPT4 extremely useful. Even if GPT4 (in it's hypothetical un-nerfed state some would argue)* is the peak LLM i think we have a ton of room to grow even within that peak. Locally running larger context windows, multiple domain experts in parallel and hooked up to a local repository of available IO actions would give a very significant "advancement" within the same conceptual "peak LLM". Now will we be able to get there on consumer hardware? Not idea. However i don't think i need more than GPT4 to find the whole technology extremely useful. Moderately cheap to run large context windows[1] entirely locally is the "next big thing" for me. I get a fair bit of value from GPT4 and Phind. The technology has sold me already. [1]: I should amend, i'm speaking loosely about context windows. What i mean is the ability to feed it refined datasets locally to have it's "local knowledge" change, which may be less about context windows and more about refined training. I'll leave that to the experts to debate, i am assuredly nothing close.
- orwin 3y ago
  > I say that because i find GPT4 extremely useful. Even if GPT4 (in it's hypothetical un-nerfed state some would argue)* is the peak LLM i think we have a ton of room to grow even within that peak. Yes. sorry my point wasn't well formulated (i'm not an expert either: i've trained LLMs on receipts in 2016, and quickly decided i hated that). I'll try doing it better by dividing the two idea i put in that post: - i think that GPT4 is probably close the peak generalist GPT, meaning that maybe Bard can surpass it (or a chatGPT4.5 or something) but the improvement won't be stellar. - If future improvement won't come from the transformer architecture itself (i absolutely can be wrong on that), it'll come from what we interface with it. Absolutely context windows will probably a great way to have "specialists" transformers, but alos linking it with different "kind" of AI might push it too (i've read a post from Wolfram a while ago that sold me on interlinking different AI models/languages/state machine). I'm sure specialist or even large users haves tons of ideas. I got some value from GPT3.5, enough for me to buy access to GPT4. with a caveat: i'm pretty sure that for work-related questions i lost a bit more time than i "gained" overall. because my mastery of the technologies we use and very specific needs that GPT cannot know. But in the last three weeks or so, i'm pretty sure it was overall a net benefit at work, as i know better when i should use it.
CuriouslyC 3y ago
I don't think we'll need to completely replace the transformer, we just need better, more annotated data sets and a really good solution to large context windows. Data set annotation has a huge impact on model output, you can see this for yourself by comparing SDXL to Stable Diffusion 1.5, the model is larger but the real juice comes from using aesthetics scores for images during training to bias output towards things that look good - compare it to other large models like DALL-E or Imagen and you'll see parameter size is not the most important thing. With larger context windows, we can use traditional search algorithms to cherry pick an educational context for the LLM for few shot learning. If that's paired with more, better and annotated data, that will take us quite far.

YeGoblynQueenne 3y ago

We're not at the beginning. AI research started in 1956 with the Dartmouth workshop [1]. We've already had several cycles of boom and bust and at least two AI winters already [2]. Btw, there was a sci-fi novel where the plot took place on a planet where winter came in periods too long for the natives of the planet to remember (they had short lifespans, or they had limited memories, I er, why, I can't remember!). I don't remember the title of the novel or the author and I've been trying to find it. I thought it was something by Ursula K. Le Guin, but it doesn't sound like it. And no, it's not Game of Thrones. I think GoT took the idea of long-coming winters from the earlier novel, or it's just a coincidence (or cryptomnesia). Anyway I'd appreciate if someone knows what I'm talking about and lets me know. ___________ [1] https://en.wikipedia.org/wiki/Dartmouth_workshop https://en.wikipedia.org/wiki/Dartmouth_workshop [2] https://en.wikipedia.org/wiki/AI_winter https://en.wikipedia.org/wiki/AI_winter

moffkalast 3y ago
We're not at the beginning, but we're not close to the end either. The fact that organic brains exist proves the baseline of what's fundamentally possible and at very low power levels too if optimized well. We haven't reached that yet, and it's impossible to say what the upper limit is. LLMs might not get there as an end-to-end solution but they'll will certainly be a significant part of one.
dri_ft 3y ago
>a planet where winter came in periods too long for the natives of the planet to remember Not Asimov's Nightfall?
- YeGoblynQueenne 3y ago
  Nope. Winter, not night-time :)
grumbel 3y ago
BingChat says it's Helliconia[1] by Brian W. Aldiss. Nightfall and the movie Pitch Black also go in that direction. [1] https://en.wikipedia.org/wiki/Helliconia https://en.wikipedia.org/wiki/Helliconia

rootusrootus 3y ago

I don't know, maybe this is the peak of this kind of AI. It seems like a lot of what drives this boom in LLMs isn't fundamental improvements in the technology, but rather the easy availability of enough computing power to do them. It's plausible that we're going to hit a wall abruptly and that the next step towards true AI will be down a different path.