3 ms·

> imaginings of what it MIGHT become, but shows no signs of becoming Just for reference, this is from nine years ago: https://youtu.be/dbQh1I_uvjo https://you

> imaginings of what it MIGHT become, but shows no signs of becoming Just for reference, this is from nine years ago: https://youtu.be/dbQh1I_uvjo https://youtu.be/dbQh1I_uvjo And this is from five years ago: "You may not need hospitalization for respiratory problems for which there is pain or even full internal combustion. Breathing is particularly hard in the high-pitched high pitch, low pitched pitches. Air you inhale and exhale can cause a shortening in your breathing, so a ventilator is best. Doping can be administered on your own. If you fall, you must be at least 1 kilometer away from you to be considered for the testing. Your turn-of-the-seat's softerness may be a sign that something has gone wrong. Your depth should not exceed the mean of your seat height when wearing the seatbelt."

talldayo 2y ago
For one, Deepdream is cool but was never designed to be photorealistic. It was intentionally degraded to create psychadelic imagery, unlike the contemporary computer vision models we know today. And for two... LLMs are also getting to be pretty old. BERT and GPT-2 are both more than 5 years old, and still have fundamental issues that we can't be sure are solvable. They lie confidently, they conflate facts and fiction, they create entirely made-up scenarios when asked about real-world happenings. It should be extremely alarming that the only successful AI strategy to-date has been scaling-up an already inefficient model. AI will improve with time, but it seems entirely plausible to me that we've already hit the proverbial "bathtub curve" of progress. I genuinely cannot imagine what a "generational leap" in LLM technology would look like, outside of fixing the hallucination issues.
- throw310822 2y ago
  > LLMs are also getting to be pretty old. BERT and GPT-2 are both more than 5 years old You sound like you have no clue that five years are the blink of an eye.
  - talldayo 2y ago
    I'm someone that used both BERT and GPT-2 5 years ago, and am incredibly disappointed by how little progress the industry has made in that time. GPT-J and GPT-NEO are still arguably competitive with the latest AI models you can use today; that's sad. Cryptocurrency apologists also used this line of logic. "Yes, today the value of crypto is nothing... but imagine where it will be in five years!" Then we all wait 5 years, and cryptocurrency is still the victim of it's own mindset. Like cryptocurrency, I think LLMs have "technically" solved what they set out to do; generate readable text. But you need more than a solution in search of a problem; there isn't really that much demand for marginally truthful text in the same way completely decentralized currency isn't really necessary for the average well-meaning civilian. I'd even go further and argue that introducing AI into our daily lives requires you to replace something else, something that was probably more accurate and better-designed than the AI replacing it. If 5 years is a "blink of an eye" for the industry, the entire field will be dead within 18 months. VC just moves that fast.
    - gaganyaan 2y ago
      "Arguably competitive" is just plain wrong. I know it's fun to be dismissive of things, but making silly claims doesn't help comments like this read seriously.
      - talldayo 2y ago
        I would legitimately argue they are competitive. GPT-J and GPT-Neo are both fundamentally quite similar to modern LLMs, and when compared against other 4b and 8b parameter models I would genuinely champion the quality of their responses over their contemporaries.
        textninja 2y ago
        > when compared against other 4b and 8b parameter models I would genuinely champion the quality of their responses You clearly have some very specific models in mind. Even if the latest 4B and 8B models don’t move the needle on the “results you would champion” metric, this does not advance your argument that the state of the art hasn’t significantly progressed from 5 years ago. > I would legitimately argue I’ll bet you would!
- mdp2021 2y ago
  > cannot imagine what a "generational leap" in LLM technology would look like Well, if it encodes a world model, and can work on its own encoding, loop it into a critical revision of its contents and you have the Real Thing - something that is informed and reasons.
  - talldayo 2y ago
    Why should I expect a world model to be any more accurate than the current and horribly flawed text and vision modalities?
    - mdp2021 2y ago
      Because it is «loop[ed] into a critical revision of its contents»... That a full world model is there is the preliminary condition. So, first you have to establish that an LLM actually draws a world model in its inner structure. Then you have to refine that world model, and be able to refine its intentionally spawned branches (e.g. "Imagine scenario X"...). If the iteration of the refining is successful, you achieved the goal. The point is implementing critical thinking; that the initial world model is flawed is the known starting point: at the beginning, it only knows it has "heard" a lot of information.
      - talldayo 2y ago
        > that the initial world model is flawed is the known starting point: at the beginning, it only knows it has "heard" a lot of information. If it has no concrete reasoning, how is this looped revision intended to be more-accurate than a zero-shot guess? I agree with the fundamental principle of not trusting input data from the start, but that just puts us back in square-one where we generate non-authoritative results dependent on a series of autoregressive parameters. It might generate better-reasoned results, but the outcome will be equally unreliable and prone to hallucination.
        mdp2021 2y ago
        > If it has no concrete reasoning, how is this looped revision intended to be more-accurate It can work if the reasoning (the logic) is part of the world model and is similarly refined. Then, like in the normal case, you gather a good amount of information than draw your conclusions based on you logic and the "wisdom" built. Were this automated, you would have an automated champion in judgement - doing its best with the imperfect data and function available. Like us, just better.

talldayo 2y ago

For one, Deepdream is cool but was never designed to be photorealistic. It was intentionally degraded to create psychadelic imagery, unlike the contemporary computer vision models we know today. And for two... LLMs are also getting to be pretty old. BERT and GPT-2 are both more than 5 years old, and still have fundamental issues that we can't be sure are solvable. They lie confidently, they conflate facts and fiction, they create entirely made-up scenarios when asked about real-world happenings. It should be extremely alarming that the only successful AI strategy to-date has been scaling-up an already inefficient model. AI will improve with time, but it seems entirely plausible to me that we've already hit the proverbial "bathtub curve" of progress. I genuinely cannot imagine what a "generational leap" in LLM technology would look like, outside of fixing the hallucination issues.

throw310822 2y ago
> LLMs are also getting to be pretty old. BERT and GPT-2 are both more than 5 years old You sound like you have no clue that five years are the blink of an eye.
- talldayo 2y ago
  I'm someone that used both BERT and GPT-2 5 years ago, and am incredibly disappointed by how little progress the industry has made in that time. GPT-J and GPT-NEO are still arguably competitive with the latest AI models you can use today; that's sad. Cryptocurrency apologists also used this line of logic. "Yes, today the value of crypto is nothing... but imagine where it will be in five years!" Then we all wait 5 years, and cryptocurrency is still the victim of it's own mindset. Like cryptocurrency, I think LLMs have "technically" solved what they set out to do; generate readable text. But you need more than a solution in search of a problem; there isn't really that much demand for marginally truthful text in the same way completely decentralized currency isn't really necessary for the average well-meaning civilian. I'd even go further and argue that introducing AI into our daily lives requires you to replace something else, something that was probably more accurate and better-designed than the AI replacing it. If 5 years is a "blink of an eye" for the industry, the entire field will be dead within 18 months. VC just moves that fast.
  - gaganyaan 2y ago
    "Arguably competitive" is just plain wrong. I know it's fun to be dismissive of things, but making silly claims doesn't help comments like this read seriously.
    - talldayo 2y ago
      I would legitimately argue they are competitive. GPT-J and GPT-Neo are both fundamentally quite similar to modern LLMs, and when compared against other 4b and 8b parameter models I would genuinely champion the quality of their responses over their contemporaries.
      - textninja 2y ago
        > when compared against other 4b and 8b parameter models I would genuinely champion the quality of their responses You clearly have some very specific models in mind. Even if the latest 4B and 8B models don’t move the needle on the “results you would champion” metric, this does not advance your argument that the state of the art hasn’t significantly progressed from 5 years ago. > I would legitimately argue I’ll bet you would!
mdp2021 2y ago
> cannot imagine what a "generational leap" in LLM technology would look like Well, if it encodes a world model, and can work on its own encoding, loop it into a critical revision of its contents and you have the Real Thing - something that is informed and reasons.
- talldayo 2y ago
  Why should I expect a world model to be any more accurate than the current and horribly flawed text and vision modalities?
  - mdp2021 2y ago
    Because it is «loop[ed] into a critical revision of its contents»... That a full world model is there is the preliminary condition. So, first you have to establish that an LLM actually draws a world model in its inner structure. Then you have to refine that world model, and be able to refine its intentionally spawned branches (e.g. "Imagine scenario X"...). If the iteration of the refining is successful, you achieved the goal. The point is implementing critical thinking; that the initial world model is flawed is the known starting point: at the beginning, it only knows it has "heard" a lot of information.
    - talldayo 2y ago
      > that the initial world model is flawed is the known starting point: at the beginning, it only knows it has "heard" a lot of information. If it has no concrete reasoning, how is this looped revision intended to be more-accurate than a zero-shot guess? I agree with the fundamental principle of not trusting input data from the start, but that just puts us back in square-one where we generate non-authoritative results dependent on a series of autoregressive parameters. It might generate better-reasoned results, but the outcome will be equally unreliable and prone to hallucination.
      - mdp2021 2y ago
        > If it has no concrete reasoning, how is this looped revision intended to be more-accurate It can work if the reasoning (the logic) is part of the world model and is similarly refined. Then, like in the normal case, you gather a good amount of information than draw your conclusions based on you logic and the "wisdom" built. Were this automated, you would have an automated champion in judgement - doing its best with the imperfect data and function available. Like us, just better.