2 ms·

Once upon a time, nay-sayers said that nobody could travel to the moon, regardless of what vehicle they used. They were wrong. Once upon a time, nay-sayers said

Once upon a time, nay-sayers said that nobody could travel to the moon, regardless of what vehicle they used. They were wrong. Once upon a time, nay-sayers said that nobody could transmute lead into gold using alchemical equipment. They were right. Nay-sayers who said that no possible algorithm could beat humans at chess and go? They were wrong. Nay-sayers who say that these algorithms cannot write better books than humans? Well…

SEGyges 3y ago
By "these algorithms", do you mean the ones that currently exist, or the ones that will exist next month, next year, or in 2034?
- wizzwizz4 3y ago
  We're not developing new algorithms all that quickly. My point is that one shouldn't dismiss criticism out-of-hand, just because some critics of some other thing turned out to be wrong: for this point to be valid, I don't need to be making criticism. On an unrelated note… Personally, I'd be referring to the family of algorithms that purely take as input a context window and provide as output a prediction of the next token likelihood. (Plus or minus iteration, to generate strings of text.) Pejoratively, one might call these "fancy Markov chains", though as with most pejoratives, that's overly reductive. All the approaches we're seeing marketed heavily are just fancy Markov chains. I expect every "new algorithm" for the next 5 years at least to be a fancy Markov chain, because that's what I expect to get funding. (I do expect that some people will be working on other approaches, but only for amateurish reasons.)
  - oasisaimlessly 3y ago
    Strictly applying the definition, the entire universe is a Markov chain (thanks to quantum discretization!) People who use "Markov chain" as a pejorative are just idiots.
  - SEGyges 3y ago
    These are fancy Markov chains in the sense that humans are just chemicals and computers just do math. Technically true, but not even "overly reductive"; it is just wrong if it is used to imply that, e.g., humans just swirl around in beakers or the most complex thing you can do with computers is trigonometry. You can make anything sound unimpressive if you describe it sufficiently poorly. And: So many different variations are published every month. There are a good number of people in serious research trying approaches that don't use cross entropy loss (ie, strictly next-token prediction). I don't know what the trajectory of the technology is over the next ten years, but I am positive no one else does either and anyone who thinks they do is wrong.
ben_w 3y ago
> Once upon a time, nay-sayers said that nobody could transmute lead into gold using alchemical equipment. They were right. Now I'm wondering if, with modern knowledge, you could build a 0.5 MeV heavy ion accelerator with only the things available to a medieval alchemist. I'm thinking probably yes? Triboelectics can get the right voltage. But how good does the vacuum need to be? > Nay-sayers who say that these algorithms cannot write better books than humans? They may be right or wrong in the specific, but I think they're asking the wrong question, too specific.

SEGyges 3y ago

By "these algorithms", do you mean the ones that currently exist, or the ones that will exist next month, next year, or in 2034?

wizzwizz4 3y ago
We're not developing new algorithms all that quickly. My point is that one shouldn't dismiss criticism out-of-hand, just because some critics of some other thing turned out to be wrong: for this point to be valid, I don't need to be making criticism. On an unrelated note… Personally, I'd be referring to the family of algorithms that purely take as input a context window and provide as output a prediction of the next token likelihood. (Plus or minus iteration, to generate strings of text.) Pejoratively, one might call these "fancy Markov chains", though as with most pejoratives, that's overly reductive. All the approaches we're seeing marketed heavily are just fancy Markov chains. I expect every "new algorithm" for the next 5 years at least to be a fancy Markov chain, because that's what I expect to get funding. (I do expect that some people will be working on other approaches, but only for amateurish reasons.)
- oasisaimlessly 3y ago
  Strictly applying the definition, the entire universe is a Markov chain (thanks to quantum discretization!) People who use "Markov chain" as a pejorative are just idiots.
- SEGyges 3y ago
  These are fancy Markov chains in the sense that humans are just chemicals and computers just do math. Technically true, but not even "overly reductive"; it is just wrong if it is used to imply that, e.g., humans just swirl around in beakers or the most complex thing you can do with computers is trigonometry. You can make anything sound unimpressive if you describe it sufficiently poorly. And: So many different variations are published every month. There are a good number of people in serious research trying approaches that don't use cross entropy loss (ie, strictly next-token prediction). I don't know what the trajectory of the technology is over the next ten years, but I am positive no one else does either and anyone who thinks they do is wrong.

ben_w 3y ago

> Once upon a time, nay-sayers said that nobody could transmute lead into gold using alchemical equipment. They were right. Now I'm wondering if, with modern knowledge, you could build a 0.5 MeV heavy ion accelerator with only the things available to a medieval alchemist. I'm thinking probably yes? Triboelectics can get the right voltage. But how good does the vacuum need to be? > Nay-sayers who say that these algorithms cannot write better books than humans? They may be right or wrong in the specific, but I think they're asking the wrong question, too specific.