4 ms·

I dunno. The people who I talk to who do seem to understand what's happening seem even more impressed than the average layperson (mainly that such a fundamental

I dunno. The people who I talk to who do seem to understand what's happening seem even more impressed than the average layperson (mainly that such a fundamentally simple technique is paying off with such crazy dividends).

dvt 4y ago
> mainly that such a fundamentally simple technique is paying off with such crazy dividends I will concede that I am amazed by this. Even more generally, I'm even more amazed by Markov chains: an even simpler technique that can generate crazy dividends as well.
- ogogmad 4y ago
  [edit: Was snark. Sorry. I also may have misunderstood the parent comment.]
  - dvt 4y ago
    What a weirdly snarky comment (it's like your third one in this thread, maybe you need to take a breather). I wasn't comparing them, though there's an obvious stochastic argument to be made; I was just commenting on the former's lack of complexity and still fascinatingly good results, I'm not sure what your point is.
    - krackers 4y ago
      Markov chains are actually not a bad comparison from what i can see as an utter layman. At a high level seems like the difference between transformer-models and markov chains is that instead of the next token being determined purely based on n-gram frequencies on the training corpus, you have the attention mechanism so in some sense it has access to the entire previous context as well as some latent space of learned embeddings. But at the end of the day it still is just a fancy way to predict the next token, so that's why it's surprising you get all this neat behavior.
int_19h 4y ago
I'm actually concerned that the results are so good with such fundamentally simple techniques. If what we do can be so closely simulated by something so primitive, how much do we actually use our vaunted intelligence, and how much of our society and its output is not driven by it?

4 ms·

I dunno. The people who I talk to who do seem to understand what's happening seem even more impressed than the average layperson (mainly that such a fundamental

by dwohnitmok 4y ago

dvt 4y ago
> mainly that such a fundamentally simple technique is paying off with such crazy dividends I will concede that I am amazed by this. Even more generally, I'm even more amazed by Markov chains: an even simpler technique that can generate crazy dividends as well.
- ogogmad 4y ago
  [edit: Was snark. Sorry. I also may have misunderstood the parent comment.]
  - dvt 4y ago
    What a weirdly snarky comment (it's like your third one in this thread, maybe you need to take a breather). I wasn't comparing them, though there's an obvious stochastic argument to be made; I was just commenting on the former's lack of complexity and still fascinatingly good results, I'm not sure what your point is.
    - krackers 4y ago
      Markov chains are actually not a bad comparison from what i can see as an utter layman. At a high level seems like the difference between transformer-models and markov chains is that instead of the next token being determined purely based on n-gram frequencies on the training corpus, you have the attention mechanism so in some sense it has access to the entire previous context as well as some latent space of learned embeddings. But at the end of the day it still is just a fancy way to predict the next token, so that's why it's surprising you get all this neat behavior.
int_19h 4y ago
I'm actually concerned that the results are so good with such fundamentally simple techniques. If what we do can be so closely simulated by something so primitive, how much do we actually use our vaunted intelligence, and how much of our society and its output is not driven by it?