4 ms·

> mainly that such a fundamentally simple technique is paying off with such crazy dividends I will concede that I am amazed by this. Even more generally, I'm e

• ogogmad 4y ago
  [edit: Was snark. Sorry. I also may have misunderstood the parent comment.]
  • dvt 4y ago
    What a weirdly snarky comment (it's like your third one in this thread, maybe you need to take a breather). I wasn't comparing them, though there's an obvious stochastic argument to be made; I was just commenting on the former's lack of complexity and still fascinatingly good results, I'm not sure what your point is.
    • krackers 4y ago
      Markov chains are actually not a bad comparison from what i can see as an utter layman. At a high level seems like the difference between transformer-models and markov chains is that instead of the next token being determined purely based on n-gram frequencies on the training corpus, you have the attention mechanism so in some sense it has access to the entire previous context as well as some latent space of learned embeddings. But at the end of the day it still is just a fancy way to predict the next token, so that's why it's surprising you get all this neat behavior.