4 ms·

The Bayes-optimal text predictor would have to implicitly model the generative process underlying texts, that is, human minds. So it’s not crazy to think this a

• pdimitar 3y ago
  Text is only the result of abstract reasoning, symbol logic and who knows what else. That's like doing a code review focusing on the build artifact and ignoring the source code.
  • canjobear 3y ago
    It's like doing billions of code reviews based on billions of build artifacts, so that you start to notice that certain patterns are more common than others, and start to be able to reverse-engineer the compiler and source language. The best predictive model for executable binaries would be a generative model over possible source codes and then a model of the compiler.
    • pdimitar 3y ago
      Well LLMs do even trillions of "reviews" and are still not much wiser than before. Clearly, pieces are missing. I find it interesting that people assume that number_of_samples correlates with intelligent_behaviour, for some reason. Your example would maybe work if an intelligent agent had access to many compilers and reverse-engineered code bases. And if the algorithms were 1000x better than today. Otherwise it can guess until the cold death of the Universe without making progress.
      • eru 3y ago
        > Well LLMs do even trillions of "reviews" and are still not much wiser than before. Huh? A trained LLM is much 'wiser' than one that's just freshly randomly initialised. You are right that they are still far from perfect, and we hope for future improvements in technology.