3 ms·

"I thoughts LLMs were glorified Markov chain generators" "the code actually looked pretty good. Not perfect, but I just told the AI to fix things, and it did.

"I thoughts LLMs were glorified Markov chain generators" "the code actually looked pretty good. Not perfect, but I just told the AI to fix things, and it did. I was shocked." These two views are by no means mutually exclusive. I find LLMs extremely useful and still believe they are glorified Markov generators. The take away should be that that is all you need and humans likely are nothing more than that.

smallnix 1y ago
> humans likely are nothing more than that Relevant post: https://news.ycombinator.com/item?id=44089156 https://news.ycombinator.com/item?id=44089156
Flemlo 1y ago
The way the input doesn't match the output should imply that it's not just statistics. As soon as compression happens, optimization happens which can lead to rules/learning of principles which got feed by statistics.
- immibis 1y ago
  That's "just" more statistics though.
  - Flemlo 1y ago
    Are you good in math definitions or is this an opinion? For me a compressed model learning rules through statistics is not statistics anymore. Physic rules are not statistics.
    - immibis 1y ago
      Of course they are. Force has a strong correlation with mass times acceleration. Objects at rest have a high chance of being observed to remain at rest. And so on.
      - Flemlo 1y ago
        Statistic is not the same as constant equations.
        immibis 1y ago
        The equations were discovered by experiment and curve-fitting.
kentonv 1y ago
I suppose it's all a continuum and we can each have different opinions on what the threshold for "glorified markov generator" is. But there have been many cases in my experience where the LLM could not possibly have been simply pattern-matching to something it had seen before. It really did "understand" the meaning of the code by any definition that makes sense to me.
- palata 1y ago
  > It really did "understand" the meaning of the code by any definition that makes sense to me. I find it dangerous to say it "understands". People are fast to say it "is sentient by any definition that makes sense to them". Also, would we say that a compiler "understands" the meaning of the code?
bufferoverflow 1y ago
> I find LLMs extremely useful and still believe they are glorified Markov generators. Then you should be able to make a markov chain generator without deep neural nets, and it should be on the same level of performance as current LLMs. But we both know you can't.
- ronsor 1y ago
  You can, but it will require far more memory than any computer has.

smallnix 1y ago

> humans likely are nothing more than that Relevant post: https://news.ycombinator.com/item?id=44089156 https://news.ycombinator.com/item?id=44089156

Flemlo 1y ago

The way the input doesn't match the output should imply that it's not just statistics. As soon as compression happens, optimization happens which can lead to rules/learning of principles which got feed by statistics.

immibis 1y ago
That's "just" more statistics though.
- Flemlo 1y ago
  Are you good in math definitions or is this an opinion? For me a compressed model learning rules through statistics is not statistics anymore. Physic rules are not statistics.
  - immibis 1y ago
    Of course they are. Force has a strong correlation with mass times acceleration. Objects at rest have a high chance of being observed to remain at rest. And so on.
    - Flemlo 1y ago
      Statistic is not the same as constant equations.
      - immibis 1y ago
        The equations were discovered by experiment and curve-fitting.

kentonv 1y ago

I suppose it's all a continuum and we can each have different opinions on what the threshold for "glorified markov generator" is. But there have been many cases in my experience where the LLM could not possibly have been simply pattern-matching to something it had seen before. It really did "understand" the meaning of the code by any definition that makes sense to me.

palata 1y ago
> It really did "understand" the meaning of the code by any definition that makes sense to me. I find it dangerous to say it "understands". People are fast to say it "is sentient by any definition that makes sense to them". Also, would we say that a compiler "understands" the meaning of the code?

bufferoverflow 1y ago

> I find LLMs extremely useful and still believe they are glorified Markov generators. Then you should be able to make a markov chain generator without deep neural nets, and it should be on the same level of performance as current LLMs. But we both know you can't.

ronsor 1y ago
You can, but it will require far more memory than any computer has.