33 ms·

So it continues to reaffirm what we’ve known: generative LLM does not have a model of the world, can not reason and can not plan. It generates text by mix-matc

So it continues to reaffirm what we’ve known: generative LLM does not have a model of the world, can not reason and can not plan. It generates text by mix-matching remembered texts and thus it can not generate truly new content. No surprise because GPT-4 is built upon the same model as GPT-3. Clever Engineering will bring us far, but breakthrough requires change of the fundamentals. Nevertheless, it’s useful and can helps us solve problems when we guide it and split the work into many smaller subunits.

sinuhe69 4y ago
I copied the opinion of Yann LeCun, one of the authorities on deep learning: (Feb 13,2023) My unwavering opinion on current (auto-regressive) LLMs 1. They are useful as writing aids. 2. They are "reactive" & don't plan nor reason. 3. They make stuff up or retrieve stuff approximately. 4. That can be mitigated but not fixed by human feedback. 5. Better systems will come. 6. Current LLMs should be used as writing aids, not much more. 7. Marrying them with tools such as search engines is highly non trivial. 8. There will be better systems that are factual, non toxic, and controllable. They just won't be auto-regressive LLMs. 9. have been consistent with the above while defending Galactica as a scientific writing aid. 10. Warning folks that AR-LLMs make stuff up and should not be used to get factual advice. 11. Warning that only a small superficial portion of human knowledge can ever be captured by LLMs. 12. Being clear that better system will be appearing, but they will be based on different principles. They will not be auto-regressive LLMs. 13. Why do LLMs appear much better at generating code than generating general text? Because, unlike the real world, the universe that a program manipulates (the state of the variables) is limited, discrete, deterministic, and fully observable. The real world is none of that. 14. Unlike what the most acerbic critics of Galactica have claimed - LLMs are being used as writing aids. - They will not destroy the fabric of society by causing the mindless masses to believe their made-up nonsense. - People will use them for what they are helpful with.
- dvt 4y ago
  Fantastic comment, saving this. It's clear that for AI to be AI, it needs what philosophers of language call a Knowledge Base and a few intrinsic axiomatic presuppositions that, no matter what happens, cannot be broken (kind of like the Pauli exclusion principle in real life).

sinuhe69 4y ago

I copied the opinion of Yann LeCun, one of the authorities on deep learning: (Feb 13,2023) My unwavering opinion on current (auto-regressive) LLMs 1. They are useful as writing aids. 2. They are "reactive" & don't plan nor reason. 3. They make stuff up or retrieve stuff approximately. 4. That can be mitigated but not fixed by human feedback. 5. Better systems will come. 6. Current LLMs should be used as writing aids, not much more. 7. Marrying them with tools such as search engines is highly non trivial. 8. There will be better systems that are factual, non toxic, and controllable. They just won't be auto-regressive LLMs. 9. have been consistent with the above while defending Galactica as a scientific writing aid. 10. Warning folks that AR-LLMs make stuff up and should not be used to get factual advice. 11. Warning that only a small superficial portion of human knowledge can ever be captured by LLMs. 12. Being clear that better system will be appearing, but they will be based on different principles. They will not be auto-regressive LLMs. 13. Why do LLMs appear much better at generating code than generating general text? Because, unlike the real world, the universe that a program manipulates (the state of the variables) is limited, discrete, deterministic, and fully observable. The real world is none of that. 14. Unlike what the most acerbic critics of Galactica have claimed - LLMs are being used as writing aids. - They will not destroy the fabric of society by causing the mindless masses to believe their made-up nonsense. - People will use them for what they are helpful with.

dvt 4y ago
Fantastic comment, saving this. It's clear that for AI to be AI, it needs what philosophers of language call a Knowledge Base and a few intrinsic axiomatic presuppositions that, no matter what happens, cannot be broken (kind of like the Pauli exclusion principle in real life).