5 ms·
That large language models are stochastic parrots is not a bias, it's a fact. You feed it a very large amount of structured data and then put up a question whic
by chx 4y ago
That large language models are stochastic parrots is not a bias, it's a fact. You feed it a very large amount of structured data and then put up a question which it'll answer with a series of those which are most likely to follow. There is nothing else.
- tempestn 4y agoYeah. And in doing so, it will usually give you the correct answers.
- simonw 4y agoI know how they work. What's so interesting to me about them is how useful they turn out to be despite the seemingly dumb and simple way they are built. I encourage you to experiment more! Here are my notes so far: https://simonwillison.net/tags/llms/ https://simonwillison.net/tags/llms/
- p-e-w 4y agoDo you have any evidence that human minds work substantially different from that?
- chx 4y agoYes. Human minds often craft sentences which contain words together which never occurred in the same sentence before, not even close: > There was a boy called Eustace Clarence Scrubb, and he almost deserved it > Everything starts somewhere, although many physicists disagree. > It was a nice day. All the days had been nice. There had been rather more than seven of them so far, and rain hadn't been invented yet > My father had a face that could stop a clock. > It is important, when killing a nun, to ensure that you bring an army of sufficient size. > In the myriadic year of our Lord—the ten thousandth year of the King Undying, the kindly Prince of Death!— Gideon Nav packed her sword, her shoes, and her dirty magazines, and she escaped from the House of the Ninth. There's just no end to these. And, of course, science is full of these too, one that jumps to mind is Shinichi Mochizuki's claimed proof of the ABC conjecture which has been proven flawed as it often happens but it was certainly a credible proof despite written in a language no mathematician have ever seen.
- simonw 4y ago"Human minds often craft sentences which contain words together which never occurred in the same sentence before, not even close" Weirdly, LLMs do that too. Have you heard of the "Let’s be bear or bunny" pattern? My iPhone hallucinated that the other day, running a LLM directly on the phone! https://simonwillison.net/2023/May/1/lets-be-bear-or-bunny/ https://simonwillison.net/2023/May/1/lets-be-bear-or-bunny/
- chx 4y ago> Weirdly, LLMs do that too. Yes but those are as every other thing they emit, indeed , "Let’s be bear or bunny", I heard it called AI hallucination I personally like to call it word salad.
- pulvinar 4y agoIt's easy to cherry pick from the vast human repertoire. Here's a sample from GPT4, where I picked one from a total of two: > The man on the moon fell off his ladder one Tuesday, a common enough occurrence that nobody really paid it any mind. Truth is, gravity's always been a bit of a show-off, even in the star-dusted emptiness of space.