5 ms·

I don’t think there is any evidence at all that they meaningfully make fewer mistakes. In addition while humans make mistakes they will not casually make contr

I don’t think there is any evidence at all that they meaningfully make fewer mistakes. In addition while humans make mistakes they will not casually make contradictory claims the way GPT* do. Basic self-contradictions of the sort even a three year old would catch. Define how they are planning their answer in any sense that matters, please.

ftxbro 4y ago
> I don’t think there is any evidence at all that they meaningfully make fewer mistakes. I'm not sure I was clear, when I wrote "increasingly advanced GPT-N are making fewer [mistakes]" I didn't necessarily mean they were making fewer mistakes than humans, but rather that GPT N+1 makes fewer mistakes than GPT N. I assume this is pretty uncontroversial because for example the evidence in the test suites where bigger N generally get better scores meaning fewer mistakes. > In addition while humans make mistakes they will not casually make contradictory claims the way GPT* do. Basic self-contradictions of the sort even a three year old would catch. OK I believe you saw some of them. Do you think the contradictory claims imply that large language models need sensory grounding for meaning and understanding? If they are mistakes then presumably the more advanced models will have fewer mistakes, but this is only an extrapolation from the evidence of model progressions on test suites. The mistakes that 3 year olds would catch I would like to see, especially if GPT 4 fails them. I will be curious to see if GPT 5 will fail them also. If they are more like, taking opposing opinion stances in different query instances, then maybe it's not a problem that prevents meaning and understanding. Maybe it's just how the AI rolls. > Define how they are planning their answer in any sense that matters, please. Honestly idk what you expect me to put here. Probably you are saying that AIs that generate things word by word are by definition not planning, even if they might have a big context window. But like I was saying, people also speak word by word. Does it mean people aren't planning either? I mean, GPT-4 scores well on suites of puzzles. Presumably those suites involve some puzzles that require thinking about (planning) the solution to the puzzle and then writing the solution word by word. I thought of two simple ones from the top of my head. Probably you won't say they require any planning but I'm not sure. The first example I prompted gpt3.5 "Find two numbers you add together and get ten." and it put "Two numbers that add up to ten are 5 and 5." Does that answer require planning? Maybe or maybe not I guess it depends how you define it. The other one I made from the top of my head was to put the last line of a story first and have the gpt3.5 make a story that ends with that line. I prompted it like "Make a story that ends with the exact sentence 'And that's why we always plant the corn so early in the year.'" and it wrote a story. I don't know if you will say it required any planning or not. Probably not. I don't know what kind of evidence you would want. Anyway here is the story it made: Once upon a time, there was a farmer named John who lived in a small village. John was known for his exceptional farming skills and his ability to grow the best corn in the entire region. One year, John decided to experiment with planting his corn later in the season, hoping to yield a better harvest. As the days went by, John noticed that his corn was not growing as well as it usually did. The plants were smaller, and the ears of corn were not as big as they should be. John was worried that he had made a mistake by planting his corn later in the year. One day, John decided to visit his neighbor, who was also a farmer, to ask for advice. His neighbor told him that planting corn later in the season was not a good idea because the soil was not warm enough, and the corn would not have enough time to grow before the cold weather set in. John realized that he had made a mistake and decided to plant his corn early the following year. He prepared the soil, planted the seeds, and took care of the plants every day. As a result, his corn grew tall and strong, and he had a bountiful harvest. From that day on, John always planted his corn early in the year, and he never had a bad harvest again. He shared his story with other farmers in the village, and they all learned from his mistake. And that's why we always plant the corn so early in the year.

ftxbro 4y ago

> I don’t think there is any evidence at all that they meaningfully make fewer mistakes. I'm not sure I was clear, when I wrote "increasingly advanced GPT-N are making fewer [mistakes]" I didn't necessarily mean they were making fewer mistakes than humans, but rather that GPT N+1 makes fewer mistakes than GPT N. I assume this is pretty uncontroversial because for example the evidence in the test suites where bigger N generally get better scores meaning fewer mistakes. > In addition while humans make mistakes they will not casually make contradictory claims the way GPT* do. Basic self-contradictions of the sort even a three year old would catch. OK I believe you saw some of them. Do you think the contradictory claims imply that large language models need sensory grounding for meaning and understanding? If they are mistakes then presumably the more advanced models will have fewer mistakes, but this is only an extrapolation from the evidence of model progressions on test suites. The mistakes that 3 year olds would catch I would like to see, especially if GPT 4 fails them. I will be curious to see if GPT 5 will fail them also. If they are more like, taking opposing opinion stances in different query instances, then maybe it's not a problem that prevents meaning and understanding. Maybe it's just how the AI rolls. > Define how they are planning their answer in any sense that matters, please. Honestly idk what you expect me to put here. Probably you are saying that AIs that generate things word by word are by definition not planning, even if they might have a big context window. But like I was saying, people also speak word by word. Does it mean people aren't planning either? I mean, GPT-4 scores well on suites of puzzles. Presumably those suites involve some puzzles that require thinking about (planning) the solution to the puzzle and then writing the solution word by word. I thought of two simple ones from the top of my head. Probably you won't say they require any planning but I'm not sure. The first example I prompted gpt3.5 "Find two numbers you add together and get ten." and it put "Two numbers that add up to ten are 5 and 5." Does that answer require planning? Maybe or maybe not I guess it depends how you define it. The other one I made from the top of my head was to put the last line of a story first and have the gpt3.5 make a story that ends with that line. I prompted it like "Make a story that ends with the exact sentence 'And that's why we always plant the corn so early in the year.'" and it wrote a story. I don't know if you will say it required any planning or not. Probably not. I don't know what kind of evidence you would want. Anyway here is the story it made: Once upon a time, there was a farmer named John who lived in a small village. John was known for his exceptional farming skills and his ability to grow the best corn in the entire region. One year, John decided to experiment with planting his corn later in the season, hoping to yield a better harvest. As the days went by, John noticed that his corn was not growing as well as it usually did. The plants were smaller, and the ears of corn were not as big as they should be. John was worried that he had made a mistake by planting his corn later in the year. One day, John decided to visit his neighbor, who was also a farmer, to ask for advice. His neighbor told him that planting corn later in the season was not a good idea because the soil was not warm enough, and the corn would not have enough time to grow before the cold weather set in. John realized that he had made a mistake and decided to plant his corn early the following year. He prepared the soil, planted the seeds, and took care of the plants every day. As a result, his corn grew tall and strong, and he had a bountiful harvest. From that day on, John always planted his corn early in the year, and he never had a bad harvest again. He shared his story with other farmers in the village, and they all learned from his mistake. And that's why we always plant the corn so early in the year.