6 ms·

Its not reasoning and solving problems. It's a statistical model. It gives the appearance of reasoning and solving problems, but the second you start asking it

Its not reasoning and solving problems. It's a statistical model. It gives the appearance of reasoning and solving problems, but the second you start asking it to solve problems that aren't in its training database it gives you nonsense. I'm surprising at how little actual testing of an LLM's ability to reason based on well-established hard problems in AI (like halting) is happening. Philosophy of mind has produced a number of logic problems that are designed to test the reasoning ability of an intelligence, and instead LLMs get tested on content that is explicitly in their training dataset.

radicalace 3y ago
[flagged]
EMM_386 3y ago
> but the second you start asking it to solve problems that aren't in its training database it gives you nonsense I would disagree. I use it for code and it comes up with unique solutions to my situations that certainly have not been seen before in its training dataset. When you inform it you are using a 3rd party UI component library, a 3rd party CSS library, and you need to alter the layout based on the current complex state of an Angular system, and it can provide HTML, CSS, and TypeScript code that works and is based on this unique arrangement in this system, I'd say that's reasoning on some level. It's not copying and pasting from StackOverflow. It has to "understand" the scenario and implement it using my guidelines. It will even take into account things like linting rules ("Never use the 'any' type") and it can do that too.
- worrycue 3y ago
  > I use it for code and it comes up with unique solutions to my situations that certainly have not been seen before in its training dataset. How do you know something similar isn't in its dataset? I think we humans are a lot less original that we believe.
- flimsypremise 3y ago
  It is quite literally copying and pasting from StackOverflow, and any other code that it has been trained on. I think you're underestimating exactly how much code there is out there, and the degree to which most of it is boilerplate. Here's an example of what it can't do. I was recently messing around with the Plaid API, building sqlalchemy models for the Plaid API objects and defining relationships between them. After about 45 minutes of playing around, I got about 80% of the way to what I created without ChatGPT. It saved me a bunch of time googling around for documentation, but it gave me an absolute garbage password encryption implementation that was literally copied out of an intro tutorial I had read elsewhere, didn't even seem to recognize that there is a Password field type in sqlalchemy_utils, and omitted a bunch of key aspects of the model relationships. It also used field arguments that were either deprecated or not best practice for what I wanted to do. I also had to ask it to model every object individually, even though I specifically asked it to model all of the fields necessary to implement transactions. Now I could go and ask it to specifically modify those things, but it doesn't "know" the best way of building this set of models. It gives me the statistically determined output of what it has consumed from all of the answered questions on stackexchange and tutorials it has been fed, and provides a halfway decent starting point, but to get from there to a working application requires actually understanding why you might build a model one way rather than another. You need to understand exactly what behavior you want to see when you delete a user, and what needs to happen to all of the related objects. You need to know that id columns should be UUIDs and not integers or strings, which is something you commonly see in tutorials but you would (hopefully) never see in a production application. All of this knowledge is the result of years of experience building applications and knowing not only what to do, but why you do it and in what context. ChatGPT is very useful reference library. It blows google searching for examples out of the water, but you need to be very careful with the results, and very knowledgeable to use them properly.
  - EMM_386 3y ago
    > It is quite literally copying and pasting from StackOverflow Not necessarily. They've already said that certain emergent properties as the models have scaled beyond certain thresholds are responsible for some of the abilities. The ability to code as well as some of them can has been included in this. > ChatGPT is very useful reference library. It blows google searching for examples out of the water, but you need to be very careful with the results, and very knowledgeable to use them properly. I agree that you need to know what you're doing to make use of them. I've been a SWE for over two decades, so I certainly know what to feed them (and what not to - such as proprietary information) to get a coherent answer. Yes, they need a lot of context on a given issue, without that the model can only guess at what you are looking for (variable names, UUIDs for primary keys, schemas, interface definitions, 3rd party components, etc). And they will occasionally suggest using deprecated methods or not the latest suggested approach, due to the cutoff date. However, I stand by the opinion that if given the right inputs, they are capable of unique solutions to complex issues. The models are seemingly capable of taking points A and B from one StackOverflow post and combining that with C from the language documentation and D from a third-party vendors site and combining them into a coherent answer.
gpderetta 3y ago
> Its not reasoning and solving problems. It's a statistical model. One does not exclude the other.
cwkoss 3y ago
You brain is largely just a statistical model.
thomasfromcdnjs 3y ago
I'm a bit more open minded to what "reason" or our own understanding of our own "reason" might be. I gave GPT-4 a bunch of code that had two regex functions to parse a given custom url. I also gave it in the same prompt an example of a url of what I wanted it to parse, and asked it to explain the code and tell me the answer. It explained the code perfectly in moments and also gave me the correct answer. Here was it's response -> https://pastebin.com/raw/CP9iZh7S https://pastebin.com/raw/CP9iZh7S I would use this as an interview question, asking a junior dev to "reason" through the code, explain their thinking and without running it, tell me what they might think the answer would be. If I had to give them feedback for that answer, I would have commended their ability to "reason" about the problem. ¯\_(ツ)_/¯ (extra: The GPT-4 response knew to not bother with the second regex because it should only run the second if the first regex had a third part match, amazing...) (side note: I can't wait until I can generate live documentation of every file in my code base)
- worrycue 3y ago
  BTW have you asked it about " davidjl"? Maybe ask it about SolidGoldMagikarp too.
- flimsypremise 3y ago
  I'm not surprised it can write that regex and explain it, because that's a pretty simple regex and regexes generally are very well documented. Here's something it can't do: I was writing a python script that took a list of tuples with x elements. Each element in the tuple represented a value in a dataset, so the first dataset would be the list of elements at position 0 in each tuple, the second dataset would be the list of elements in positions 1 etc. I wanted to extract each dataset into a list, calculate the derivate of each element in the list, and then reassemble the original list of tuples with the derivates in place of the original values. I tried to get ChatGPT to give me a one liner to do this for _hours_. It would generally get step one and two correct, splitting the list of tuples of x length into x lists, and then calculating the derivative of each element in the list, but inevitably at the last step it would do something screwy. Most commonly it would reassembly a list of tuples, except the elements of each tuple would be from the indexes in the _second_ list. So I would get a list of all the derivates in position one in every dataset. Even if I explained what was wrong with the output, it would apologize and then blithely give me the exact same answer! Why did it do this? Because it doesn't actually understand the prompt, and the answer to my question is not in its dataset in a way that can be statistically derived.
jackmott42 3y ago
"you are not reasoning you are just neurons firing in a big network" "its not reasoning its just flipping bits" "its cant be alive its just nand gates"
Closi 3y ago
This general comment of "It can't handle things outside it's training database" is very common and also entirely wrong IMO. Take the following question which I just wrote and I'm fairly certain hasn't been asked before: > How many giraffes would it take to fly a plane in an imaginary world, assuming that the plane was designed to be flown by a giraffe. Assume that giraffe brains can be attached together to create a human-like intelligence assuming enough giraffes are connected, and assume that there are 500 passengers that will need to be served food (e.g. as per airline staff). Assume that the full airport is also staffed by giraffes. > Use logic and reasoning to estimate the number of giraffes that would be required, and detail specific practical considerations for the design of the aircraft. That's not been asked before - I just made it up. GPT-4 gives as good as an answer to this as I can come up with: > This is an interesting thought experiment. It's important to note that it's highly speculative and relies on numerous assumptions about the cognitive capabilities of giraffes, their physical abilities, and the mechanics of designing and operating an aircraft specifically for them. However, assuming we are in this hypothetical scenario, let's break down some of the elements: > Flying the Plane: Normally, a commercial aircraft is operated by two pilots. The main brain functions involved in flying a plane include spatial awareness, understanding complex systems, decision making, problem-solving, and communication. The human brain has approximately 86 billion neurons. In contrast, a giraffe's brain has approximately 1.2 billion neurons. Therefore, to achieve the complexity and cognitive capacity of one human pilot, approximately 72 giraffes would be needed (86 divided by 1.2). Since there are typically two pilots on a plane, this gives us a rough estimate of 144 giraffes. > Cabin Crew: Flight attendants serve multiple purposes including ensuring the safety of passengers and providing service such as meals. The average ratio of flight attendants to passengers on long-haul flights is around 1:50. Therefore, for 500 passengers, we would require about 10 human-equivalent flight attendants. Assuming the same ratio of giraffe brains to human brains, this would mean we need about 720 giraffes (72 giraffes equivalent to 1 human multiplied by 10). > Airport Staff: This could include a wide range of roles such as baggage handlers, check-in staff, security personnel, air traffic controllers, etc. It's difficult to estimate the exact number as it can vary widely depending on the size and busyness of the airport. However, as an approximation, let's say a small airport requires a staff of 100 human equivalents. This would mean approximately 7200 giraffes. > Design of the Aircraft and Airport: The design of the aircraft and airport would need to be significantly altered to accommodate giraffes, considering their size, physical abilities, and requirements. The size of the cockpit and cabin would need to increase substantially to allow for multiple giraffes. The controls would also need to be adapted to be usable by giraffes, likely involving some sort of body harness or direct neural interface given giraffes lack opposable thumbs or similar gripping ability. The height of the plane and airport infrastructure would also need to be increased due to the giraffes' tall stature. These changes would certainly increase the weight and fuel requirements of the plane. > Adding the numbers up, we get a rough total of about 8064 giraffes to run this operation (144 for pilots, 720 for cabin crew, and 7200 for airport staff). Please note, however, that these are extremely rough estimates and in reality, the number could be significantly different based on the many factors and variables involved in such a hypothetical situation. > In this scenario, the number of giraffes needed is estimated based on an equivalence of the number of neurons in giraffe brains to those in human brains. However, it's critical to remember that this is a simplified way of comparing cognitive abilities. In reality, intelligence and cognitive function are not solely determined by the number of neurons, and they certainly don't scale linearly. The organization and connections within the brain, the type of neurons, and the role of other brain cells are also critical. This serves as an imaginative exploration rather than a scientifically or practically accurate prediction. I mean, it's not perfect but it's clearly formulating something there.
blovescoffee 3y ago
But what's the difference between reasoning etc. and giving the appearance of reasoning?
- mrandish 3y ago
  This is a key question. With the latest advanced LLMs we've created something which exceeds our pre-existing conceptual framework's scope leading us into Searle's 'Chinese Room' which philosophers of mind have been debating for decades.
  - flimsypremise 3y ago
    What's funny is that ChatGPT is quite literally a Chinese room.

radicalace 3y ago

[flagged]

EMM_386 3y ago

> but the second you start asking it to solve problems that aren't in its training database it gives you nonsense I would disagree. I use it for code and it comes up with unique solutions to my situations that certainly have not been seen before in its training dataset. When you inform it you are using a 3rd party UI component library, a 3rd party CSS library, and you need to alter the layout based on the current complex state of an Angular system, and it can provide HTML, CSS, and TypeScript code that works and is based on this unique arrangement in this system, I'd say that's reasoning on some level. It's not copying and pasting from StackOverflow. It has to "understand" the scenario and implement it using my guidelines. It will even take into account things like linting rules ("Never use the 'any' type") and it can do that too.

worrycue 3y ago
> I use it for code and it comes up with unique solutions to my situations that certainly have not been seen before in its training dataset. How do you know something similar isn't in its dataset? I think we humans are a lot less original that we believe.
flimsypremise 3y ago
It is quite literally copying and pasting from StackOverflow, and any other code that it has been trained on. I think you're underestimating exactly how much code there is out there, and the degree to which most of it is boilerplate. Here's an example of what it can't do. I was recently messing around with the Plaid API, building sqlalchemy models for the Plaid API objects and defining relationships between them. After about 45 minutes of playing around, I got about 80% of the way to what I created without ChatGPT. It saved me a bunch of time googling around for documentation, but it gave me an absolute garbage password encryption implementation that was literally copied out of an intro tutorial I had read elsewhere, didn't even seem to recognize that there is a Password field type in sqlalchemy_utils, and omitted a bunch of key aspects of the model relationships. It also used field arguments that were either deprecated or not best practice for what I wanted to do. I also had to ask it to model every object individually, even though I specifically asked it to model all of the fields necessary to implement transactions. Now I could go and ask it to specifically modify those things, but it doesn't "know" the best way of building this set of models. It gives me the statistically determined output of what it has consumed from all of the answered questions on stackexchange and tutorials it has been fed, and provides a halfway decent starting point, but to get from there to a working application requires actually understanding why you might build a model one way rather than another. You need to understand exactly what behavior you want to see when you delete a user, and what needs to happen to all of the related objects. You need to know that id columns should be UUIDs and not integers or strings, which is something you commonly see in tutorials but you would (hopefully) never see in a production application. All of this knowledge is the result of years of experience building applications and knowing not only what to do, but why you do it and in what context. ChatGPT is very useful reference library. It blows google searching for examples out of the water, but you need to be very careful with the results, and very knowledgeable to use them properly.
- EMM_386 3y ago
  > It is quite literally copying and pasting from StackOverflow Not necessarily. They've already said that certain emergent properties as the models have scaled beyond certain thresholds are responsible for some of the abilities. The ability to code as well as some of them can has been included in this. > ChatGPT is very useful reference library. It blows google searching for examples out of the water, but you need to be very careful with the results, and very knowledgeable to use them properly. I agree that you need to know what you're doing to make use of them. I've been a SWE for over two decades, so I certainly know what to feed them (and what not to - such as proprietary information) to get a coherent answer. Yes, they need a lot of context on a given issue, without that the model can only guess at what you are looking for (variable names, UUIDs for primary keys, schemas, interface definitions, 3rd party components, etc). And they will occasionally suggest using deprecated methods or not the latest suggested approach, due to the cutoff date. However, I stand by the opinion that if given the right inputs, they are capable of unique solutions to complex issues. The models are seemingly capable of taking points A and B from one StackOverflow post and combining that with C from the language documentation and D from a third-party vendors site and combining them into a coherent answer.

gpderetta 3y ago

> Its not reasoning and solving problems. It's a statistical model. One does not exclude the other.

cwkoss 3y ago

You brain is largely just a statistical model.

thomasfromcdnjs 3y ago

I'm a bit more open minded to what "reason" or our own understanding of our own "reason" might be. I gave GPT-4 a bunch of code that had two regex functions to parse a given custom url. I also gave it in the same prompt an example of a url of what I wanted it to parse, and asked it to explain the code and tell me the answer. It explained the code perfectly in moments and also gave me the correct answer. Here was it's response -> https://pastebin.com/raw/CP9iZh7S https://pastebin.com/raw/CP9iZh7S I would use this as an interview question, asking a junior dev to "reason" through the code, explain their thinking and without running it, tell me what they might think the answer would be. If I had to give them feedback for that answer, I would have commended their ability to "reason" about the problem. ¯\_(ツ)_/¯ (extra: The GPT-4 response knew to not bother with the second regex because it should only run the second if the first regex had a third part match, amazing...) (side note: I can't wait until I can generate live documentation of every file in my code base)

worrycue 3y ago
BTW have you asked it about " davidjl"? Maybe ask it about SolidGoldMagikarp too.
flimsypremise 3y ago
I'm not surprised it can write that regex and explain it, because that's a pretty simple regex and regexes generally are very well documented. Here's something it can't do: I was writing a python script that took a list of tuples with x elements. Each element in the tuple represented a value in a dataset, so the first dataset would be the list of elements at position 0 in each tuple, the second dataset would be the list of elements in positions 1 etc. I wanted to extract each dataset into a list, calculate the derivate of each element in the list, and then reassemble the original list of tuples with the derivates in place of the original values. I tried to get ChatGPT to give me a one liner to do this for _hours_. It would generally get step one and two correct, splitting the list of tuples of x length into x lists, and then calculating the derivative of each element in the list, but inevitably at the last step it would do something screwy. Most commonly it would reassembly a list of tuples, except the elements of each tuple would be from the indexes in the _second_ list. So I would get a list of all the derivates in position one in every dataset. Even if I explained what was wrong with the output, it would apologize and then blithely give me the exact same answer! Why did it do this? Because it doesn't actually understand the prompt, and the answer to my question is not in its dataset in a way that can be statistically derived.

jackmott42 3y ago

"you are not reasoning you are just neurons firing in a big network" "its not reasoning its just flipping bits" "its cant be alive its just nand gates"

Closi 3y ago

This general comment of "It can't handle things outside it's training database" is very common and also entirely wrong IMO. Take the following question which I just wrote and I'm fairly certain hasn't been asked before: > How many giraffes would it take to fly a plane in an imaginary world, assuming that the plane was designed to be flown by a giraffe. Assume that giraffe brains can be attached together to create a human-like intelligence assuming enough giraffes are connected, and assume that there are 500 passengers that will need to be served food (e.g. as per airline staff). Assume that the full airport is also staffed by giraffes. > Use logic and reasoning to estimate the number of giraffes that would be required, and detail specific practical considerations for the design of the aircraft. That's not been asked before - I just made it up. GPT-4 gives as good as an answer to this as I can come up with: > This is an interesting thought experiment. It's important to note that it's highly speculative and relies on numerous assumptions about the cognitive capabilities of giraffes, their physical abilities, and the mechanics of designing and operating an aircraft specifically for them. However, assuming we are in this hypothetical scenario, let's break down some of the elements: > Flying the Plane: Normally, a commercial aircraft is operated by two pilots. The main brain functions involved in flying a plane include spatial awareness, understanding complex systems, decision making, problem-solving, and communication. The human brain has approximately 86 billion neurons. In contrast, a giraffe's brain has approximately 1.2 billion neurons. Therefore, to achieve the complexity and cognitive capacity of one human pilot, approximately 72 giraffes would be needed (86 divided by 1.2). Since there are typically two pilots on a plane, this gives us a rough estimate of 144 giraffes. > Cabin Crew: Flight attendants serve multiple purposes including ensuring the safety of passengers and providing service such as meals. The average ratio of flight attendants to passengers on long-haul flights is around 1:50. Therefore, for 500 passengers, we would require about 10 human-equivalent flight attendants. Assuming the same ratio of giraffe brains to human brains, this would mean we need about 720 giraffes (72 giraffes equivalent to 1 human multiplied by 10). > Airport Staff: This could include a wide range of roles such as baggage handlers, check-in staff, security personnel, air traffic controllers, etc. It's difficult to estimate the exact number as it can vary widely depending on the size and busyness of the airport. However, as an approximation, let's say a small airport requires a staff of 100 human equivalents. This would mean approximately 7200 giraffes. > Design of the Aircraft and Airport: The design of the aircraft and airport would need to be significantly altered to accommodate giraffes, considering their size, physical abilities, and requirements. The size of the cockpit and cabin would need to increase substantially to allow for multiple giraffes. The controls would also need to be adapted to be usable by giraffes, likely involving some sort of body harness or direct neural interface given giraffes lack opposable thumbs or similar gripping ability. The height of the plane and airport infrastructure would also need to be increased due to the giraffes' tall stature. These changes would certainly increase the weight and fuel requirements of the plane. > Adding the numbers up, we get a rough total of about 8064 giraffes to run this operation (144 for pilots, 720 for cabin crew, and 7200 for airport staff). Please note, however, that these are extremely rough estimates and in reality, the number could be significantly different based on the many factors and variables involved in such a hypothetical situation. > In this scenario, the number of giraffes needed is estimated based on an equivalence of the number of neurons in giraffe brains to those in human brains. However, it's critical to remember that this is a simplified way of comparing cognitive abilities. In reality, intelligence and cognitive function are not solely determined by the number of neurons, and they certainly don't scale linearly. The organization and connections within the brain, the type of neurons, and the role of other brain cells are also critical. This serves as an imaginative exploration rather than a scientifically or practically accurate prediction. I mean, it's not perfect but it's clearly formulating something there.

blovescoffee 3y ago

But what's the difference between reasoning etc. and giving the appearance of reasoning?

mrandish 3y ago
This is a key question. With the latest advanced LLMs we've created something which exceeds our pre-existing conceptual framework's scope leading us into Searle's 'Chinese Room' which philosophers of mind have been debating for decades.
- flimsypremise 3y ago
  What's funny is that ChatGPT is quite literally a Chinese room.