4 ms·

I tried asking ChatGPT easy but completely off-the-wall questions and it consistently got the answers surprisingly right. For example, I asked it "Which is more

by chc 4y ago

I tried asking ChatGPT easy but completely off-the-wall questions and it consistently got the answers surprisingly right. For example, I asked it "Which is more powerful, a DEC VAX or a GTX 1070?" and it both noted that this comparison was questionable, as the two are not used in the same context, and correctly answered that the GeForce has much more processing power than a VAX. There are no Google results that directly answer that question, and yet ChatGPT knew that was what it should say. So while it is obviously just guessing what to say, the idea that it is just regurgitating existing material seems wrong to me. It does some pretty impressive synthesis to come up with its answers.

jcranmer 4y ago
I wonder how much of that is because the way such questions are asked is so formulaic that it's possible to "guess" the response from that information. If you were to ask a truly nonsense question, like "Which is more powerful, a GTX 1070 or an F-16?", what would the response be?
- azeirah 4y ago
  "It's difficult to compare a graphics card like the GTX 1070 and an aircraft like the F-16 as they are designed for very different purposes and operate on completely different scales. The GTX 1070 is a powerful graphics card that is designed to render high-quality graphics for gaming, video editing, and other visual computing applications. It has impressive processing power and can handle complex graphics and visual effects with ease. On the other hand, the F-16 is a highly advanced fighter aircraft designed for air-to-air and air-to-ground combat missions. It is a complex machine that requires a highly skilled pilot to operate and is equipped with state-of-the-art avionics, weapons systems, and engines. In terms of raw power, the F-16 is undoubtedly more powerful than the GTX 1070, with its advanced capabilities in speed, agility, and firepower. However, it's important to note that these are two very different types of machines, and a direct comparison of their power is not really possible."
  - panarky 4y ago
    LLMs keep demonstrating that they have an internal "world model" representation of concepts and the relationships between them. Meanwhile ostensible "expert" humans continue to regurgitate obvious falsehoods like "it's just a fancy autocomplete" or "you're being fooled by a system that just predicts the most probable next word".
    - anticensor 4y ago
      That world model is an emergent phenomenon arisen out of training, they are not explicitly trained a world model.
      - DougBTX 4y ago
        I’d phrase it the other way around, learning a world model is necessary in order to make such good token predictions.
    - flir 4y ago
      I don't see why "internal model" and "predicts the most probable next word" are mutually exclusive.
      - panarky 4y ago
        Who said they're mutually exclusive?
    - Jensson 4y ago
      The internal "world model" are things like "GPU's are faster than CPU's", "Military hardware beats nonmilitary things" etc. Then when you ask the comparison question it categorises the items, and for the plane vs card you get military vs nonmilitary response. Or computer vs gpu you get the cpu vs gpu response or similar. I showed the example of how to reason about this in a post below. Anyway, while you can call that a "world model", that world model is very brittle since its just a bunch of heuristics, if you reverse engineer the heuristic it uses it is really easy to find cases where it fails completely. Like if you ask it to compare a modern integrated GPU to an old dedicated GPU, it will say that the old dedicated GPU is faster even though it is ancient and slow, since when people talk about dedicated vs integrated GPU they almost always say that the dedicated GPU is faster.
- ianmcgowan 4y ago
  Why wonder? "It's not appropriate to compare a graphics card (GTX 1070) to a military fighter jet (F-16) as they are completely different things designed for entirely different purposes. A GTX 1070 is a high-end graphics card designed for gaming and other graphics-intensive tasks. It has a lot of processing power and can handle complex visual computations quickly and efficiently. On the other hand, an F-16 is a military fighter jet designed for air-to-air and air-to-ground combat. It is equipped with advanced weapons systems, avionics, and other technologies that allow it to perform a wide range of military operations, including surveillance, reconnaissance, and combat missions. In short, while a GTX 1070 is a powerful graphics card, it is not designed for the same purposes as an F-16, which is a highly advanced and specialized military aircraft."
- deleted 4y ago
  [deleted]
Jensson 4y ago
Try asking if it is faster than a low end modern card, in a comparison it looks at comparisons people have made before and decide based on those, and most comparisons with 1070 the 1070 probably came up ahead since it is a relatively high end card. It probably hasn't seen many DEC VAX comparisons, so it would decide mostly based on the other part, and if that is low end likely it would say the DEC VAX is faster since it is so used to say that the other part is slower. Or it has a separate category in its state space for old computers and will say the same thing for all of those in these comparisons, but you can trip it up using the weakness I talked about above in most cases. Its just that the comparisons people come up with can often be answered correctly by that method since people like to compare "powerful thing to weak thing", and in those it just looks at how often each wins and decides based on that, instead of comparing them directly.
- chc 4y ago
  I tried doing the same thing with the Radeon R7 250 (an old GPU that was weak when it was introduced) and got essentially the same answer, so it doesn't seem like it's just regurgitating praise. What impressed me just as much about these answers, though, was that it correctly identified that the comparison was silly. I don't believe it has seen that exact comparison before, so it seems like it has to have a granular enough categorization of tokens to know that people would say a comparison between a GPU and a computer is silly, and that those are a GPU and computer, and that one is old and the other is new.
  - Jensson 4y ago
    GPU's typically compute faster than CPU's, so maybe that is the conversation it identifies it as? Meaning, when people ask about power difference between CPU and GPU they say they aren't directly comparable, but the GPU has much more processing power. That would mean it would identify an old GPU as faster than a modern CPU, even though it isn't, since it would slot that into the same conversation. The hard part with finding out the source for the logic is that it can map items to similar items in many ways, and there are trillions of conversations out there that it could map it to, so it can do quite a lot of heuristics that will work fairly well for naive questions even though it can't apply any direct logic. If it could link to similar conversations that it bases its reasoning on then that would probably greatly help understand where its logic would fail. So before it can do that we probably wont be able to eliminate these failure modes, because humans aren't random enough to generate the edge cases unless they understand what logic the model used. For example, comparing an old computer to a modern GPU might seem random to you, but to ChatGPT it just sees something like "This looks like a CPU vs GPU comparison, aha so I map it to: {CPU} isn't really comparable to {GPU}, but {GPU} has much more processing power". If you could see something like that then it would be obvious where it would fail, but without that it looks like magic.

by chc 4y ago

jcranmer 4y ago

I wonder how much of that is because the way such questions are asked is so formulaic that it's possible to "guess" the response from that information. If you were to ask a truly nonsense question, like "Which is more powerful, a GTX 1070 or an F-16?", what would the response be?

azeirah 4y ago
"It's difficult to compare a graphics card like the GTX 1070 and an aircraft like the F-16 as they are designed for very different purposes and operate on completely different scales. The GTX 1070 is a powerful graphics card that is designed to render high-quality graphics for gaming, video editing, and other visual computing applications. It has impressive processing power and can handle complex graphics and visual effects with ease. On the other hand, the F-16 is a highly advanced fighter aircraft designed for air-to-air and air-to-ground combat missions. It is a complex machine that requires a highly skilled pilot to operate and is equipped with state-of-the-art avionics, weapons systems, and engines. In terms of raw power, the F-16 is undoubtedly more powerful than the GTX 1070, with its advanced capabilities in speed, agility, and firepower. However, it's important to note that these are two very different types of machines, and a direct comparison of their power is not really possible."
- panarky 4y ago
  LLMs keep demonstrating that they have an internal "world model" representation of concepts and the relationships between them. Meanwhile ostensible "expert" humans continue to regurgitate obvious falsehoods like "it's just a fancy autocomplete" or "you're being fooled by a system that just predicts the most probable next word".
  - anticensor 4y ago
    That world model is an emergent phenomenon arisen out of training, they are not explicitly trained a world model.
    - DougBTX 4y ago
      I’d phrase it the other way around, learning a world model is necessary in order to make such good token predictions.
  - flir 4y ago
    I don't see why "internal model" and "predicts the most probable next word" are mutually exclusive.
    - panarky 4y ago
      Who said they're mutually exclusive?
  - Jensson 4y ago
    The internal "world model" are things like "GPU's are faster than CPU's", "Military hardware beats nonmilitary things" etc. Then when you ask the comparison question it categorises the items, and for the plane vs card you get military vs nonmilitary response. Or computer vs gpu you get the cpu vs gpu response or similar. I showed the example of how to reason about this in a post below. Anyway, while you can call that a "world model", that world model is very brittle since its just a bunch of heuristics, if you reverse engineer the heuristic it uses it is really easy to find cases where it fails completely. Like if you ask it to compare a modern integrated GPU to an old dedicated GPU, it will say that the old dedicated GPU is faster even though it is ancient and slow, since when people talk about dedicated vs integrated GPU they almost always say that the dedicated GPU is faster.
ianmcgowan 4y ago
Why wonder? "It's not appropriate to compare a graphics card (GTX 1070) to a military fighter jet (F-16) as they are completely different things designed for entirely different purposes. A GTX 1070 is a high-end graphics card designed for gaming and other graphics-intensive tasks. It has a lot of processing power and can handle complex visual computations quickly and efficiently. On the other hand, an F-16 is a military fighter jet designed for air-to-air and air-to-ground combat. It is equipped with advanced weapons systems, avionics, and other technologies that allow it to perform a wide range of military operations, including surveillance, reconnaissance, and combat missions. In short, while a GTX 1070 is a powerful graphics card, it is not designed for the same purposes as an F-16, which is a highly advanced and specialized military aircraft."
deleted 4y ago
[deleted]

Jensson 4y ago

Try asking if it is faster than a low end modern card, in a comparison it looks at comparisons people have made before and decide based on those, and most comparisons with 1070 the 1070 probably came up ahead since it is a relatively high end card. It probably hasn't seen many DEC VAX comparisons, so it would decide mostly based on the other part, and if that is low end likely it would say the DEC VAX is faster since it is so used to say that the other part is slower. Or it has a separate category in its state space for old computers and will say the same thing for all of those in these comparisons, but you can trip it up using the weakness I talked about above in most cases. Its just that the comparisons people come up with can often be answered correctly by that method since people like to compare "powerful thing to weak thing", and in those it just looks at how often each wins and decides based on that, instead of comparing them directly.

chc 4y ago
I tried doing the same thing with the Radeon R7 250 (an old GPU that was weak when it was introduced) and got essentially the same answer, so it doesn't seem like it's just regurgitating praise. What impressed me just as much about these answers, though, was that it correctly identified that the comparison was silly. I don't believe it has seen that exact comparison before, so it seems like it has to have a granular enough categorization of tokens to know that people would say a comparison between a GPU and a computer is silly, and that those are a GPU and computer, and that one is old and the other is new.
- Jensson 4y ago
  GPU's typically compute faster than CPU's, so maybe that is the conversation it identifies it as? Meaning, when people ask about power difference between CPU and GPU they say they aren't directly comparable, but the GPU has much more processing power. That would mean it would identify an old GPU as faster than a modern CPU, even though it isn't, since it would slot that into the same conversation. The hard part with finding out the source for the logic is that it can map items to similar items in many ways, and there are trillions of conversations out there that it could map it to, so it can do quite a lot of heuristics that will work fairly well for naive questions even though it can't apply any direct logic. If it could link to similar conversations that it bases its reasoning on then that would probably greatly help understand where its logic would fail. So before it can do that we probably wont be able to eliminate these failure modes, because humans aren't random enough to generate the edge cases unless they understand what logic the model used. For example, comparing an old computer to a modern GPU might seem random to you, but to ChatGPT it just sees something like "This looks like a CPU vs GPU comparison, aha so I map it to: {CPU} isn't really comparable to {GPU}, but {GPU} has much more processing power". If you could see something like that then it would be obvious where it would fail, but without that it looks like magic.