8 ms·

The Georgia Tech Knowledge Based AI course involved building a program to answer ravens progressive matrix questions. The course was offered in the online MS pr

The Georgia Tech Knowledge Based AI course involved building a program to answer ravens progressive matrix questions. The course was offered in the online MS program, so thousands of students have taken the course. The most impressive result I saw was one student who got nearly perfect results in about 25 lines of Python code. This may be a case where humans do well on the test, but you can do very well on the test without doing anything the way a human would. The fact that GPTs aren’t very good at the test isn’t probably evidence that they’re not really very smart, but it doesn’t really mean that if we fix them to do very well on the test that they’ve gotten any smarter.

mjburgess 3y ago
There are an infinite number of algorithms to compute A from Q, given a se tof (Q, A). Almost none, surely, are intelligent. These proxy measure of intelligence are just arguments from ignorance, "I don't know how the machine computed A from Q, therefore...". But of course some of us do know how the machine did it; we can quite easily describe the algorithm. It just turns out no one wants to because its really dumb. Esp. if the alg is, as in all ML, "start with billions/trillions of data points in the (Q, A) space; generate a compressed representation ZipQA; and for novel Q' find decompressed A located close to Q similar to Q'" There are no theories of intelligence which would label that intelligence. And let me say, most such "theories" are ad-hoc PR that are rigged to make whatever the latest gizmo "intelligent". Any plausible theory begins form the initial intuition, "intelligence is what you do when you dont know what you're doing".
- thfuran 3y ago
  But does the fact that we don't really know how human intelligence works actually suggest that it is categorically smarter?
  - mjburgess 3y ago
    That's a great argument and way of reversing the argument-from-ignorance line. That said, I think people who argue from ignorance suppose we don't know how AI works either. Since the admen selling it tell them that. We know exactly and precisely how AI works; we can fully explain it. What we don't know are circumstantial parts of the explanation (eg., what properties of the training data + alg gave rise to the specific weight w1=0.01). This is like knowing why the thermometer reads 21 deg C (since the motion of the molecules in the water, etc. etc.) -- but not knowing which molecules specifically bounced off it. This confusion about "what we dont know" allows the prophetic tech grifter class to prognosticate in their interest. Since "we dont know how AI works, then it might work such that i'll be very rich, so invest with me!" All this said, it's a great observation.
- andoando 3y ago
  I remember getting a lot of flak for saying a purely statistical framework is not going to achieve human level intelligence, but I still firmly believe that. I also believe the path forward is research in knowledge representation, and even now when I search for it, I can barely find anything interesting happening in the field. ML has gotten so much interest and hype because it’s produced fast practical results, but I think it’s going to reach a standstill without something fundamentally new.
  - mjburgess 3y ago
    I think this is effectively provable from extraordinarily plausible premises. 1. We want to infer A from Q. 2. Most A we dont know, or have no data for, or the data is *in the future*. 3. Most Q we cannot conceptualise accurately since we have no explanatory theory in which to phrase it or to provide measures of it. 4. All statistical approaches require knowing frequencies of (Q, A) pairs (by def.) 5. In the cases where there is a unique objective frequency of (Q,A) we often cannot know it (2, 3) 6. In most cases there is no unique objective frequency (eg., there is no single animal any given photograph corresponds to, nor any objective frequency of such association). So, conclusion: In most cases the statistical approach either necessarily fails (its about future data; its about non-objective associations; it's impossible to measure or obtain objective frequences); OR if it doesnt necessarily fail, fails in practice (it is to expensive, or otherwise impossible, to obtain the authoritative QA-frequency). Now, of course, if your grift is generating nice cartoons or stealing cheap copy from ebooks you can convince the audience in the magical power of associating text tokens. This, of course, should be ignored when addressing the bigger methodological questions.
  - dennisy 3y ago
    I do agree here! Bit of a tangent from the thread but what have been the most valuable advances in knowledge representation in the last 20 years? Any articles you could share would be lovely!
    - andoando 3y ago
      I'm not expert and I don't know anything unfortunately. It is something I have spent countless hours walking around my room and pondering myself though for the last 3-4 years. I think I have some interesting ideas and I would love to get a PhD studying it, if I ever get enough financial independence that I don't have to worry about money.
- mewpmewp2 3y ago
  What is more special about how human intelligence works? In the end we are all particles and it all could be trillions of data points very simplistically interacting with each other resulting in emergent behaviour and complex, intelligent results. We know how common molecules can interact with each other. Does this mean that anything built on top of them is not intelligent? Everything in life could just be "statistics".
  - couchand 3y ago
    > In the end we are all particles citation needed
    - ducttapecrown 3y ago
      Citation: particle physics
    - WhitneyLand 3y ago
      https://en.wikipedia.org/wiki/Physics https://en.wikipedia.org/wiki/Physics
      - couchand 3y ago
        Where in that article does it say that the self, that human consciousness, that a soul arises purely from the arrangement of particles in our bodies?
        tsimionescu 3y ago
        In the part where it says that all that there is in the universe that is perceivable at human levels of energy and scales is made out of particles in the standard model (dark matter doesn't interact in a way where it could influence us, and dark energy only has effects on extraordinarily large scales). All measurements and all experiments ever done with matter and fields confirm that it behaves according to the laws of quantum mechanics. Those laws leave absolutely no room for a self that is not an emergent phenomenon of some kind. They also don't leave room for something like a free will that allows "you" to control "your body" by your "will", which is what I assume you might mean by a soul. That is, they clearly show that me writing this reply could have been (in principle) foretold, or at least had a calculated probability.
- Legend2440 3y ago
  >There are no theories of intelligence which would label that intelligence. Actually, compression-is-intelligence has been a relatively popular theory for a couple decades. The Hutter prize from 2006 is based around that premise. The idea is that compression and abstraction are fundamentally the same operation. If you had a perfect compressor that could compress the digits of pi into the 10-line algorithm that created them, in a deep sense it has understood pi and could generate as many more digits as you want.
  - mjburgess 3y ago
    Compression as intelligence is "popular" only with people who don't study intelligence. The number of problems with it a vastly too many to list. Perhaps the most obvious is its a form of inductivism, as it supposes that you can build representations of the targets of the measure domain from the measurement domain itself. This is just the same as drawing lines around patterns of stars, calling them "star signs" and then believing they are actual objects. This has absolutely nothing to do with intelligence; and is simply a description of statistical AI. It is no coincidence that this "theory" is only even known in that community, let alone popular elsewhere. You cannot derive theories (explanations, concepts, understanding, representations) by induction (compression) across measurements. There's lots and lots of work on this, and it is beyond reproach -- for a popular introduction see the first three chapters of David Deutch's fabric of reality. "Compressionists" might say that representations of the target domain are "compressed" with respect to their measures insofar as they take up less space. This is quite a funnily ridiculous thing to say: of course they do. This has nothing to do with the process of compression... it is a feature only of there being an infinite number of ways to measure any given causal property. The role "compression" here plays, insofar as it is relevant at all, is a trivial an uninformative observation that "explanations are finite, and measures infinite" -- one is not a compression of the other.
    - andoando 3y ago
      I dont follow. What are "actual" objects? Everything we conceptualize is some abstraction of data. We the see the world in trained concepts. A "door" for example isn't a fundamental object of the universe. Its a collection of atoms or quarks or whatever you consider fundamental objects of reality, (or even a completely abstract object having some resemblance in space/time), but each part in itself means nothing. It is the collection of them in a certain configuration which we recognize as being a "door". It is precisely the "compression" of of bunch of data points into a simpler concept. The same goes for concepts like shapes, diagonal, alternating, etc. There are endless infinite patterns from which we learn to distinguish
epups 3y ago
How many libraries were included in these 25 lines of code?
- deleted 3y ago
  [deleted]
- samatman 3y ago
  At least one, obviously, and a rather large one at that. The point of the comment you replied to is that conventional CV software can recognize the patterns in tests like Ravens Progressive Matrices just fine, and a simple logic tree can then solve them, while the LLM approach is still struggling to get the same result. This a commonplace shortcoming of the current generation of LLMs: ironically, they often fail at tasks which computers can do perfectly using conventional software.
- cgearhart 3y ago
  Obviously “Python” was used…and then it used numpy because the image format in the assignment was numpy arrays. However, the 25 lines was basically “sum the rows and sum the columns then compare those vectors” or something like that. This wasn’t really a case of all the complexity being hidden in a dependency; it was a case of finding a very simple heuristic that made the problem trivial.
bee_rider 3y ago
Could a GPT write those 20 lines of code?
- tflol 3y ago
  maybe if those 20 lines of code are in a private github repo somewhere
AlienRobot 3y ago
What I find odd is the human obsession with making AI as smart as a human. In the early periods of AI image generation, you would see AI generate images so uncanny no human could make something like that. It was genuinely unique and otherhumanly, worth of being called art. We "fixed" that bug so we could make the world's most expensive face swap app.
- bongodongobob 3y ago
  Nonsense, you can still make weird/creative stuff. It seems like you're speaking about the marketing materials that are pushed out.
miguelrochefort 3y ago
CS7637 was a fun course! One can get pretty far (80% accuracy) by using xor masks and comparing centroids.

mjburgess 3y ago

There are an infinite number of algorithms to compute A from Q, given a se tof (Q, A). Almost none, surely, are intelligent. These proxy measure of intelligence are just arguments from ignorance, "I don't know how the machine computed A from Q, therefore...". But of course some of us do know how the machine did it; we can quite easily describe the algorithm. It just turns out no one wants to because its really dumb. Esp. if the alg is, as in all ML, "start with billions/trillions of data points in the (Q, A) space; generate a compressed representation ZipQA; and for novel Q' find decompressed A located close to Q similar to Q'" There are no theories of intelligence which would label that intelligence. And let me say, most such "theories" are ad-hoc PR that are rigged to make whatever the latest gizmo "intelligent". Any plausible theory begins form the initial intuition, "intelligence is what you do when you dont know what you're doing".

thfuran 3y ago
But does the fact that we don't really know how human intelligence works actually suggest that it is categorically smarter?
- mjburgess 3y ago
  That's a great argument and way of reversing the argument-from-ignorance line. That said, I think people who argue from ignorance suppose we don't know how AI works either. Since the admen selling it tell them that. We know exactly and precisely how AI works; we can fully explain it. What we don't know are circumstantial parts of the explanation (eg., what properties of the training data + alg gave rise to the specific weight w1=0.01). This is like knowing why the thermometer reads 21 deg C (since the motion of the molecules in the water, etc. etc.) -- but not knowing which molecules specifically bounced off it. This confusion about "what we dont know" allows the prophetic tech grifter class to prognosticate in their interest. Since "we dont know how AI works, then it might work such that i'll be very rich, so invest with me!" All this said, it's a great observation.
andoando 3y ago
I remember getting a lot of flak for saying a purely statistical framework is not going to achieve human level intelligence, but I still firmly believe that. I also believe the path forward is research in knowledge representation, and even now when I search for it, I can barely find anything interesting happening in the field. ML has gotten so much interest and hype because it’s produced fast practical results, but I think it’s going to reach a standstill without something fundamentally new.
- mjburgess 3y ago
  I think this is effectively provable from extraordinarily plausible premises. 1. We want to infer A from Q. 2. Most A we dont know, or have no data for, or the data is *in the future*. 3. Most Q we cannot conceptualise accurately since we have no explanatory theory in which to phrase it or to provide measures of it. 4. All statistical approaches require knowing frequencies of (Q, A) pairs (by def.) 5. In the cases where there is a unique objective frequency of (Q,A) we often cannot know it (2, 3) 6. In most cases there is no unique objective frequency (eg., there is no single animal any given photograph corresponds to, nor any objective frequency of such association). So, conclusion: In most cases the statistical approach either necessarily fails (its about future data; its about non-objective associations; it's impossible to measure or obtain objective frequences); OR if it doesnt necessarily fail, fails in practice (it is to expensive, or otherwise impossible, to obtain the authoritative QA-frequency). Now, of course, if your grift is generating nice cartoons or stealing cheap copy from ebooks you can convince the audience in the magical power of associating text tokens. This, of course, should be ignored when addressing the bigger methodological questions.
- dennisy 3y ago
  I do agree here! Bit of a tangent from the thread but what have been the most valuable advances in knowledge representation in the last 20 years? Any articles you could share would be lovely!
  - andoando 3y ago
    I'm not expert and I don't know anything unfortunately. It is something I have spent countless hours walking around my room and pondering myself though for the last 3-4 years. I think I have some interesting ideas and I would love to get a PhD studying it, if I ever get enough financial independence that I don't have to worry about money.
mewpmewp2 3y ago
What is more special about how human intelligence works? In the end we are all particles and it all could be trillions of data points very simplistically interacting with each other resulting in emergent behaviour and complex, intelligent results. We know how common molecules can interact with each other. Does this mean that anything built on top of them is not intelligent? Everything in life could just be "statistics".
- couchand 3y ago
  > In the end we are all particles citation needed
  - ducttapecrown 3y ago
    Citation: particle physics
  - WhitneyLand 3y ago
    https://en.wikipedia.org/wiki/Physics https://en.wikipedia.org/wiki/Physics
    - couchand 3y ago
      Where in that article does it say that the self, that human consciousness, that a soul arises purely from the arrangement of particles in our bodies?
      - tsimionescu 3y ago
        In the part where it says that all that there is in the universe that is perceivable at human levels of energy and scales is made out of particles in the standard model (dark matter doesn't interact in a way where it could influence us, and dark energy only has effects on extraordinarily large scales). All measurements and all experiments ever done with matter and fields confirm that it behaves according to the laws of quantum mechanics. Those laws leave absolutely no room for a self that is not an emergent phenomenon of some kind. They also don't leave room for something like a free will that allows "you" to control "your body" by your "will", which is what I assume you might mean by a soul. That is, they clearly show that me writing this reply could have been (in principle) foretold, or at least had a calculated probability.
Legend2440 3y ago
>There are no theories of intelligence which would label that intelligence. Actually, compression-is-intelligence has been a relatively popular theory for a couple decades. The Hutter prize from 2006 is based around that premise. The idea is that compression and abstraction are fundamentally the same operation. If you had a perfect compressor that could compress the digits of pi into the 10-line algorithm that created them, in a deep sense it has understood pi and could generate as many more digits as you want.
- mjburgess 3y ago
  Compression as intelligence is "popular" only with people who don't study intelligence. The number of problems with it a vastly too many to list. Perhaps the most obvious is its a form of inductivism, as it supposes that you can build representations of the targets of the measure domain from the measurement domain itself. This is just the same as drawing lines around patterns of stars, calling them "star signs" and then believing they are actual objects. This has absolutely nothing to do with intelligence; and is simply a description of statistical AI. It is no coincidence that this "theory" is only even known in that community, let alone popular elsewhere. You cannot derive theories (explanations, concepts, understanding, representations) by induction (compression) across measurements. There's lots and lots of work on this, and it is beyond reproach -- for a popular introduction see the first three chapters of David Deutch's fabric of reality. "Compressionists" might say that representations of the target domain are "compressed" with respect to their measures insofar as they take up less space. This is quite a funnily ridiculous thing to say: of course they do. This has nothing to do with the process of compression... it is a feature only of there being an infinite number of ways to measure any given causal property. The role "compression" here plays, insofar as it is relevant at all, is a trivial an uninformative observation that "explanations are finite, and measures infinite" -- one is not a compression of the other.
  - andoando 3y ago
    I dont follow. What are "actual" objects? Everything we conceptualize is some abstraction of data. We the see the world in trained concepts. A "door" for example isn't a fundamental object of the universe. Its a collection of atoms or quarks or whatever you consider fundamental objects of reality, (or even a completely abstract object having some resemblance in space/time), but each part in itself means nothing. It is the collection of them in a certain configuration which we recognize as being a "door". It is precisely the "compression" of of bunch of data points into a simpler concept. The same goes for concepts like shapes, diagonal, alternating, etc. There are endless infinite patterns from which we learn to distinguish

epups 3y ago

How many libraries were included in these 25 lines of code?

deleted 3y ago
[deleted]
samatman 3y ago
At least one, obviously, and a rather large one at that. The point of the comment you replied to is that conventional CV software can recognize the patterns in tests like Ravens Progressive Matrices just fine, and a simple logic tree can then solve them, while the LLM approach is still struggling to get the same result. This a commonplace shortcoming of the current generation of LLMs: ironically, they often fail at tasks which computers can do perfectly using conventional software.
cgearhart 3y ago
Obviously “Python” was used…and then it used numpy because the image format in the assignment was numpy arrays. However, the 25 lines was basically “sum the rows and sum the columns then compare those vectors” or something like that. This wasn’t really a case of all the complexity being hidden in a dependency; it was a case of finding a very simple heuristic that made the problem trivial.

bee_rider 3y ago

Could a GPT write those 20 lines of code?

tflol 3y ago
maybe if those 20 lines of code are in a private github repo somewhere

AlienRobot 3y ago

What I find odd is the human obsession with making AI as smart as a human. In the early periods of AI image generation, you would see AI generate images so uncanny no human could make something like that. It was genuinely unique and otherhumanly, worth of being called art. We "fixed" that bug so we could make the world's most expensive face swap app.

bongodongobob 3y ago
Nonsense, you can still make weird/creative stuff. It seems like you're speaking about the marketing materials that are pushed out.

miguelrochefort 3y ago

CS7637 was a fun course! One can get pretty far (80% accuracy) by using xor masks and comparing centroids.