10 ms·

Compression is a component of general intelligence. A few years ago I was very sceptical of machine learning ever leading to general intelligence. I've since ch

• goaaron 5y ago
  The human brain also forgets, something that may be a feature instead of a bug. Also, beyond compression––brains are simulation machines: imagining new scenarios. Curios to understand if ML provides anything analogous to simulation that isn't rote interpolation.
  • nh23423fefe 5y ago
    I think the simulation aspects of conscious and intelligence are fundamental. We don't simulate the world, we simulate what we might experience.
    • loa_in_ 5y ago
      I don't think it's true. I can imagine a lot of aspects of systems around me I cannot possibly experience in any way, except maybe them leading to some outcome that I might experience as well. I sometimes do verify this experimentally, but that comes later.
      • nh23423fefe 5y ago
        I don't see how you are disagreeing with me? GP said something about compression being a component of intelligence, parent said also brains simulate, then I said yeah i agree, and believe that the content of the simulation is the experience itself, whereas people often think there are 2 things, themselves and the world. I don't believe that at all. There is only one thing happening, the experience you are having now. I can't follow what you are trying to say.
        • loa_in_ 5y ago
          With this definition of experience, sure. My definition of experience is observable, palpable, etc. portion of the world
          • nh23423fefe 5y ago
            Ah ok, that doesn't work for me because I want to call dreams and hallucinations experiences as well. And those are disconnected from some idea of reality. The primary aspect is first person perspective, not congruence with the external world.
  • Traubenfuchs 5y ago
    People with hyperthymesia don‘t forget and don‘t necessarily seem to have any other potentially disabling neuroatypicality like autism. Having it is a premium feature. https://en.m.wikipedia.org/wiki/Hyperthymesia https://en.m.wikipedia.org/wiki/Hyperthymesia
    • jamiek88 5y ago
      >In fact, she was not very good at memorizing anything at all, according to the study published in Neurocase.[1] Hyperthymestic individuals appear to have poorer than average memory for arbitrary information. So no, not premium. A trade off.
  • kaba0 5y ago
    I am quite a novice in ML topics, but isn’t this concept of simultaneously training a generator and validator sort of this? I don’t know the exact term but I think of deep fake generators with an accompanying deep fake recognizer working in tandem bettering each other constantly?
  • uoaei 5y ago
    Absolutely. Generative methods are all the rage now. Those methods work on learning information-rich representation spaces. You could argue it's still "interpolation" but instead of interpolating in data-space per se you are interpolating in representation-space.
• beaconstudios 5y ago
  This also ties in to the cybernetic concept of the law of requisite variety, where adaptable entities need to be able to compress their sense-data about their environment into an internal model that corresponds in complexity to their need to act - this necessarily involves compression as the totality of reality is effectively infinite and can't fit between your ears. There's also the Hutter prize that ties data compression directly to intelligence through Kolmagorov complexity. Information and cybernetic theories cut pretty close to a general theory of intelligence in my opinion!
  • captaincaveman 5y ago
    Which also seems a cornerstone of Fristons Active inference (from the little I understand of it).
• mherrmann 5y ago
  A nice definition of intelligence I've heard is exactly the ability to form models of the world with predictive power. And a model is essentially a compression of real-world data. Physical laws are a great example of this.
  • copperx 5y ago
    Creating models with predictive power is also a precise definition of science.
    • rektide 5y ago
      Instead of reasoned & formula based models, now we have purely empirical models. See Wolfram's New Kind Of Science.
      • jjoonathan 5y ago
        Does ANKS engage with empirical models beyond what is necessary to hype up cellular automata?
      • 8note 5y ago
        Isn't wolfram's new kind of science purely rational? No observations of the universe needed
        • mensetmanusman 5y ago
          The universe is required to run Mathematica.
          • fsckboy 5y ago
            not if the universe is running inside Mathematica (our mathematica doesn't have the space, but the one we are inside does)
        • imtringued 5y ago
          That's actually one of the flaws of Austrian economics. Mises just postulates that certain of his personality traits apply to all humans and ignores the personality traits of other humans. He is also confusing the informational theoretical world inside his mind with the harsh reality of the real world. The word "capital depreciation" kind of summarizes all sorts of flaws. Gold is considered an ideal currency yet it suffers from no capital depreciation at all, while all other forms of physical capital depreciate. Once you have built all the houses, roads and schools, etc there is nothing left to build. If you happen to have built too many schools, then you are a fool and should convert the school back to gold by demolishing it and selling the scrap or in other words, idle capital, that is left to depreciate, must be eliminated. When you think about this, this is secretly rewarding monopolies and all sorts of other nonsense that capitalism suffers from. For 10 people there are exactly 10 jobs. If a workaholic hoards a job by working 80 hours per week to retire early, then somehow there won't be enough "jobs" for everyone. This gives employers the upper hand during negotiations as employees start competing with each other instead of employers competing over employees. The current system is also dumb because it tries to represent capital depreciation through an increase in the price level i.e. 2% inflation targeting.
• bgroat 5y ago
  In the incredible story "Funes the Memorious" the eponymous Funes has an absolutely perfect memory, but is functionally mentally handicapped. He can't even abstract to the existence of "trees" because he can recall and diff all of the details of every tree he's ever seen. He can't even identify that he's seen a particular tree before, because he can diff how different it looked in a particular configuration of leaves and shadows because of different wind and cloud cover
  • tartakovsky 5y ago
    Makes me think of … Asperger’s.
    • BizarroLand 5y ago
      I would think it's being an megasavant, sort of like Kim Peek. https://en.wikipedia.org/wiki/Kim_Peek https://en.wikipedia.org/wiki/Kim_Peek Not capable of functioning independently or surviving for any long period unassisted but having a brain and cognition setup that allowed for amazing feats of mental wizardry. If you could have that ability and function normally in society you could do some astounding things.
      • bgroat 5y ago
        That's how I understood it. A great story about a mega-savant who can function is "Understand" by Ted Chiang, if you're interested
      • Lich 5y ago
        I thought that the idea of Mentats (human computers from the Dune novel) were kind of ridiculous, but yeah, when you look at savants like Peek, makes you kind of wonder if such a thing would be possible.
        • BizarroLand 5y ago
          Yeah, if you had the appropriate brain surgery as an infant and took highly specialized drugs to maintain or accelerate your mental functioning, you too could be like Kim Peek. I like that Frank Herbert imagined that it would be possible and went with it before we had any real proof that it could be, and then he turned out to be right.
• Joel1234 5y ago
  Que gracioso tio
  • deleted 5y ago
    [deleted]
• metamuas 5y ago
  I have always thought that the best measure of intelligence is compression of information. If you can create a smaller, abstract model that is still accurate despite a loss in details, then you are intelligent.
  • meowface 5y ago
    Interesting counterargument from AI researcher François Chollet (creator of Keras and one of the main contributors to TensorFlow): https://www.youtube.com/watch?v=-V-vOXLyKGw https://www.youtube.com/watch?v=-V-vOXLyKGw
• samstave 5y ago
  How do theories such as "The 100th Monkey" as well as transferred information via DNA to offspring translate to ML|AI at all? For example, couldn't a sufficiently developed AI modify some code/libraries it utilizes/learns from/creates, to ensure any new spawns of said AI/ML/Bot has the learned previous behaviors? I doubt 100th Monkey will ever hit AI. So that's an interesting aspect to the limits to AI 'evolving'
• samstave 5y ago
  >able to compress data into a _useful_ representation. Note that as a result of this, the quality of the compression is important. Some forms of compression might be very efficient but they also tangle concepts together, resulting in loss of composability --- I wonder if various factors inform how/what compression is used on a memory... For example, a memory of putting the object back where it belongs/got it from vs the memory of a violent attack is through the lens of emotional (trauma) and thus the memories will be stored differently. Its interesting in that I have been wanting to post an ASK HN on memory and dreams... Now with this post, and your comment, I will post that. --- The idea is that the surunding meta-information of a memory is important. Lenses of senses that colour a memory are many, and individualistic. i.e. A person who is a psychopath, has an emotional block on the lens that they would see their actions through (remorse, guilt, empathy, etc) - thus they may not recall or RE-MIND themselves of an action/situation. A memory that is laid with a sensuous experience, such as sex with someone you love/lust deeply may last a lifetime. Certain things that one does/says can also lead to a lifetime of regret ; a cringe-worthy action/comment from decades ago can still haunt your thoughts. --- I think the mystique btwn ML and biological memories is a really interesting space, as an ML|AI based system will never achieve the 100th monkey or DNA|biological transfer of information, but an approximation/facsimile based on evolved|updated libraries/files/code which are maintained exclusively by the AI entity will/does exist
  • axg11 5y ago
    Speculating here: if the brain really uses embeddings similar (in concept) to neural network embeddings, the mechanism could explain a lot of the peculiarities of the brain. Embeddings are naturally entangled, so are memories. For example, a specific smell can evoke a previous memory.
• Sohcahtoa82 5y ago
  I saw a presentation once (It was not recorded, so I can't link it) that said the difference between real intelligence and artificial intelligence is the ability to quickly learn. As an example, he said imagine if he invented a word. Poditon. And he told us that a poditon is any object that can fit on a podium. Instantly, you know whether or not any object can be classified as a poditon. A laptop is a poditon, but a car is not. We are not at the stage where we can just tell a program "Any object that can fit on a podium is a poditon" and then ask "Is X a poditon?" and get a correct answer. And we probably won't be there for another couple decades.
  • gjm11 5y ago
    So, I tried this out with GPT-3 in the OpenAI Playground. (The model called text-davinci-002.) My prompt looked like this: Definition: A blorple is anything that is red and more or less round. Question: Is a tennis ball a blorple? Answer: No, because although tennis balls are round they aren't red. Question: Is a cherry a blorple? Answer: Yes, because cherries are red and approximately round. Definition: A poditon is anything that can fit on top of a podium. Question: Is a laptop computer a poditon? Answer: GPT-3 says: "Yes, because laptop computers are small enough to fit on top of a podium." Is a normal-sized automobile a poditon? "No, an automobile is too large to fit on top of a podium." Is the sun a poditon? "No, because the sun is too large to fit on top of a podium." Is a human being a poditon? "Yes, because human beings are small enough to fit on top of a podium." Is a house a poditon? "No, because a house is too large to fit on top of a podium." While generating those answers it also spontaneously answered the question for tennis balls (yes) and books (yes). Decades sure do go by quickly, these days.
    • Sohcahtoa82 5y ago
      FWIW, the presentation was in 2018, and GPT-3 came out in 2020. But yeah, that's pretty amazing.
  • stevenhuang 5y ago
    If you've seen examples of GPT3 you'd know this is already possible.
• konschubert 5y ago
  I find it funny how I can "see" a map of the world in front of me when I imagine it, but I totally cannot draw it. Clearly, much less information is stored than the whole image... yet my mind DALL-E style fills in the gaps and "sees" a map.
  • axg11 5y ago
    Already plugged this book elsewhere in the thread, you might be interested in "The Mind is Flat". One chapter of the book explores the concept you're describing. Our brain creates the illusion of a "full picture" when often our imagination and internal representation is quite sparse. I think that's one of the key impressive qualities of our brains and general intelligence. We only do the minimum necessary imagination and computation. As we explore a particular concept or scene, our brains augment the scene with more details. In other words, our mind is making it up as we go along.
    • deleted 5y ago
      [deleted]
    • bengale 5y ago
      A thousand brains is another really interesting book I read recently.
  • amelius 5y ago
    Keep drawing until what is on paper equals what is in your imagination. Seriously, try it.
    • playpause 5y ago
      Can you expand on this? Can you give an example of a kind of image it might work well with? I’ve always assumed the apparent detail of mental images was a kind of illusion, a bit like the illusion of detail outside the centre of the visual field.
      • loa_in_ 5y ago
• freediver 5y ago
  I co-authored a paper exploring this topic few years ago, while I was pretty excited about the possiblity of using embeddings for generalization. "Towards conceptual generalization in the embedding space" https://arxiv.org/abs/1906.01873 https://arxiv.org/abs/1906.01873 I still think the approach outlined in the paper (using embeddings to map the physical world) is sound especially for the field of self-driving which is in dire need of generalization, but I've since changed my mind and currently do not believe we can achieve AGI (ever). While embeddings are a great tool for compressing information, they do not provide inherent mechanisms for manipulating the information stored in order to generalize and infer outcomes in new, unseen situations. And even if we would start producing embeddings in a way where they would have some basic understanding of the physical world, we could never achieve it to the level of detail necessary - because physical world is not a discrete function. Otherwise we would be creating a perfect simulation (within a simulation?). And the last time I was playing God, was in "Populous".
  • ascar 5y ago
    > I've since changed my mind and currently do not believe we can achieve AGI (ever). Considering we (as in humans) developed general intelligence, isn't that already in contradiction with your statement? If it happened for us and is "easily" replicated through our DNA, it certainly can be developed again in an artificial medium. But the solution might not have anything to do with what we call machine learning today and sure we might go extinct before (but I didn't have the feeling that's what you were implying).
    • trompetenaccoun 5y ago
      It's semantics at this point but we did not create ourselves, it was a complex process that took billions of years to create each one of us. Something being conceivable isn't the same as it being practically possible. I can imagine what you propose, but the same goes for traveling to distant stars or a time machine for going to the future. All perfectly possible in theory.
      • jamiek88 5y ago
• tonguez 5y ago
  “Compression is a component of general intelligence.” if you say so “A few years ago I was very sceptical of machine learning ever leading to general intelligence. I've since changed my mind.” thanks for sharing “…both [human brains and artificial neural networks] are able to compress data into a _useful_ representation.” thanks for pointing this out
  • dang 5y ago
    Could you please stop posting unsubstantive and/or flamebait comments to HN? You've been doing it a ton, unfortunately, and we have ban that sort of account. It's not what this site is for, and it destroys what it is for. What we want here is thoughtful, curious conversation, not people bashing each other's comments or inflicting snark on each other or ideological talking points. If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.
• skhr0680 5y ago
  I’m bilingual from moving countries at a young age and now a lot of my early childhood memories are in the “wrong” language
• jasfi 5y ago
  If you're interested in AI, you can see/follow my project: https://lxagi.com https://lxagi.com.
• a_square_peg 5y ago
  > A prerequisite for generalization is the ability to take something high-dimensional and reduce it to a lower-dimensional representation to allow comparison and grouping of concepts. I've been thinking that it might actually the other way around - intelligence is taking lower dimensional data and being able to infer higher level representation in terms of context, meanings and other abstractions. I.e. understanding when a stop sign isn't a stop sign.
  • nl 5y ago
    It's actually the compression that forces it to learn higher level concepts. In your stop sign example, say we are trying to teach a visual model the difference between toy stop signs and real stop signs. To train it you feed it a 3D model of the world and the actions a person takes in response (ie, ignoring toy stop signs but stopping for real ones). Once the embedding is well trained (with lots of data) if you then run it through something like UMAP to reduce the number of dimensions in the embedding from hundreds to 2 or 3 you'll see it has "discovered" the concept of "scale" - all the small toy stop signs will be clustered together and the real ones clustered elsewhere. That generalisation forced by compression is where the abstraction of "scale" comes from. (Of course in real life you'd use a more complex model than just an embedding for this, but in principle this is the idea).
• EricMausler 5y ago
  >The ideal compression (from an intelligence point of view) is both information efficient and maximally composable This leads the compression to be overfitted to the learning environment. Not that our intelligence is entirely immune to that either
• fasteo 5y ago
  >>> Intelligence requires the ability to generalize Counterexample: I saw a TED talk video [1] by Dr. Grandin (autistic) telling that she is unable to generalize a "church" (starting at 02:50). I would say that she is extremely intelligent though [1] https://www.ted.com/talks/temple_grandin_the_world_needs_all_kinds_of_minds?language=en https://www.ted.com/talks/temple_grandin_the_world_needs_all...
• raxxorraxor 5y ago
  I think compression is a bad word or description. Another definition of intelligence is sometimes to differentiate essential from superficial information. Of course that often aligns with the application of compression of information.