3 ms·

You used a lot of fancy words, so Ugh had to think a lot. Ugh is now tired. Let me try this. Ugh uses stupid words because he is not as clever. If an athlete

You used a lot of fancy words, so Ugh had to think a lot. Ugh is now tired. Let me try this. Ugh uses stupid words because he is not as clever. If an athlete catches a ball, does he remember, somewhere, somehow, every possible trajectory of every possible permutation of “ball” and every possible response to it? Or does he learn patterns of movement, generalized reactions to “ball-like things” flying “in my general direction”? It’s of course hard to describe non-verbal abstraction which is why we usually resort to easier subjects. I don’t however see your fundamental point: how else besides abstracting, by “compressing”, does an “athlete” learn? I suspect a lot of people get sidetracked by modality. You see it with LLMs. It’s only text! Yeah, well, it’s tokens and everything can be tokens. See the latest transformer based audio generator Bark. I don’t see why movement is different. It’s about the structure in the data, not the shape, context, use or “community” of it. Edit: > We don't call someone intelligent because we can point to some lossless compression algorithm implemented by his neurons, we call him intelligent when he produces the behaviors we desire in a given situation. “We don’t call a car red because of particular wavelengths of light activating sensors in our retinas, we call it red because it looks that way”. I’d be careful about attaching importance to how we use words in daily life. There are so, so many shortcuts we take.

voidhorse 3y ago
Ok, try another example. Humans are not the only animals capable of learning. Do you think it's fair to say a crow learns how to crack a nut via compression and abstraction? I think it's a combination of community and pattern (structure). Sure, once the LLM is trained, it can pattern match. Once the crow watches another crow, it can crack the nut, but the crow that does not exist in the community of nut cracking knowledge-sharers will not learn to recognize the pattern merely off its structure alone. In other words, compression (or, induction) is a part of the story, sure, but it is not the only factor.
- SanderNL 3y ago
  > Do you think it's fair to say a crow learns how to crack a nut via compression and abstraction? Well, yes. Do you know another way? Edit: oh I forgot your second part (I’m Ugh). You say a crow alone does not learn this. How do you propose the first crow learned? At some point a single crow learns. I know it propagates fast through a group, but that’s propagation. A different problem.
  - voidhorse 3y ago
    Well, we'll agree to disagree then. When we talk about abstractions, I think we typically mean to point to, predominantly, linguistic representations. I'm confident a crow could crack a nut but I am not confident it could produce a representation of how to crack a nut. Because of the latter point, I'm reluctant to claim a crow is capable of abstraction. I might be wrong of course...at this point it sort of feels like we've already taken "compression and abstraction" to mean "whatever it is the brain is doing" eschewing details.
    - SanderNL 3y ago
      This makes me think of Wittgensteins language games. I’m used to the word abstraction in a different context and it’s not linguistic to me at all. I’m not the greatest communicator so that doesn’t help either. The definition “disassociated from any specific instance” would be my take in this context. “Learning” would mean something like generating a sufficiently abstracted model that is useful over a wide variety of inputs. Aka you don’t store every instance of every type of nut, you store “nutness”. This is where the compression metaphor is quite obvious. You reduce the astronomically complex world of nuts down to “nutness”, in a way that still makes sense to you. Not dissimilar, I think, to how MP3 or JPEG drops significant amounts of information but still remains useful to us. Only thing I would argue is that it’s lossy not lossless. Lossless would imply the world can be reduced without dropping information and I don’t think it can to any significant degree. Anyway not trying to persuade you or anything. Your input does make me think and that’s not a bad thing, right? Thanks for indulging me.

voidhorse 3y ago

Ok, try another example. Humans are not the only animals capable of learning. Do you think it's fair to say a crow learns how to crack a nut via compression and abstraction? I think it's a combination of community and pattern (structure). Sure, once the LLM is trained, it can pattern match. Once the crow watches another crow, it can crack the nut, but the crow that does not exist in the community of nut cracking knowledge-sharers will not learn to recognize the pattern merely off its structure alone. In other words, compression (or, induction) is a part of the story, sure, but it is not the only factor.

SanderNL 3y ago
> Do you think it's fair to say a crow learns how to crack a nut via compression and abstraction? Well, yes. Do you know another way? Edit: oh I forgot your second part (I’m Ugh). You say a crow alone does not learn this. How do you propose the first crow learned? At some point a single crow learns. I know it propagates fast through a group, but that’s propagation. A different problem.
- voidhorse 3y ago
  Well, we'll agree to disagree then. When we talk about abstractions, I think we typically mean to point to, predominantly, linguistic representations. I'm confident a crow could crack a nut but I am not confident it could produce a representation of how to crack a nut. Because of the latter point, I'm reluctant to claim a crow is capable of abstraction. I might be wrong of course...at this point it sort of feels like we've already taken "compression and abstraction" to mean "whatever it is the brain is doing" eschewing details.
  - SanderNL 3y ago
    This makes me think of Wittgensteins language games. I’m used to the word abstraction in a different context and it’s not linguistic to me at all. I’m not the greatest communicator so that doesn’t help either. The definition “disassociated from any specific instance” would be my take in this context. “Learning” would mean something like generating a sufficiently abstracted model that is useful over a wide variety of inputs. Aka you don’t store every instance of every type of nut, you store “nutness”. This is where the compression metaphor is quite obvious. You reduce the astronomically complex world of nuts down to “nutness”, in a way that still makes sense to you. Not dissimilar, I think, to how MP3 or JPEG drops significant amounts of information but still remains useful to us. Only thing I would argue is that it’s lossy not lossless. Lossless would imply the world can be reduced without dropping information and I don’t think it can to any significant degree. Anyway not trying to persuade you or anything. Your input does make me think and that’s not a bad thing, right? Thanks for indulging me.