3 ms·

Grant Sanderson has an excellent video on the same topic [0]. It's part of a series that is ongoing. [0] Compression is Intelligence Part 1 - https://youtu.be/

Grant Sanderson has an excellent video on the same topic [0]. It's part of a series that is ongoing. [0] Compression is Intelligence Part 1 - https://youtu.be/l6DKRf-fAAM?si=yyLWq8x4sSRkWd98 https://youtu.be/l6DKRf-fAAM?si=yyLWq8x4sSRkWd98

zahrevsky 2mo ago
I wonder if the author of the article knew about the series, or do they both just independently came across this topic to talk about it.
- cyanydeez 2mo ago
  it was vaguely in my understanding of information & intelligence with compression; it was also brought up in several of the initial trials against AI companies where they discussed how the AI is akin to compression. So they're both sourcing a bit broader zeitgeist.
- deleted 2mo ago
  [deleted]
- soulofmischief 2mo ago
  It's basic information theory, which has been around since the end of WWII. It's a common topic today because some of its subtle insights are becoming increasingly relevant in our current era of AI, as we learn to understand these black boxes.
- epistasis 2mo ago
  Anybody working in the field will be very familiar with these concepts.
  - larodi 2mo ago
    …for years. Because it is so apparent if you actually try to look at the problem and what is being solved by it. The extraction of features from a corpus, the features significant to certain solution, is always and since day zero - compression. As this is the definition of compression - efficient and potentially lossless feature extraction.
- internetter 2mo ago
  common theory. see https://prize.hutter1.net/ https://prize.hutter1.net/
  - AnotherGoodName 2mo ago
    And the Hutter Prize for AI which measures how good AI is by measuring how well it compresses data is over 20 years old now just to really drive the point home.
- tulio_ribeiro 2mo ago
  In the article, she credits the 2023 DeepMind paper Language Modeling Is Compression as the source. She also links to this post from 2015: https://colah.github.io/posts/2015-09-Visual-Information/ https://colah.github.io/posts/2015-09-Visual-Information/
samuell 2mo ago
Thank you for sharing! Small off topic thing: I recommend removing the bit in the URL from ?si=... and forward, unless you want Google to track every user who clicks this link to your share.

zahrevsky 2mo ago

I wonder if the author of the article knew about the series, or do they both just independently came across this topic to talk about it.

cyanydeez 2mo ago
it was vaguely in my understanding of information & intelligence with compression; it was also brought up in several of the initial trials against AI companies where they discussed how the AI is akin to compression. So they're both sourcing a bit broader zeitgeist.
deleted 2mo ago
[deleted]
soulofmischief 2mo ago
It's basic information theory, which has been around since the end of WWII. It's a common topic today because some of its subtle insights are becoming increasingly relevant in our current era of AI, as we learn to understand these black boxes.
epistasis 2mo ago
Anybody working in the field will be very familiar with these concepts.
- larodi 2mo ago
  …for years. Because it is so apparent if you actually try to look at the problem and what is being solved by it. The extraction of features from a corpus, the features significant to certain solution, is always and since day zero - compression. As this is the definition of compression - efficient and potentially lossless feature extraction.
internetter 2mo ago
common theory. see https://prize.hutter1.net/ https://prize.hutter1.net/
- AnotherGoodName 2mo ago
  And the Hutter Prize for AI which measures how good AI is by measuring how well it compresses data is over 20 years old now just to really drive the point home.
tulio_ribeiro 2mo ago
In the article, she credits the 2023 DeepMind paper Language Modeling Is Compression as the source. She also links to this post from 2015: https://colah.github.io/posts/2015-09-Visual-Information/ https://colah.github.io/posts/2015-09-Visual-Information/

samuell 2mo ago

Thank you for sharing! Small off topic thing: I recommend removing the bit in the URL from ?si=... and forward, unless you want Google to track every user who clicks this link to your share.