4 ms·

It seems absolutely wild to me that there are cases where you can prune 90% of neurons and still get a functional network out the other side. I'm definitely int

by mkaic 6y ago

It seems absolutely wild to me that there are cases where you can prune 90% of neurons and still get a functional network out the other side. I'm definitely interested in experimenting with some of these techniques myself soon!

elteto 6y ago
Please go easy on yourself...
- perl4ever 6y ago
  "A man with an unusually tiny brain manages to live an entirely normal life despite his condition, which was caused by a fluid build-up in his skull. Scans of the 44-year-old man's brain showed that a huge fluid-filled chamber called a ventricle took up most of the room in his skull, leaving little more than a thin sheet of actual brain tissue" An article goes on to say that he was tested at an IQ of 75, below normal but not disabled; he was employed as a civil servant, and married with children. https://web.archive.org/web/20150712092909if_/http://www.newscientist.com/article/dn12301-man-with-tiny-brain-shocks-doctors.html https://web.archive.org/web/20150712092909if_/http://www.new...
  - mjburgess 6y ago
    biology has no relevance here why the body can be regulated with a subset of brain function has nothing to do with why parameters covary in large statistical models
sillysaurusx 6y ago
It's not so strange if you visualize the actual weights of a model. Notice that most of the visualizations on https://battle.shawwn.com/sdb/visualizations/2020-09-16-117m-gpt-revnet/index.html https://battle.shawwn.com/sdb/visualizations/2020-09-16-117m... are solid colors.
- yorwba 6y ago
  If you zoom in until you can distinguish individual pixels, it's clear that they aren't actually solid colors. It's just that all the variability is at the pixel level and there are no large-scale structures. (There couldn't possibly be, because the ordering of rows and columns is arbitrary: you could shuffle them around and get an equivalent model. The most you could expect is some banding where values in the same row or column are related.)
mjburgess 6y ago
Measure the position of a pen relative to 1000 other objects in your room. Move the pen. How many of those measurements co-vary? Nearly all of them. Can you reduce that down to a subset of measurements that vary always independently? With care, you only need 3.
- robertacion 6y ago
  This is very interesting, can you recommend any link where I can learn more about this concept?
  - mjburgess 6y ago
    ggl factor analysis also: degree of freedom, dimension

by mkaic 6y ago

elteto 6y ago

Please go easy on yourself...

perl4ever 6y ago
"A man with an unusually tiny brain manages to live an entirely normal life despite his condition, which was caused by a fluid build-up in his skull. Scans of the 44-year-old man's brain showed that a huge fluid-filled chamber called a ventricle took up most of the room in his skull, leaving little more than a thin sheet of actual brain tissue" An article goes on to say that he was tested at an IQ of 75, below normal but not disabled; he was employed as a civil servant, and married with children. https://web.archive.org/web/20150712092909if_/http://www.newscientist.com/article/dn12301-man-with-tiny-brain-shocks-doctors.html https://web.archive.org/web/20150712092909if_/http://www.new...
- mjburgess 6y ago
  biology has no relevance here why the body can be regulated with a subset of brain function has nothing to do with why parameters covary in large statistical models

sillysaurusx 6y ago

It's not so strange if you visualize the actual weights of a model. Notice that most of the visualizations on https://battle.shawwn.com/sdb/visualizations/2020-09-16-117m-gpt-revnet/index.html https://battle.shawwn.com/sdb/visualizations/2020-09-16-117m... are solid colors.

yorwba 6y ago
If you zoom in until you can distinguish individual pixels, it's clear that they aren't actually solid colors. It's just that all the variability is at the pixel level and there are no large-scale structures. (There couldn't possibly be, because the ordering of rows and columns is arbitrary: you could shuffle them around and get an equivalent model. The most you could expect is some banding where values in the same row or column are related.)

mjburgess 6y ago

Measure the position of a pen relative to 1000 other objects in your room. Move the pen. How many of those measurements co-vary? Nearly all of them. Can you reduce that down to a subset of measurements that vary always independently? With care, you only need 3.

robertacion 6y ago
This is very interesting, can you recommend any link where I can learn more about this concept?
- mjburgess 6y ago
  ggl factor analysis also: degree of freedom, dimension