5 ms·

If I understand this correctly, they added a property to a NN of distance, and set the training to penalize increased distance between nodes to simulate a physi

by tudorw 3y ago

If I understand this correctly, they added a property to a NN of distance, and set the training to penalize increased distance between nodes to simulate a physical constraint, under these conditions 'hubs' emerged which facilitated connections across distance, the other observation was that "the same node might be able to encode multiple locations of a maze, rather than needing specialised nodes for encoding specific locations". The work suggests that existing approaches to neural network architecture would benefit from more closely emulating the operation of the brain in this regard.

pfd1986 3y ago
If this is the case, the finding sounds trivial. I thought it was literally a theorem that hubs emerge from networks if you minimize distance or wiring cost.
- tudorw 3y ago
  Maybe, I think the research here is from a brain perspective, AI and Brain Science are having an interesting time, with learnings from both feeding in, so this article is focused on the physicality of networks in the brain, and looking at whether by increasing the alignment between the way a neural network works and the way brains work they can develop a 'digital brain', for me this is distinct from machine intelligence, this is emulating brain activity to provide cognitive and psychological insight in meat space. Topology is hard for me, visualising multi-dimensional topological manifolds even more so, however, I am intrigued by the opportunities, we use graphs and other forms of visualisation to reveal topology, and it would seem reasonable that behaviours of NN that are 'transferrable' e.g.. trained on one set of data and able to operate on another category of data, may be a 'shape', a complicated one, but perhaps one that would once revealed would aid in the understanding of what happens under the hood.
  - Merrill 3y ago
    There must be some difficulty in studying biological brain intelligence to distinguish between features that contribute to intelligence and those which are simply path dependencies from millions of years of nervous system evolution. Over that time, brain function was mostly concerned with aiding the organism to find food, grow, reproduce, and avoid being eaten, rather than language, logic, mathematics, arts, and so forth. Its rather astonishing that humans are somehow now able to do the latter, using brains evolved to do the former.
    - edgyquant 3y ago
      Why would there be a difference at all here? Finding food and avoiding being eaten requires a great deal of intelligence, is surely the basis of human logic, and is the foundation for language. It’s all one interconnect network and every bit of it is a piece of the greater intelligence.
    - mycall 3y ago
      It isn't just humans that can do math. Ants to geolocation [0] by counting their steps in relation to the distance they walked from their nest. They do this with just a small amount of neurons. [0] https://blog.cambridgecoaching.com/ants-go-marching-fun-facts-about-how-ants-navigate#:~:text=The%20few%20neurons%20in%20the,the%20length%20of%20their%20legs https://blog.cambridgecoaching.com/ants-go-marching-fun-fact....
      - genman 3y ago
        Chickens can count. Humans can do abstract math, any math.
        fluidcruft 3y ago
        Well, humans can do any math that human brains can comprehend. That doesn't mean "any math" any more than if a chicken we're to claim that chickens can do any math (because they cannot comprehend human math).
    - swatcoder 3y ago
      You're overstating human difference. We appear to have a capacity for substantially greater sophsitication in those domains, but none are unique to us except when we artificially define them to be. Remember that words like "language", "logic", "art", etc are cultural inventions with a fuzzy and fluid relationship to whatever real-word "stuff" they refer to, not natural kinds that themselves have sharp and perennial definitions. Unless you choose to define the word as that which only humans can acheive, a spider's web elegently reflects "mathematics" just as much as some beautiful proof in set theory; a conflicted bird debating itself over which stem to use in its nest reflects artistic attention just as as a painter choosing their next color; a cat chirping or mewing or yowling reflects language just as me writing this comment. The sophistication doesn't go as far, by our eye at least, in any of these animal examples, and so we don't expect the spider to confirm Fermat's Last Theorem or the bird to feature their nest in a gallery (actually...) or a cat to compose formal poetry, but the essential bits that we extend with our sophistication are all ancient and widespread throughout nature. It's still astonishing that any life can do so many of the things it but I guess that's apparently what billions of years of "pretraining" on unfathomably efficient machines gets you. Incidentally, it's wild to see people believe that a stream of fmults pushing through a trillion transistors would get you even close to the sophistication of any of life's intelligence. For current-AI-skeptical materialists, it's usually not a doubt about whether silicon and software might conceivably be intelligent, but it can just seem absurd to believe the grossly crude and narrow innovations of recent years are even close. You need to have a very shallow, narrow, almost willfully blinded, appreciation of the "intelligence" exhibited throughout all biological life to think that you unlocked the silicon version of it all in a pretty-good chatbot running on Azure.
TeMPOraL 3y ago
> "the same node might be able to encode multiple locations of a maze, rather than needing specialised nodes for encoding specific locations" That may sound surprising, but it's actually not unusual. It's worth remembering that the difference between highly local encoding and everything encoding everything at the same time, is often a matter of a reversible transformation. For example, take FFT of an image, and the result will be an image where each pixel encodes information about all pixels of the original image simultaneously. And Fourier transform - a shift from a time/spatial domain to frequency domain - is quite simple, very useful, and occurs in nature.
- taneq 3y ago
  Sounds a bit like it's tending towards the behaviour of grid cells. Rather than encoding a specific, unique place, grid cells light up periodically in the latent space they encode, with different grid cells having different periods and different latent spaces. A given combination of grid cells uniquely identifies a thing/space/concept/location/datapoint/... token? Of course for any given location this wraps around with a period of the least common multiple between the grid cells' periods (I think? Had to ask Bing about that one, he was helpful for once). This is why if you try to map locations you get a hypertorus.
- saltwatercowboy 3y ago
  FFTs can be useful when constructing CNN pipelines in unsupervised domain adaptation too. Fourier domain adaptation (FDA) uses the calculated amplitude of a target set to modify the source (train) set. Cool and quick to implement.
  - dylan604 3y ago
    I wonder how pleased Joseph Fourier would be seeing how his work has become so pivotal in a world probably unimaginable to his time. Not just for ego sake, but just how damn useful something one's effort did becomes to everyone else.
    - SpaceManNabs 3y ago
westurner 3y ago
> existing approaches to neural network architecture would benefit from more closely emulating the operation of the brain in this regard. From https://news.ycombinator.com/item?id=38334538#38336861 https://news.ycombinator.com/item?id=38334538#38336861 : > Which NN architectures could be sufficient to simulate the entire human brain with spreading activation in 11 dimensions?
andrewflnr 3y ago
> The work suggests that existing approaches to neural network architecture would benefit from more closely emulating the operation of the brain in this regard. Does it? My read is that biological neutral architecture is a consequence of biological constraints that artificial NNs mostly don't face, so we don't necessarily need to try to copy it.
- kelseyfrog 3y ago
  There's a reason airplanes were invented before helicopters.
- spenczar5 3y ago
  Depends on your goal. If you want to simulate human intelligence, maybe you want to replicate constraints. A fair chunk of AI work boils down to “make something that acts like a human.” On the other end of the spectrum is stuff that is more specialized, like very targeted classifiers; there is no reason to expect those would benefit from this.
  - andrewflnr 3y ago
    Eh, maybe. The question there becomes, how much of human behavior is specifically a consequence of the physical constraints of laying out our neurons? I'm guessing not much. The article doesn't mention any change in output prompted by the change in internal architecture, which doesn't really confirm my guess, but doesn't falsify it either.
    - stefs 3y ago
      there's been physics simulation of bipedal models learning to walk via reinforcement learning. the results always were a bit choppy and robotic, until they implemented signals propagation delays. this led to more natural, fluid movements that definitely looked more "human". sorry, can't find the video. personally i absolutely do think that for generating convincingly human-like intelligence you also need some human constraints, otherwise you will get some uncanny valley. another example would be alpha-zeros play style. AIs don't play like humans, they maximize their chance of winning in the long term without going for good looking opportunities that hurt their chances in the long run (like human players do).
generalizations 3y ago
I'm just surprised this hasn't been done before. Isn't this the kind of low-hanging fruit that would have been studied at some point in the last 40 years?
- fho 3y ago
  It was ... look up "AI winter". Vastly simplified, we collectively ignored ANNs until somebody figured out that we now have enough recourses to create larger ANNs than before.
  - generalizations 3y ago
    Definitely familiar with the AI winter...which is why it seemed odd this current research as being presented as new. Guess it's just a rehash of the old ideas.
    - tudorw 3y ago
      Which is a fine place to start, a lot of our current shiny toys are a rehash of old ideas in a modern context, https://people.idsia.ch/~juergen/deep-learning-history.html https://people.idsia.ch/~juergen/deep-learning-history.html
      - fho 3y ago
        You should post that here. (if it is not something that is already posted again and again).

by tudorw 3y ago

pfd1986 3y ago

If this is the case, the finding sounds trivial. I thought it was literally a theorem that hubs emerge from networks if you minimize distance or wiring cost.

tudorw 3y ago
Maybe, I think the research here is from a brain perspective, AI and Brain Science are having an interesting time, with learnings from both feeding in, so this article is focused on the physicality of networks in the brain, and looking at whether by increasing the alignment between the way a neural network works and the way brains work they can develop a 'digital brain', for me this is distinct from machine intelligence, this is emulating brain activity to provide cognitive and psychological insight in meat space. Topology is hard for me, visualising multi-dimensional topological manifolds even more so, however, I am intrigued by the opportunities, we use graphs and other forms of visualisation to reveal topology, and it would seem reasonable that behaviours of NN that are 'transferrable' e.g.. trained on one set of data and able to operate on another category of data, may be a 'shape', a complicated one, but perhaps one that would once revealed would aid in the understanding of what happens under the hood.
- Merrill 3y ago
  There must be some difficulty in studying biological brain intelligence to distinguish between features that contribute to intelligence and those which are simply path dependencies from millions of years of nervous system evolution. Over that time, brain function was mostly concerned with aiding the organism to find food, grow, reproduce, and avoid being eaten, rather than language, logic, mathematics, arts, and so forth. Its rather astonishing that humans are somehow now able to do the latter, using brains evolved to do the former.
  - edgyquant 3y ago
    Why would there be a difference at all here? Finding food and avoiding being eaten requires a great deal of intelligence, is surely the basis of human logic, and is the foundation for language. It’s all one interconnect network and every bit of it is a piece of the greater intelligence.
  - mycall 3y ago
    It isn't just humans that can do math. Ants to geolocation [0] by counting their steps in relation to the distance they walked from their nest. They do this with just a small amount of neurons. [0] https://blog.cambridgecoaching.com/ants-go-marching-fun-facts-about-how-ants-navigate#:~:text=The%20few%20neurons%20in%20the,the%20length%20of%20their%20legs https://blog.cambridgecoaching.com/ants-go-marching-fun-fact....
    - genman 3y ago
      Chickens can count. Humans can do abstract math, any math.
      - fluidcruft 3y ago
        Well, humans can do any math that human brains can comprehend. That doesn't mean "any math" any more than if a chicken we're to claim that chickens can do any math (because they cannot comprehend human math).
  - swatcoder 3y ago
    You're overstating human difference. We appear to have a capacity for substantially greater sophsitication in those domains, but none are unique to us except when we artificially define them to be. Remember that words like "language", "logic", "art", etc are cultural inventions with a fuzzy and fluid relationship to whatever real-word "stuff" they refer to, not natural kinds that themselves have sharp and perennial definitions. Unless you choose to define the word as that which only humans can acheive, a spider's web elegently reflects "mathematics" just as much as some beautiful proof in set theory; a conflicted bird debating itself over which stem to use in its nest reflects artistic attention just as as a painter choosing their next color; a cat chirping or mewing or yowling reflects language just as me writing this comment. The sophistication doesn't go as far, by our eye at least, in any of these animal examples, and so we don't expect the spider to confirm Fermat's Last Theorem or the bird to feature their nest in a gallery (actually...) or a cat to compose formal poetry, but the essential bits that we extend with our sophistication are all ancient and widespread throughout nature. It's still astonishing that any life can do so many of the things it but I guess that's apparently what billions of years of "pretraining" on unfathomably efficient machines gets you. Incidentally, it's wild to see people believe that a stream of fmults pushing through a trillion transistors would get you even close to the sophistication of any of life's intelligence. For current-AI-skeptical materialists, it's usually not a doubt about whether silicon and software might conceivably be intelligent, but it can just seem absurd to believe the grossly crude and narrow innovations of recent years are even close. You need to have a very shallow, narrow, almost willfully blinded, appreciation of the "intelligence" exhibited throughout all biological life to think that you unlocked the silicon version of it all in a pretty-good chatbot running on Azure.

TeMPOraL 3y ago

> "the same node might be able to encode multiple locations of a maze, rather than needing specialised nodes for encoding specific locations" That may sound surprising, but it's actually not unusual. It's worth remembering that the difference between highly local encoding and everything encoding everything at the same time, is often a matter of a reversible transformation. For example, take FFT of an image, and the result will be an image where each pixel encodes information about all pixels of the original image simultaneously. And Fourier transform - a shift from a time/spatial domain to frequency domain - is quite simple, very useful, and occurs in nature.

taneq 3y ago
Sounds a bit like it's tending towards the behaviour of grid cells. Rather than encoding a specific, unique place, grid cells light up periodically in the latent space they encode, with different grid cells having different periods and different latent spaces. A given combination of grid cells uniquely identifies a thing/space/concept/location/datapoint/... token? Of course for any given location this wraps around with a period of the least common multiple between the grid cells' periods (I think? Had to ask Bing about that one, he was helpful for once). This is why if you try to map locations you get a hypertorus.
saltwatercowboy 3y ago
FFTs can be useful when constructing CNN pipelines in unsupervised domain adaptation too. Fourier domain adaptation (FDA) uses the calculated amplitude of a target set to modify the source (train) set. Cool and quick to implement.
- dylan604 3y ago
  I wonder how pleased Joseph Fourier would be seeing how his work has become so pivotal in a world probably unimaginable to his time. Not just for ego sake, but just how damn useful something one's effort did becomes to everyone else.
  - SpaceManNabs 3y ago

westurner 3y ago

> existing approaches to neural network architecture would benefit from more closely emulating the operation of the brain in this regard. From https://news.ycombinator.com/item?id=38334538#38336861 https://news.ycombinator.com/item?id=38334538#38336861 : > Which NN architectures could be sufficient to simulate the entire human brain with spreading activation in 11 dimensions?

andrewflnr 3y ago

> The work suggests that existing approaches to neural network architecture would benefit from more closely emulating the operation of the brain in this regard. Does it? My read is that biological neutral architecture is a consequence of biological constraints that artificial NNs mostly don't face, so we don't necessarily need to try to copy it.

kelseyfrog 3y ago
There's a reason airplanes were invented before helicopters.
spenczar5 3y ago
Depends on your goal. If you want to simulate human intelligence, maybe you want to replicate constraints. A fair chunk of AI work boils down to “make something that acts like a human.” On the other end of the spectrum is stuff that is more specialized, like very targeted classifiers; there is no reason to expect those would benefit from this.
- andrewflnr 3y ago
  Eh, maybe. The question there becomes, how much of human behavior is specifically a consequence of the physical constraints of laying out our neurons? I'm guessing not much. The article doesn't mention any change in output prompted by the change in internal architecture, which doesn't really confirm my guess, but doesn't falsify it either.
  - stefs 3y ago
    there's been physics simulation of bipedal models learning to walk via reinforcement learning. the results always were a bit choppy and robotic, until they implemented signals propagation delays. this led to more natural, fluid movements that definitely looked more "human". sorry, can't find the video. personally i absolutely do think that for generating convincingly human-like intelligence you also need some human constraints, otherwise you will get some uncanny valley. another example would be alpha-zeros play style. AIs don't play like humans, they maximize their chance of winning in the long term without going for good looking opportunities that hurt their chances in the long run (like human players do).

generalizations 3y ago

I'm just surprised this hasn't been done before. Isn't this the kind of low-hanging fruit that would have been studied at some point in the last 40 years?

fho 3y ago
It was ... look up "AI winter". Vastly simplified, we collectively ignored ANNs until somebody figured out that we now have enough recourses to create larger ANNs than before.
- generalizations 3y ago
  Definitely familiar with the AI winter...which is why it seemed odd this current research as being presented as new. Guess it's just a rehash of the old ideas.
  - tudorw 3y ago
    Which is a fine place to start, a lot of our current shiny toys are a rehash of old ideas in a modern context, https://people.idsia.ch/~juergen/deep-learning-history.html https://people.idsia.ch/~juergen/deep-learning-history.html
    - fho 3y ago
      You should post that here. (if it is not something that is already posted again and again).