5 ms·

Isn’t this just a specific application of an autoencoder though? Because random noise isn’t learnable, it gets filtered out if you teach the network to compress

Isn’t this just a specific application of an autoencoder though? Because random noise isn’t learnable, it gets filtered out if you teach the network to compress/decompress a sample with itself as target. https://blog.keras.io/building-autoencoders-in-keras.html https://blog.keras.io/building-autoencoders-in-keras.html With that said, it’s amazing how effective this concept is for cleaning up scientific data. Many have also used variational autoencoders to take it one step further and also cluster data by latent space features. I’ve used it myself to uncover various groups of behaviors in time-series from other cellular processes.

iandanforth 6y ago
I think if you take "just" out of your comment you're spot on. :) Also, I know what you mean but for completeness, random noise is learnable, we do this all the time. You can identify the onset and offset not just of noise but different types of noise. If you mix blue and brown noise and listen to it for a while and then drop one of the noise channels you definitely notice it. You aren't learning the specific values of any part of the noise (as you say this is unpredictable) but the frequency components are consistent enough that we constantly subtract them from our sensory streams all the time. Noise tolerance is one of the most fascinating aspects of biological intelligence IMHO :)
- fpgaminer 6y ago
  > Also, I know what you mean but for completeness, random noise is learnable, we do this all the time. You are absolutely correct. The fact that brains learn randomness is _mind blowing_, and it has immense implications for our perception of reality. Let me take fellow HNers on a journey. Consider the following snippets about human vision: > However, there’s very little connectivity between the retina and the visual cortex. For a visual area roughly one-quarter the size of a full moon, there are only about 10 nerve cells connecting the retina to the visual cortex. These cells make up the LGN, or lateral geniculate nucleus, the only pathway through which visual information travels from the outside world into the brain. > “But the brain doesn’t take a picture, the retina does, and the information passed from the retina to the visual cortex is sparse.” > While the cortex and the retina are connected by relatively few neurons, the cortex itself is dense with nerve cells. For every 10 LGN neurons that snake back from the retina, there are 4,000 neurons in just the initial “input layer” of the visual cortex — and many more in the rest of it. Source: https://getpocket.com/explore/item/a-mathematical-model-unlocks-the-secrets-of-vision https://getpocket.com/explore/item/a-mathematical-model-unlo... > However, the retina only accounts for a small percentage of LGN input. As much as 95% of input in the LGN comes from the visual cortex, superior colliculus, pretectum, thalamic reticular nuclei, and local LGN interneurons. Source: https://en.wikipedia.org/wiki/Lateral_geniculate_nucleus https://en.wikipedia.org/wiki/Lateral_geniculate_nucleus In other words, contrary to popular belief, there are scarce few connections between your brain and your retinas. Our retinas may themselves have immense resolution, but the bandwidth of the connection between our retinas and our visual cortex is the neural equivalent of dial-up internet. Very little "real" information is being passed up to the brain. The implication is that what we "see", or rather what we perceive as vision, is almost entirely fabricated. We don't directly see the information coming from our eyes. Instead, we only see what our brain predicted we would see. The limited information coming from our retinas is, most likely, only used to verify and adjust our brain's internal fantasy. Now to bring this back on topic and blow some minds. I'm sure everyone has looked at static on a TV before, right? Black and white noise. But wait ... we know our vision doesn't have the bandwidth to actually see that noise. Noise is, after all, pure entropy, incompressible, unpredictable. Yet we _do_ see it. We do perceive it. Therefore, that noise that _you_ see, it's completely faked by your brain. The noise you perceive is different from the noise someone else looking at the same screen might perceive. Weird, right? The only reason this isn't common knowledge is ... how would you know? Take two people and sit them in front of the same static filled TV. How would they feasibly compare their perceptions? How would they know they aren't seeing the same thing? We can't faithfully memorize and draw the static we see. There's no way to compare our perceptions of noise. So it's no wonder that this "glitch" of human perception isn't commonly known. This of course only occurs with very noisy situations. A coin flip is also random, but it's only a single bit of information, so it's easy for our brains to quickly adjust their model to reality and absorb that bit. That's true of a lot of our world; our brains spend decades perfecting the ability to predict the world, and in so doing requiring as few bits as possible from our senses to correct mispredictions. Further proof/mind blowingness: There are recorded medical cases of people encountering sudden, complete damage to their vision, yet having absolutely no clue that such a thing has occurred. I'm talking things like their optic nerves completely failing, so that there is absolutely 0 visual information coming into your brain. Yet they act as if they have perfect sight. For a little while at least. (NOTE: This differs from blind-sight, a related but different phenomena where there is still some connection between your retinas and brain). Hell, even my own eye doctor has seen patients who went completely blind in one eye without ever noticing (until he told them so). What's more, it's likely all of our senses are this way. The totality of your experience is top-down, derived primarily from the internal model of the world your brain has built, with your physical senses acting more like error inputs than the bottom-up inputs we believe them to be. It's why hallucinations are so convincing. It's why we're able to have vivid dreams.
  - dangom 6y ago
    We should remember that there is a lot of computing going on in the retina already, and that rod cells of the retina are so sensitive that they can detect even single photons [1]. The density of cells also depends on the location of the FoV that we are talking about. We have many more connections and computation power reserved for the fovea than for our peripheral vision. With that said, I'm pretty sure that our perception of noise is not as dissimilar as you suggest. By doing the simple experiment of freezing on a static noise frame and asking participants to tell me whether a pixel was black or white, I'd expect most people to get it right. https://www.cns.nyu.edu/csh/csh04/Articles/Rieke1998.pdf https://www.cns.nyu.edu/csh/csh04/Articles/Rieke1998.pdf
    - fpgaminer 6y ago
      > With that said, I'm pretty sure that our perception of noise is not as dissimilar as you suggest. By doing the simple experiment of freezing on a static noise frame and asking participants to tell me whether a pixel was black or white, I'd expect most people to get it right. Right, my suggestion was that the phenomena would occur if the noise exceeded our visual "bandwidth". If you freeze frame then you're giving your mental model enough time to "catch up" with reality, so to speak.

iandanforth 6y ago

I think if you take "just" out of your comment you're spot on. :) Also, I know what you mean but for completeness, random noise is learnable, we do this all the time. You can identify the onset and offset not just of noise but different types of noise. If you mix blue and brown noise and listen to it for a while and then drop one of the noise channels you definitely notice it. You aren't learning the specific values of any part of the noise (as you say this is unpredictable) but the frequency components are consistent enough that we constantly subtract them from our sensory streams all the time. Noise tolerance is one of the most fascinating aspects of biological intelligence IMHO :)

fpgaminer 6y ago
> Also, I know what you mean but for completeness, random noise is learnable, we do this all the time. You are absolutely correct. The fact that brains learn randomness is _mind blowing_, and it has immense implications for our perception of reality. Let me take fellow HNers on a journey. Consider the following snippets about human vision: > However, there’s very little connectivity between the retina and the visual cortex. For a visual area roughly one-quarter the size of a full moon, there are only about 10 nerve cells connecting the retina to the visual cortex. These cells make up the LGN, or lateral geniculate nucleus, the only pathway through which visual information travels from the outside world into the brain. > “But the brain doesn’t take a picture, the retina does, and the information passed from the retina to the visual cortex is sparse.” > While the cortex and the retina are connected by relatively few neurons, the cortex itself is dense with nerve cells. For every 10 LGN neurons that snake back from the retina, there are 4,000 neurons in just the initial “input layer” of the visual cortex — and many more in the rest of it. Source: https://getpocket.com/explore/item/a-mathematical-model-unlocks-the-secrets-of-vision https://getpocket.com/explore/item/a-mathematical-model-unlo... > However, the retina only accounts for a small percentage of LGN input. As much as 95% of input in the LGN comes from the visual cortex, superior colliculus, pretectum, thalamic reticular nuclei, and local LGN interneurons. Source: https://en.wikipedia.org/wiki/Lateral_geniculate_nucleus https://en.wikipedia.org/wiki/Lateral_geniculate_nucleus In other words, contrary to popular belief, there are scarce few connections between your brain and your retinas. Our retinas may themselves have immense resolution, but the bandwidth of the connection between our retinas and our visual cortex is the neural equivalent of dial-up internet. Very little "real" information is being passed up to the brain. The implication is that what we "see", or rather what we perceive as vision, is almost entirely fabricated. We don't directly see the information coming from our eyes. Instead, we only see what our brain predicted we would see. The limited information coming from our retinas is, most likely, only used to verify and adjust our brain's internal fantasy. Now to bring this back on topic and blow some minds. I'm sure everyone has looked at static on a TV before, right? Black and white noise. But wait ... we know our vision doesn't have the bandwidth to actually see that noise. Noise is, after all, pure entropy, incompressible, unpredictable. Yet we _do_ see it. We do perceive it. Therefore, that noise that _you_ see, it's completely faked by your brain. The noise you perceive is different from the noise someone else looking at the same screen might perceive. Weird, right? The only reason this isn't common knowledge is ... how would you know? Take two people and sit them in front of the same static filled TV. How would they feasibly compare their perceptions? How would they know they aren't seeing the same thing? We can't faithfully memorize and draw the static we see. There's no way to compare our perceptions of noise. So it's no wonder that this "glitch" of human perception isn't commonly known. This of course only occurs with very noisy situations. A coin flip is also random, but it's only a single bit of information, so it's easy for our brains to quickly adjust their model to reality and absorb that bit. That's true of a lot of our world; our brains spend decades perfecting the ability to predict the world, and in so doing requiring as few bits as possible from our senses to correct mispredictions. Further proof/mind blowingness: There are recorded medical cases of people encountering sudden, complete damage to their vision, yet having absolutely no clue that such a thing has occurred. I'm talking things like their optic nerves completely failing, so that there is absolutely 0 visual information coming into your brain. Yet they act as if they have perfect sight. For a little while at least. (NOTE: This differs from blind-sight, a related but different phenomena where there is still some connection between your retinas and brain). Hell, even my own eye doctor has seen patients who went completely blind in one eye without ever noticing (until he told them so). What's more, it's likely all of our senses are this way. The totality of your experience is top-down, derived primarily from the internal model of the world your brain has built, with your physical senses acting more like error inputs than the bottom-up inputs we believe them to be. It's why hallucinations are so convincing. It's why we're able to have vivid dreams.
- dangom 6y ago
  We should remember that there is a lot of computing going on in the retina already, and that rod cells of the retina are so sensitive that they can detect even single photons [1]. The density of cells also depends on the location of the FoV that we are talking about. We have many more connections and computation power reserved for the fovea than for our peripheral vision. With that said, I'm pretty sure that our perception of noise is not as dissimilar as you suggest. By doing the simple experiment of freezing on a static noise frame and asking participants to tell me whether a pixel was black or white, I'd expect most people to get it right. https://www.cns.nyu.edu/csh/csh04/Articles/Rieke1998.pdf https://www.cns.nyu.edu/csh/csh04/Articles/Rieke1998.pdf
  - fpgaminer 6y ago
    > With that said, I'm pretty sure that our perception of noise is not as dissimilar as you suggest. By doing the simple experiment of freezing on a static noise frame and asking participants to tell me whether a pixel was black or white, I'd expect most people to get it right. Right, my suggestion was that the phenomena would occur if the noise exceeded our visual "bandwidth". If you freeze frame then you're giving your mental model enough time to "catch up" with reality, so to speak.