3 ms·

Not sure whether that was a jab at Numenta or not, but I do think the combination of these two comments cuts to the heart of the PR problem for Numenta: they're

Not sure whether that was a jab at Numenta or not, but I do think the combination of these two comments cuts to the heart of the PR problem for Numenta: they're neither trying to do 0.1% better on the classic perception benchmarks than other algorithms do, nor trying to publish accurate, testable descriptions of the true details of meatspace neuroscience. To me, exploring alternative network architectures and algorithms seems an extremely worthwhile goal even if it's only loosely tethered to actual biology, but from a PR perspective they really need to be better about priming the conversation if they want people to care. Bad (neuroscience-focused): "We're doing a lot of research on neuroscience, and finding some really interesting stuff, so we built a model that doesn't exactly match the way the brain works but is still interesting. No, we haven't tried to make it work to classify ImageNet test cases, that's not our goal. But look, it's closer to biology, and we have working code that we're playing with!" Better (ML-focused): "We're developing a novel neural network architecture that performs online unsupervised learning using only local update rules. Though it performs competently at classic benchmarks X, Y, and Z when a small WTA layer is thrown on top, it can also tackle problems A, B, and C that classical deep learning networks can't make any progress on." To be fair, I'm not even sure if Numenta's networks could perform competently at any classic benchmarks (I'm guessing that if they could, it would take some work to get them to do so), and I have no idea what new problems it could work on. But they really do need to reframe the conversation and emphasize that sort of innovation if they want to be taken more seriously - focusing on neuroscience underpinnings is not a great move if they're not engaged in research that can actually win over neuroscientists, and just pointing out that they're focusing on those things is not a way to win over industry ML folks if they don't have any results to point at.

eli_gottlieb 9y ago
>To be fair, I'm not even sure if Numenta's networks could perform competently at any classic benchmarks (I'm guessing that if they could, it would take some work to get them to do so), and I have no idea what new problems it could work on. But they really do need to reframe the conversation and emphasize that sort of innovation if they want to be taken more seriously - focusing on neuroscience underpinnings is not a great move if they're not engaged in research that can actually win over neuroscientists, and just pointing out that they're focusing on those things is not a way to win over industry ML folks if they don't have any results to point at. It was definitely a jab, but I've also got some sympathy for their project. I genuinely agree that, well, theoretical and computational neuroscience need to become more genuinely computational! We're seeing an emerging computational paradigm for neuroscience that isn't just about jamming "network architectures" or "neural circuits" together and hoping something works; it supposedly has strong mathematical principles. Ok, so where's the code? Sincere question. Some papers do simulations in Matlab, R, or Python that's just not shared. This includes even papers that purport to be applying these neuroscience-derived principles to robotics problems. Computational cognitive science does a bunch better: their custom-built Matlab gets shared! If we really believe our theories, we should put them to the computational test. If we put them to the test and they don't work well, we should either revise the theories, or revise the benchmarks. Maybe ImageNet classification scores are a bad idea for how to measure precise, accurate sensorimotor inference! New benchmarks for measuring the performance of "real" cognitive systems are a great idea! Let's do it! But that requires that we do the slow work of trying to merge theoretical/computational neurosci, cognitive science, and ML/AI back together, at least in some subfields. This is challenging, because nobody's gonna give us our own journal for it until a few prestigious people advocate for one.

eli_gottlieb 9y ago

>To be fair, I'm not even sure if Numenta's networks could perform competently at any classic benchmarks (I'm guessing that if they could, it would take some work to get them to do so), and I have no idea what new problems it could work on. But they really do need to reframe the conversation and emphasize that sort of innovation if they want to be taken more seriously - focusing on neuroscience underpinnings is not a great move if they're not engaged in research that can actually win over neuroscientists, and just pointing out that they're focusing on those things is not a way to win over industry ML folks if they don't have any results to point at. It was definitely a jab, but I've also got some sympathy for their project. I genuinely agree that, well, theoretical and computational neuroscience need to become more genuinely computational! We're seeing an emerging computational paradigm for neuroscience that isn't just about jamming "network architectures" or "neural circuits" together and hoping something works; it supposedly has strong mathematical principles. Ok, so where's the code? Sincere question. Some papers do simulations in Matlab, R, or Python that's just not shared. This includes even papers that purport to be applying these neuroscience-derived principles to robotics problems. Computational cognitive science does a bunch better: their custom-built Matlab gets shared! If we really believe our theories, we should put them to the computational test. If we put them to the test and they don't work well, we should either revise the theories, or revise the benchmarks. Maybe ImageNet classification scores are a bad idea for how to measure precise, accurate sensorimotor inference! New benchmarks for measuring the performance of "real" cognitive systems are a great idea! Let's do it! But that requires that we do the slow work of trying to merge theoretical/computational neurosci, cognitive science, and ML/AI back together, at least in some subfields. This is challenging, because nobody's gonna give us our own journal for it until a few prestigious people advocate for one.