4 ms·

As far as I can tell, this is just a clever trick. The cultured neurons aren't really learning anything. The researchers are stimulating one side of the array (

As far as I can tell, this is just a clever trick. The cultured neurons aren't really learning anything. The researchers are stimulating one side of the array (top or bottom), to indicate if the ball is above or below the paddle, and based on the ensuing response the computer moves the paddle up or down. Here's the trick - if you stimulate a group of neurons, their spontaneous activity will be (briefly) different from an unstimulated group. You can use this fact to make it seem like the NN is learning to play pong. It would be like claiming a network of cats can learn to play pong. I put cats in two different boxes. To tell the cats whether the ball is above or below the paddle I will spray water into one of the boxes. Then I will listen for hissing from both boxes. Based on where I hear the most hissing is the direction i move the paddle.

gentleman11 5y ago
I’m really disillusioned by science this year. Almost every study I’ve read about on hn in the last year had serious methodology issues
- wolverine876 5y ago
  Perhaps that's in the HN comments more than in the research. IME, for as long as I've read HN, commenters have claimed to have found errors in methodology or similar basic problems. I don't know if I've seen any research posted that didn't receive that response.
  - zzt123 5y ago
    I feel like academica kind of incentivizes this. It’s hard to resist without a good base of support, and I feel like having such a base isn’t the average. I notice this pressure felt strongly across the board by my friends who are adjuncts (or tenure-track, but not yet tenured) in the soft sciences. But this is just anecdotal of course.
    - wolverine876 5y ago
      > I feel like academica kind of incentivizes this. It’s hard to resist without a good base of support Incentivizes what? What is hard to resist?
      - zzt123 5y ago
        Getting a certain volume of publications out. It can be problematic for example when you discover a minor data issue which might not materially change the thesis in your opinion, but if one were wholly following academic guidelines, would warrant delays while you attempt to fix the data issue, if possible. I’ve seen this happen a few times in papers that were never fixed, nor the issue brought up as far as I’m aware. Granted, in each case the thesis was still fine but… makes me wonder about the cases I don’t know about. One of the cases skirted really close to the edge of bs.
  - gus_massa 5y ago
    Note that it's increasingly common to discuss preprints, like in this case. A preprint is as reliable as a blog post in WordPress, unless you work in the area and know that the author has a solid track record. Also, most results that reach the front page here are groundbreaking. But groundbreaking results are very rare, so most of them must be fake, because they have an error or because they are an exaggeration. Most good results are too technical and boring. My favorite example is Magnetoresistance. It's a weird quantum effect that is caused because electrons can have spin up or down. It has a huge effect in the applications and almost everyone had a device that used it. Now the applications are fading. But no one knows it. There are probably 1000 times more discussions here about faster than light travel than magnetoresistance.
    - wolverine876 5y ago
      > A preprint is as reliable as a blog post in WordPress, unless you work in the area and know that the author has a solid track record. While preprints lack some work and signals of reliability, such as peer review, revisions by the author, etc., I think they are far beyond blog posts. Imagine a blog post, by an expert in the domain, with the amount of research and precision that goes into a preprint - there's no comparison. Nothing, not even a peer-reviewed article in Nature, is so reliable that we can just put away our critical thinking.
      - gus_massa 5y ago
        There is a lot of variation, for example https://terrytao.wordpress.com/ https://terrytao.wordpress.com/ and I remember some guy that claimed that got a fantastic method for factorization that broke RSA and posted it in the arxiv, and the first 5 page of the preprint was an explanation of the rule of nines. (If the sum of the digits of a number is divisible by 9 then the number is divisible by 9.)
- sdenton4 5y ago
  I think many non scientists have a bit of a wrong idea of how science actually works. As has been said, reality has a surprising amount of detail. This means that, outside mathematics, perfect methodology is impossible. So you do your level best to create a good methodology under the constraints of time, technology and proper understanding, and roll out the experiment. Filling in the cracks of doubt is a big part of what replication and productionizing is for. Of you think a study has a big enough hole in it's methodology, you are free to go design your own study to try to plug the hole or refute the result. A corollary, though, is that there is always going to be a 'hole' big enough for a jackass on the internet to criticize. But the question is whether you're willing to put the skin in the game to see if the result holds when you design a study to exploit the hole.
  - wolverine876 5y ago
    > outside mathematics, perfect methodology is impossible Even in mathematics ... others on HN are better qualified than I am to comment, but I've read mathematicians say that almost no reader of a paper works out all the math; to a degree they go by reputation of the author that the mathematics is generally correct. Similarly to your comment, while outsiders envision perfection, people inside have limited time for it. It may be more productive to spend that time on new research than on investigating every little detail of old research.
  - jhoelzel 5y ago
    I believe this is only partially true! We call it science because when we find methods that we all agree on, as a culture to use then it becomes scientific. For Instance psychology, in the freudean psychology the methology is still science even though he speaks of urges of "your mother" and interesting things like the "subconciousness". It has been met with great doubt over it's "initiation" but is now a widely acceptable practise. So what has changed? Well the field concentrated on scientific core value which (almost) boil down to: Find a methology, prove that it works and replicated it again.scientific quality criteria. I agree with op! Stimulating cells with electricity is not teaching them. Its assisted muscle reaction. In this case its really clear cut too: if I need to give the braincells electro schocks (the thing they operate on), If anything, I am remote controlling brain cells. The whole idea of perception is flawed in this experiment, you know, because there is no actual perceptor of the paddel to begin with. Sorry but as someone with a bachelor in psychology and 20 years of dev experience the starting poing of "I think many non scientists have a bit of a wrong idea of how science actually works." is just condecending theory. This is literally what "scientific quality criteria" is based on.
- SubiculumCode 5y ago
  I'll break it down in a couple of bullet points. 1. Science is hard. 2. There is no such thing as a perfect study. Every method/methodology has weaknesses. Look for converging evidence across multiple studies which use different designs. 3. Initial studies often do not have the funding needed to do things as optimally as possible in a single study, but they do provide evidence for ideas that can be pursued further. 4. It is often socially safer to point out deficiencies than to praise strengths. Same as snobs raining down on you when you say you like some midrange wine. 5. Often issues will be raised that are already acknowledged by the authors. 6. Complaints are often based on inaccurate news reporting, not actual research. 7. Science is hard.
runnerup 5y ago
This seems a bit different than what you suggest. The paper shows the "average rally length" and "# of hits per minute" increasing as time goes on for the group which received additional stimulus as feedback when it hit the ball. It did not find an increase in these metrics for groups where the input signal went silent as feedback after hitting the ball, nor for groups which merely continued receiving the usual input ("no feedback"). I have no idea if this is repeatable, but it seems like a valid conclusion from this data that the neuron colonies which were given feedback in the form of extra stimulation, exhibited better performance after the first 5 minutes of each trial, and better performance each subsequent day when they would run another 20 minute trial.
yorwba 5y ago
Their electrode configuration has predetermined non-overlapping sensory and motor areas. So it would be more like having a whole bunch of cats lined up and spraying water onto the cat corresponding to the ball position, while listening for the hissing from some other cats to determine where to move the paddle. The random stimulus as feedback when the ball resets corresponds to spraying the cats unpredictably. To avoid that punishment, the cats need to work together to make the right cats hiss at the right time to intercept the ball with the paddle. (The underlying theory is that cats are okay with being sprayed as long as they can predict it, a.k.a. the Free Energy Principle.) The fact that this feedback increases performance relative to the other experimental conditions indicates that some adaptation is going on in the network.
- dr1337 5y ago
  That's fascinating. It reminds me of this clip from Adam Curtis's All Watched Over By Machines of Loving Grace where they got lots of random people to play Pong in a theatre in the 90s: https://vimeo.com/78043173 https://vimeo.com/78043173

gentleman11 5y ago

I’m really disillusioned by science this year. Almost every study I’ve read about on hn in the last year had serious methodology issues

wolverine876 5y ago
Perhaps that's in the HN comments more than in the research. IME, for as long as I've read HN, commenters have claimed to have found errors in methodology or similar basic problems. I don't know if I've seen any research posted that didn't receive that response.
- zzt123 5y ago
  I feel like academica kind of incentivizes this. It’s hard to resist without a good base of support, and I feel like having such a base isn’t the average. I notice this pressure felt strongly across the board by my friends who are adjuncts (or tenure-track, but not yet tenured) in the soft sciences. But this is just anecdotal of course.
  - wolverine876 5y ago
    > I feel like academica kind of incentivizes this. It’s hard to resist without a good base of support Incentivizes what? What is hard to resist?
    - zzt123 5y ago
      Getting a certain volume of publications out. It can be problematic for example when you discover a minor data issue which might not materially change the thesis in your opinion, but if one were wholly following academic guidelines, would warrant delays while you attempt to fix the data issue, if possible. I’ve seen this happen a few times in papers that were never fixed, nor the issue brought up as far as I’m aware. Granted, in each case the thesis was still fine but… makes me wonder about the cases I don’t know about. One of the cases skirted really close to the edge of bs.
- gus_massa 5y ago
  Note that it's increasingly common to discuss preprints, like in this case. A preprint is as reliable as a blog post in WordPress, unless you work in the area and know that the author has a solid track record. Also, most results that reach the front page here are groundbreaking. But groundbreaking results are very rare, so most of them must be fake, because they have an error or because they are an exaggeration. Most good results are too technical and boring. My favorite example is Magnetoresistance. It's a weird quantum effect that is caused because electrons can have spin up or down. It has a huge effect in the applications and almost everyone had a device that used it. Now the applications are fading. But no one knows it. There are probably 1000 times more discussions here about faster than light travel than magnetoresistance.
  - wolverine876 5y ago
    > A preprint is as reliable as a blog post in WordPress, unless you work in the area and know that the author has a solid track record. While preprints lack some work and signals of reliability, such as peer review, revisions by the author, etc., I think they are far beyond blog posts. Imagine a blog post, by an expert in the domain, with the amount of research and precision that goes into a preprint - there's no comparison. Nothing, not even a peer-reviewed article in Nature, is so reliable that we can just put away our critical thinking.
    - gus_massa 5y ago
      There is a lot of variation, for example https://terrytao.wordpress.com/ https://terrytao.wordpress.com/ and I remember some guy that claimed that got a fantastic method for factorization that broke RSA and posted it in the arxiv, and the first 5 page of the preprint was an explanation of the rule of nines. (If the sum of the digits of a number is divisible by 9 then the number is divisible by 9.)
sdenton4 5y ago
I think many non scientists have a bit of a wrong idea of how science actually works. As has been said, reality has a surprising amount of detail. This means that, outside mathematics, perfect methodology is impossible. So you do your level best to create a good methodology under the constraints of time, technology and proper understanding, and roll out the experiment. Filling in the cracks of doubt is a big part of what replication and productionizing is for. Of you think a study has a big enough hole in it's methodology, you are free to go design your own study to try to plug the hole or refute the result. A corollary, though, is that there is always going to be a 'hole' big enough for a jackass on the internet to criticize. But the question is whether you're willing to put the skin in the game to see if the result holds when you design a study to exploit the hole.
- wolverine876 5y ago
  > outside mathematics, perfect methodology is impossible Even in mathematics ... others on HN are better qualified than I am to comment, but I've read mathematicians say that almost no reader of a paper works out all the math; to a degree they go by reputation of the author that the mathematics is generally correct. Similarly to your comment, while outsiders envision perfection, people inside have limited time for it. It may be more productive to spend that time on new research than on investigating every little detail of old research.
- jhoelzel 5y ago
  I believe this is only partially true! We call it science because when we find methods that we all agree on, as a culture to use then it becomes scientific. For Instance psychology, in the freudean psychology the methology is still science even though he speaks of urges of "your mother" and interesting things like the "subconciousness". It has been met with great doubt over it's "initiation" but is now a widely acceptable practise. So what has changed? Well the field concentrated on scientific core value which (almost) boil down to: Find a methology, prove that it works and replicated it again.scientific quality criteria. I agree with op! Stimulating cells with electricity is not teaching them. Its assisted muscle reaction. In this case its really clear cut too: if I need to give the braincells electro schocks (the thing they operate on), If anything, I am remote controlling brain cells. The whole idea of perception is flawed in this experiment, you know, because there is no actual perceptor of the paddel to begin with. Sorry but as someone with a bachelor in psychology and 20 years of dev experience the starting poing of "I think many non scientists have a bit of a wrong idea of how science actually works." is just condecending theory. This is literally what "scientific quality criteria" is based on.
SubiculumCode 5y ago
I'll break it down in a couple of bullet points. 1. Science is hard. 2. There is no such thing as a perfect study. Every method/methodology has weaknesses. Look for converging evidence across multiple studies which use different designs. 3. Initial studies often do not have the funding needed to do things as optimally as possible in a single study, but they do provide evidence for ideas that can be pursued further. 4. It is often socially safer to point out deficiencies than to praise strengths. Same as snobs raining down on you when you say you like some midrange wine. 5. Often issues will be raised that are already acknowledged by the authors. 6. Complaints are often based on inaccurate news reporting, not actual research. 7. Science is hard.

runnerup 5y ago

This seems a bit different than what you suggest. The paper shows the "average rally length" and "# of hits per minute" increasing as time goes on for the group which received additional stimulus as feedback when it hit the ball. It did not find an increase in these metrics for groups where the input signal went silent as feedback after hitting the ball, nor for groups which merely continued receiving the usual input ("no feedback"). I have no idea if this is repeatable, but it seems like a valid conclusion from this data that the neuron colonies which were given feedback in the form of extra stimulation, exhibited better performance after the first 5 minutes of each trial, and better performance each subsequent day when they would run another 20 minute trial.

yorwba 5y ago

Their electrode configuration has predetermined non-overlapping sensory and motor areas. So it would be more like having a whole bunch of cats lined up and spraying water onto the cat corresponding to the ball position, while listening for the hissing from some other cats to determine where to move the paddle. The random stimulus as feedback when the ball resets corresponds to spraying the cats unpredictably. To avoid that punishment, the cats need to work together to make the right cats hiss at the right time to intercept the ball with the paddle. (The underlying theory is that cats are okay with being sprayed as long as they can predict it, a.k.a. the Free Energy Principle.) The fact that this feedback increases performance relative to the other experimental conditions indicates that some adaptation is going on in the network.

dr1337 5y ago
That's fascinating. It reminds me of this clip from Adam Curtis's All Watched Over By Machines of Loving Grace where they got lots of random people to play Pong in a theatre in the 90s: https://vimeo.com/78043173 https://vimeo.com/78043173