5 ms·

I hinted at this issue a few days ago already here on HN: This is why about 90% of the papers Google publishes are useless. They describe deep learning succes

I hinted at this issue a few days ago already here on HN: This is why about 90% of the papers Google publishes are useless. They describe deep learning successes, with datasets that aren't public, on hardware that isn't public, with software that isn't public. They describe technologies, be it spanner, borg, flumejava or others, relying on closed hard- and software. If an average university professor can't replicate research 1:1 based solely on existing, open technology and the paper, the paper should not be accepted by any journal. In fact, it shouldn't even be accepted on conferences, or be able to get a doi code. That's not worthy of being called science. And it’s problematic when a large amount of AI research happens at Google, Facebook and co., and none of it can be replicated. Science requires replication, if AI research can’t be replicated, it’s not science.

biggodoggo 9y ago
The problem is when BigCo does AI research, they are not doing it for the greater good, they are trying to find a competitive edge. Unfortunately there are no good incentives (besides doing the "right" thing) for companies to open their datasets to the public, so until there is we can expect more of the same...
- IntronExon 9y ago
  Fair enough, so don’t publish them. If you can’t replicate results, it really isn’t science.
- chaosfox 9y ago
  >they are not doing it for the greater good, they are trying to find a competitive edge thats just fine. they don't really need to publish anything, but if they do want to publish it, it should be done in a way other people can reproduce their findings.
- tensor 9y ago
  This isn't fair to researchers at companies. While the goal of the organization may be to make money, the goal of the individual researchers who publish the papers is very likely doing it for the greater good.
  - YeGoblynQueenne 9y ago
    Yeah, I don't see that. If you're an employee of a company (not least one who's paying you a very fat salary) there is an expectation that you will do work that benefits the company's bottom line. If that happens to be aligned with the "greater good" then OK, but there is no such guarantee- and if it isn't... well then your research will only benefit the company.
    - tensor 9y ago
      Of course your work benefits the bottom line, but being able to publish is often a perk that companies have to give to be able to attract and keep people who care about those things. It's actually incredibly insulting that you would assume you know the motivations of individual employees. The world isn't as black and white as you'd like it to be.
      - janekm 9y ago
        If journals/conferences were to refuse un-replicable results, companies that wish to offer this perk would have to make a sufficient part of their dataset public in order to offer this perk to their employees. This would be a good thing.
carlmr 9y ago
I agree, I'd even go as far to say that not publishing non replicable results is better for research than publishing it. These papers are noise that is pointed to for marketing purposes. A higher signal-to-noise ratio will lead to faster research, because people don't have to sift through 10 papers to find one good one.
2sk21 9y ago
The other problem is that academics simply may not have the computing resources to replicate anything that Google does anyway even if everything was open.
bluetwo 9y ago
If that's true, then you shouldn't read any journals that don't publish data and methods. You've got to vote with your feet (or in this case, attention).

biggodoggo 9y ago

The problem is when BigCo does AI research, they are not doing it for the greater good, they are trying to find a competitive edge. Unfortunately there are no good incentives (besides doing the "right" thing) for companies to open their datasets to the public, so until there is we can expect more of the same...

IntronExon 9y ago
Fair enough, so don’t publish them. If you can’t replicate results, it really isn’t science.
chaosfox 9y ago
>they are not doing it for the greater good, they are trying to find a competitive edge thats just fine. they don't really need to publish anything, but if they do want to publish it, it should be done in a way other people can reproduce their findings.
tensor 9y ago
This isn't fair to researchers at companies. While the goal of the organization may be to make money, the goal of the individual researchers who publish the papers is very likely doing it for the greater good.
- YeGoblynQueenne 9y ago
  Yeah, I don't see that. If you're an employee of a company (not least one who's paying you a very fat salary) there is an expectation that you will do work that benefits the company's bottom line. If that happens to be aligned with the "greater good" then OK, but there is no such guarantee- and if it isn't... well then your research will only benefit the company.
  - tensor 9y ago
    Of course your work benefits the bottom line, but being able to publish is often a perk that companies have to give to be able to attract and keep people who care about those things. It's actually incredibly insulting that you would assume you know the motivations of individual employees. The world isn't as black and white as you'd like it to be.
    - janekm 9y ago
      If journals/conferences were to refuse un-replicable results, companies that wish to offer this perk would have to make a sufficient part of their dataset public in order to offer this perk to their employees. This would be a good thing.

carlmr 9y ago

I agree, I'd even go as far to say that not publishing non replicable results is better for research than publishing it. These papers are noise that is pointed to for marketing purposes. A higher signal-to-noise ratio will lead to faster research, because people don't have to sift through 10 papers to find one good one.

2sk21 9y ago

The other problem is that academics simply may not have the computing resources to replicate anything that Google does anyway even if everything was open.

bluetwo 9y ago

If that's true, then you shouldn't read any journals that don't publish data and methods. You've got to vote with your feet (or in this case, attention).