4 ms·

After few years of research in DL I learned not to trust any single paper at all. 99% of DL papers are not scientific but rather 'hey guys, look this trick is o

by novaRom 10y ago

After few years of research in DL I learned not to trust any single paper at all. 99% of DL papers are not scientific but rather 'hey guys, look this trick is our new awesome discovery'. Also I learned to communicate with other teams and exchange with ideas proved by practices - this really helps. To improve, I would suggest to publish whole setup, all the parameters used, the programming code, and publish either all the data or a reference to a large free data set (no MNIST anymore in papers, please).

daveguy 10y ago
> (no MNIST anymore in papers, please) If you have a breakthrough in transfer learning then you will be able to very effectively demonstrate it with MNIST. The race to the bottom is essentially over, but that doesn't mean MNIST can't be used to demonstrate learning. Regarding setup and parameters. I hope AI researchers move toward something like pachyderm [https://pachyderm.io https://pachyderm.io] -- providing a single docker image to completely replicate their work. However, I sincerely doubt that will happen. As "open" as research is the details are almost always obfuscated to prevent competition with the spin-out company (or other researchers).
- AIMunchkin 10y ago
  How about unlocking access to ImageNet? Unless one has a .edu account, its overlords seem to ignore requests to access it. Mind you, it's relatively easy to social engineer access to it, but why should this be necessary? OpenAI and Google have both knocked it out of the park with easy access to datasets and examples. But sadly, IMO at the amateur-level, TensorFlow considered harmful. I have repeatedly observed novices blow the thing up by starting from one of many of its amazing and fantastic teaching examples. It's not a question of the TensowFlow API, but rather of the engineering quality of its underlying engine, which kind of sucks. Nothing ruins an enthusiastic data scientist's day like a cryptic seg fault for no apparent reason whatsoever. And I know they're working on it, but fer cryin' out loud, the API is great, and Google has the bottomless pockets to do a lot better than this. It's been over a year and I still see people throw their hands up in frustration trying to make use of the thing. Of course, Google has never been a customer-driven company, but if we don't want an AI Fall, methinks this needs to be fixed.
  - daveguy 10y ago
    I agree. I think all reference datasets should be free and open source. Also, researchers shouldn't publish on datasets that are not free and open source. That is a basic requirement for repeatability. As far as the TensorFlow API is concerned. This may be a tradeoff between speed and robustness. In order to have every operation checked every time would certainly slow down the code for general use. Probably better how-to / setup / use guides are a better solution for this (unless it is a flat out bug).

by novaRom 10y ago

daveguy 10y ago

> (no MNIST anymore in papers, please) If you have a breakthrough in transfer learning then you will be able to very effectively demonstrate it with MNIST. The race to the bottom is essentially over, but that doesn't mean MNIST can't be used to demonstrate learning. Regarding setup and parameters. I hope AI researchers move toward something like pachyderm [https://pachyderm.io https://pachyderm.io] -- providing a single docker image to completely replicate their work. However, I sincerely doubt that will happen. As "open" as research is the details are almost always obfuscated to prevent competition with the spin-out company (or other researchers).

AIMunchkin 10y ago
How about unlocking access to ImageNet? Unless one has a .edu account, its overlords seem to ignore requests to access it. Mind you, it's relatively easy to social engineer access to it, but why should this be necessary? OpenAI and Google have both knocked it out of the park with easy access to datasets and examples. But sadly, IMO at the amateur-level, TensorFlow considered harmful. I have repeatedly observed novices blow the thing up by starting from one of many of its amazing and fantastic teaching examples. It's not a question of the TensowFlow API, but rather of the engineering quality of its underlying engine, which kind of sucks. Nothing ruins an enthusiastic data scientist's day like a cryptic seg fault for no apparent reason whatsoever. And I know they're working on it, but fer cryin' out loud, the API is great, and Google has the bottomless pockets to do a lot better than this. It's been over a year and I still see people throw their hands up in frustration trying to make use of the thing. Of course, Google has never been a customer-driven company, but if we don't want an AI Fall, methinks this needs to be fixed.
- daveguy 10y ago
  I agree. I think all reference datasets should be free and open source. Also, researchers shouldn't publish on datasets that are not free and open source. That is a basic requirement for repeatability. As far as the TensorFlow API is concerned. This may be a tradeoff between speed and robustness. In order to have every operation checked every time would certainly slow down the code for general use. Probably better how-to / setup / use guides are a better solution for this (unless it is a flat out bug).