4 ms·

Implementing Neural Turing Machines

• a_d 8y ago
  “how the memory contents of a NTM are initialized may be a defining factor in the success of a NTM implementation” Does it mean that one can expect better explainability in the future from these models?
  • albertzeyer 8y ago
    The result was that constant/zero initialization was the best, which is the most natural choice anyway (for me at least), and definitely also the most simple option. I'm a bit surprised that they emphasise so much on that. Also, not really sure what to learn from this.
    • stochastic_monk 8y ago
      Does this correspond to an intuition that a tabula rasa may be the best way for a learner to start?
• carlc75 8y ago
  The associated github repo is at https://github.com/MarkPKCollier/NeuralTuringMachine https://github.com/MarkPKCollier/NeuralTuringMachine
• YorkshireSeason 8y ago
  Important and unsurprising sentence from the paper's abstract: "A number of open source implementations of NTMs exist but are unstable during training and/or fail to replicate the reported performance of NTMs"
  • simias 8y ago
    Why do you deem it unsurprising? Do you think that the opensource neural network implementations are subpar?
    • YorkshireSeason 8y ago
      Anecdotally: I've tried to replicate some recent AI/ML papers and failed. So have some of my acquaintances.
    • albertzeyer 8y ago
      I would expect that it was just as unstable for DeepMind. But they can automatically run lots of experiments with different hyper params so that is less an issue for them.
    • nickpsecurity 8y ago
      Some have written about a "reproducibility crisis" in machine learning research: https://petewarden.com/2018/03/19/the-machine-learning-reproducibility-crisis/ https://petewarden.com/2018/03/19/the-machine-learning-repro...
• yters 8y ago
  It'd be interesting to see a Neural TM that can increase its Kolmogorov complexity.
• orbifold 8y ago
  The author states in his github repository (https://github.com/MarkPKCollier/NeuralTuringMachine https://github.com/MarkPKCollier/NeuralTuringMachine) that his work is based on (https://github.com/snowkylin/ntm https://github.com/snowkylin/ntm). If that is the case then I find it kind of strange that he has relicensed the work to MIT from LGPL3 and removed any reference to the original author in the LICENSE. LGPL3 requires any derivative work to be licensed the same and retain mentions of the authors, more over it also requires a clear explanation of the modifications that were undertaken. Also compared to the open source implementation (https://github.com/snowkylin/ntm https://github.com/snowkylin/ntm) it seems like his main novel claim is that he looked at different memory initialisation patterns. Edit: compare the original: https://github.com/snowkylin/ntm/blob/master/ntm/ntm_cell.py https://github.com/snowkylin/ntm/blob/master/ntm/ntm_cell.py to the derivative work: https://github.com/MarkPKCollier/NeuralTuringMachine/blob/master/ntm.py https://github.com/MarkPKCollier/NeuralTuringMachine/blob/ma... from what I can tell the main innovation is that the derivative work uses a named tuple instead of a dictionary for state keeping and there is new memory initialisation code. The original author apparently initialised the memory randomly. I also feel like the paper should cite the implementation they are basing their work on. The paper https://arxiv.org/pdf/1807.08518.pdf https://arxiv.org/pdf/1807.08518.pdf merely states that other implementations exist on page one and makes no mention of the fact that their implementation is based on one of those. Combine that with the fact that they are asking people in the Readme to cite their paper feels like not a very good idea.
  • markpkcollier 8y ago
    I am the author of this work. I was not aware of the difference in the licenses and just used the GitHub default. I have updated it accordingly, thanks for pointing this out. A note on the difference between our work and Snowkylin's: the code implementing the operations of the NTM are very similar and both are similar to other open source NTM implementations as the operations for a NTM are defined by the equations of the original NTM paper and have a clear mapping into code. The primary difference is that our implementation works and is stable - the code changes to achieve this are minimal but this still required substantial experimentation and work to figure out what was causing slow convergence and the gradients becoming NaN (causing training to fail) in other implementations. Thus our primary contribution is not to put a NTM into code but to get that code to reliably and quickly train.