4 ms·

I don't find this very convincing. Continuations are well known to be completely mystifying. I wrote up what I believe to be a much simpler reverse mode AD tr

by tome 7y ago

I don't find this very convincing. Continuations are well known to be completely mystifying. I wrote up what I believe to be a much simpler reverse mode AD transformation that uses only basic concepts: http://h2.jaguarpaw.co.uk/posts/automatic-differentiation-worked-examples/ http://h2.jaguarpaw.co.uk/posts/automatic-differentiation-wo...

samth 7y ago
Your notes, while simple, don't cover many of the things the paper does, such as binding or functions or sharing.
- tome 7y ago
  You're right they don't cover functions. I don't understand what about binding or sharing they don't deal with though. Could you elaborate?
  - tiarkrompf 7y ago
    For starters, what if you want to differentiate through code that traverses a collection, for example: val x = ... ys.map(y => y + x) Each loop iteration needs to contribute a gradient update to x, which is defined in an outer scope. And what if y => y + x is not given as an inline lambda, but defined elsewhere. It doesn't seem like your blog posts discusses any such cases.
    - tome 7y ago
      Correct, it doesn't discuss that case, indeed it doesn't treat user-defined functions at all. I just don't understand in what sense it doesn't treat binding or sharing.
bjz_ 7y ago
> Continuations are well known to be completely mystifying. Could you elaborate on what leads you to think this?
- tome 7y ago
  I don't believe that continuations are a corner of programming language theory that is well-loved and and well-understood by programmers at large, are they? Even amongst experienced functional programmers they seem to be thought of as rather difficult. Oleg Kiselyov has written extensively on them but his work has not been brought to the masses in the same way that his work in iteratees has. http://okmij.org/ftp/continuations/index.html http://okmij.org/ftp/continuations/index.html
  - bjz_ 7y ago
    Ahhh! Right! See, the context here is important. Continuations are pretty widely known and important in the context of programming language implementation and theory, which is why I was confused by your comment. In the context where this paper was presented (to programming language researchers), making the link between delimited continuations _would_ demystify a bunch, and could allow for transferring of knowledge and techniques between the fields.
    - tome 7y ago
      Having looked again, the authors do explain why they believe that continuations (as opposed to simply higher-order functions) are necessary. See my other comment at https://news.ycombinator.com/item?id=22347428 https://news.ycombinator.com/item?id=22347428.

by tome 7y ago

samth 7y ago

Your notes, while simple, don't cover many of the things the paper does, such as binding or functions or sharing.

tome 7y ago
You're right they don't cover functions. I don't understand what about binding or sharing they don't deal with though. Could you elaborate?
- tiarkrompf 7y ago
  For starters, what if you want to differentiate through code that traverses a collection, for example: val x = ... ys.map(y => y + x) Each loop iteration needs to contribute a gradient update to x, which is defined in an outer scope. And what if y => y + x is not given as an inline lambda, but defined elsewhere. It doesn't seem like your blog posts discusses any such cases.
  - tome 7y ago
    Correct, it doesn't discuss that case, indeed it doesn't treat user-defined functions at all. I just don't understand in what sense it doesn't treat binding or sharing.

bjz_ 7y ago

> Continuations are well known to be completely mystifying. Could you elaborate on what leads you to think this?

tome 7y ago
I don't believe that continuations are a corner of programming language theory that is well-loved and and well-understood by programmers at large, are they? Even amongst experienced functional programmers they seem to be thought of as rather difficult. Oleg Kiselyov has written extensively on them but his work has not been brought to the masses in the same way that his work in iteratees has. http://okmij.org/ftp/continuations/index.html http://okmij.org/ftp/continuations/index.html
- bjz_ 7y ago
  Ahhh! Right! See, the context here is important. Continuations are pretty widely known and important in the context of programming language implementation and theory, which is why I was confused by your comment. In the context where this paper was presented (to programming language researchers), making the link between delimited continuations _would_ demystify a bunch, and could allow for transferring of knowledge and techniques between the fields.
  - tome 7y ago
    Having looked again, the authors do explain why they believe that continuations (as opposed to simply higher-order functions) are necessary. See my other comment at https://news.ycombinator.com/item?id=22347428 https://news.ycombinator.com/item?id=22347428.