4 ms·

Why would a language where it is possible to manipulate certain functions to get their derivatives require a new revision control system or IDE? And speaking o

Why would a language where it is possible to manipulate certain functions to get their derivatives require a new revision control system or IDE? And speaking of Lisp - wasn't symbolic differentiation a fairly common thing in Lisp? (basically as a neat example of what you can do once your code is easy to manipulate as data).

joaogui1 7y ago
For the IDE part they have written some cool debuggers that help you understand the differentiation part and catch bugs on it. But I'm not sure why you couldn't just use the debugger, instead of a whole new IDE, much less why you would need a new RCS
noelwelsh 7y ago
Symbolic differentiation is a fairly common exercise but it is inefficient (the size of the derivative grows exponentially in the size of the original expression). "Automatic differentiation" is the term for the class of algorithms usually used in practice, which are more efficient while still being exact (for whatever "exact" means when you're using floating point :-)
- arethuza 7y ago
  Thanks - I didn't think it was literally doing symbolic differentiation (I don't work in the area so literally had no idea) - but the basic idea that you apply some process to your code to get some other code doesn't sound that surprising to anyone who has used lisp (and I used to write tools in lisp to write numerical engineering simulations - admittedly a long time ago)
- thechao 7y ago
  AD still explodes for “interesting” derivatives: efficiently computing the adjoint of the Jacobian is NP-complete. And, naturally, the Jacobian is what you want when doing machine learning. There’s papers from the mid-90’s discussing the difficulties in adding AD Jacobian operators to programming languages to support neural networks. This article is just rehashing 25 year old problems.
  - moultano 7y ago
    The Jacobian is used all the time, but where do you end up needing the adjoint?
  - jedbrown 7y ago
    Correction: Finding the optimal algorithm (minimal number of operations) for computing a Jacobian is NP-complete, but evaluating it in a multiple of the cost of a forward evaluation is standard. Also, many optimizers that are popular in ML only need gradients (in which case the Jacobian is just the gradient vector). Second order methods are important in applications with ill-conditioning (such as bundle adjustment or large-scale GPR), but they have lots of exploitable structure/sparsity. The situation is not nearly as dire as you suggest.
    - thechao 7y ago
      Yep; I was imprecise!
  - hinkley 7y ago
    Around 2000 I was accidentally inventing a Bloom filter variant (to this day I don’t know how I missed the Google papers at the time) for doing a large set intersection test between two machines. Somehow, I ended up with a calculus equation for determining the right number of bits per entry and rounds to do to winnow the lists, for any given pair of machines where machine A found n entries and machine B found m. But I couldn’t solve it. Then I discovered that even though I did poorly at calculus, I still remembered more than anyone else on the team, and then couldn’t find help from any other engineer in the building either. Eventually I located a QA person who used to TA calculus. She informed me that my equation probably could not be solved by hand. I gave it another day or so and then gave up. If I couldn’t do it by hand I wasn’t going to be able to write a heuristic for it anyway. For years, this would be the longest period in my programming career where I didn’t touch a computer. I just sat with pen and paper pounding away at it and getting nowhere. And that’s also the last time I knowingly touched calculus at work. (although you might argue some of my data vis discussions amount to determining whether we show the either the sum or rate of change of a trend line to explain it better. The S curve that shows up so often in project progress charts is just the integral of a normal distribution, after all)
- kgwgk 7y ago
  It’s also easy to do in lisp: https://people.eecs.berkeley.edu/~fateman/papers/ADIL.pdf https://people.eecs.berkeley.edu/~fateman/papers/ADIL.pdf
- jjgreen 7y ago
  exp(x) ?
- steerablesafe 7y ago
  > the size of the derivative grows exponentially in the size of the original expression. Only if you treat it as a tree, not a DAG. edit: Sorry no, it's still linear, even for a tree.
davedx 7y ago
A quick Google revealed DVC (Data Version Control): https://dvc.org/ https://dvc.org/ Smashing your ML models into git or other text-based VCS probably isn't the best way to do it
- jakear 7y ago
  Its not clear how thats different from git's LFS.
  - arethuza 7y ago
    To be fair they do have a page giving a comparison with git LFS and other related approaches to storing large files: https://dvc.org/doc/understanding-dvc/related-technologies#git-lfs-large-file-storage https://dvc.org/doc/understanding-dvc/related-technologies#g... Mind you - DVC seems to be a platform on top of git rather than a replacement for git. So I'd argue that it's not really a new revision control system.
  - dchichkov 7y ago
    It is different. DVC is a server-less management tool that helps you organize and link to your storage backends. And move data from these backends to your workspace. Git LFS requires a dedicated server. And you can store data only on that server, instead of moving data between a multitude of storage backends (like Google Drive, S3, GCP, local drive).

joaogui1 7y ago

For the IDE part they have written some cool debuggers that help you understand the differentiation part and catch bugs on it. But I'm not sure why you couldn't just use the debugger, instead of a whole new IDE, much less why you would need a new RCS

noelwelsh 7y ago

Symbolic differentiation is a fairly common exercise but it is inefficient (the size of the derivative grows exponentially in the size of the original expression). "Automatic differentiation" is the term for the class of algorithms usually used in practice, which are more efficient while still being exact (for whatever "exact" means when you're using floating point :-)

arethuza 7y ago
Thanks - I didn't think it was literally doing symbolic differentiation (I don't work in the area so literally had no idea) - but the basic idea that you apply some process to your code to get some other code doesn't sound that surprising to anyone who has used lisp (and I used to write tools in lisp to write numerical engineering simulations - admittedly a long time ago)
thechao 7y ago
AD still explodes for “interesting” derivatives: efficiently computing the adjoint of the Jacobian is NP-complete. And, naturally, the Jacobian is what you want when doing machine learning. There’s papers from the mid-90’s discussing the difficulties in adding AD Jacobian operators to programming languages to support neural networks. This article is just rehashing 25 year old problems.
- moultano 7y ago
  The Jacobian is used all the time, but where do you end up needing the adjoint?
- jedbrown 7y ago
  Correction: Finding the optimal algorithm (minimal number of operations) for computing a Jacobian is NP-complete, but evaluating it in a multiple of the cost of a forward evaluation is standard. Also, many optimizers that are popular in ML only need gradients (in which case the Jacobian is just the gradient vector). Second order methods are important in applications with ill-conditioning (such as bundle adjustment or large-scale GPR), but they have lots of exploitable structure/sparsity. The situation is not nearly as dire as you suggest.
  - thechao 7y ago
    Yep; I was imprecise!
- hinkley 7y ago
  Around 2000 I was accidentally inventing a Bloom filter variant (to this day I don’t know how I missed the Google papers at the time) for doing a large set intersection test between two machines. Somehow, I ended up with a calculus equation for determining the right number of bits per entry and rounds to do to winnow the lists, for any given pair of machines where machine A found n entries and machine B found m. But I couldn’t solve it. Then I discovered that even though I did poorly at calculus, I still remembered more than anyone else on the team, and then couldn’t find help from any other engineer in the building either. Eventually I located a QA person who used to TA calculus. She informed me that my equation probably could not be solved by hand. I gave it another day or so and then gave up. If I couldn’t do it by hand I wasn’t going to be able to write a heuristic for it anyway. For years, this would be the longest period in my programming career where I didn’t touch a computer. I just sat with pen and paper pounding away at it and getting nowhere. And that’s also the last time I knowingly touched calculus at work. (although you might argue some of my data vis discussions amount to determining whether we show the either the sum or rate of change of a trend line to explain it better. The S curve that shows up so often in project progress charts is just the integral of a normal distribution, after all)
kgwgk 7y ago
It’s also easy to do in lisp: https://people.eecs.berkeley.edu/~fateman/papers/ADIL.pdf https://people.eecs.berkeley.edu/~fateman/papers/ADIL.pdf
jjgreen 7y ago
exp(x) ?
steerablesafe 7y ago
> the size of the derivative grows exponentially in the size of the original expression. Only if you treat it as a tree, not a DAG. edit: Sorry no, it's still linear, even for a tree.

davedx 7y ago

A quick Google revealed DVC (Data Version Control): https://dvc.org/ https://dvc.org/ Smashing your ML models into git or other text-based VCS probably isn't the best way to do it

jakear 7y ago
Its not clear how thats different from git's LFS.
- arethuza 7y ago
  To be fair they do have a page giving a comparison with git LFS and other related approaches to storing large files: https://dvc.org/doc/understanding-dvc/related-technologies#git-lfs-large-file-storage https://dvc.org/doc/understanding-dvc/related-technologies#g... Mind you - DVC seems to be a platform on top of git rather than a replacement for git. So I'd argue that it's not really a new revision control system.
- dchichkov 7y ago
  It is different. DVC is a server-less management tool that helps you organize and link to your storage backends. And move data from these backends to your workspace. Git LFS requires a dedicated server. And you can store data only on that server, instead of moving data between a multitude of storage backends (like Google Drive, S3, GCP, local drive).