9 ms·
In all honesty, this sounds to me like a whole lot of BS and hype. The name is pretentious, the quotes are ridiculous ("Deep Learning est mort. Vive Differentia
by gonzus 7y ago
In all honesty, this sounds to me like a whole lot of BS and hype. The name is pretentious, the quotes are ridiculous ("Deep Learning est mort. Vive Differentiable Programming."; "there will be a need for the creation of a whole new set of new tools, such as a new Git, new IDEs, and of course new programming languages"). Maybe I am just ignorant and fail to grasp the importance of such capabilities in a machine learning context, but I have to say, the grandstanding is a bit grating.
Who knows, perhaps this will become the greatest thing since Lisp... What do I know.
- pjc50 7y ago> need for the creation of a whole new set of new tools, such as a new Git, new IDEs, and of course new programming languages The greatest possible barrier to adoption.
- baybal2 7y agoDeep learning is the new Object Oriented Programming and Service Oriented Architecture
- DeathArrow 7y agoWhat about SPA and serverless?
- arethuza 7y agoWhy would a language where it is possible to manipulate certain functions to get their derivatives require a new revision control system or IDE? And speaking of Lisp - wasn't symbolic differentiation a fairly common thing in Lisp? (basically as a neat example of what you can do once your code is easy to manipulate as data).
- joaogui1 7y agoFor the IDE part they have written some cool debuggers that help you understand the differentiation part and catch bugs on it. But I'm not sure why you couldn't just use the debugger, instead of a whole new IDE, much less why you would need a new RCS
- noelwelsh 7y agoSymbolic differentiation is a fairly common exercise but it is inefficient (the size of the derivative grows exponentially in the size of the original expression). "Automatic differentiation" is the term for the class of algorithms usually used in practice, which are more efficient while still being exact (for whatever "exact" means when you're using floating point :-)
- arethuza 7y agoThanks - I didn't think it was literally doing symbolic differentiation (I don't work in the area so literally had no idea) - but the basic idea that you apply some process to your code to get some other code doesn't sound that surprising to anyone who has used lisp (and I used to write tools in lisp to write numerical engineering simulations - admittedly a long time ago)
- thechao 7y agoAD still explodes for “interesting” derivatives: efficiently computing the adjoint of the Jacobian is NP-complete. And, naturally, the Jacobian is what you want when doing machine learning. There’s papers from the mid-90’s discussing the difficulties in adding AD Jacobian operators to programming languages to support neural networks. This article is just rehashing 25 year old problems.
- moultano 7y agoThe Jacobian is used all the time, but where do you end up needing the adjoint?
- jedbrown 7y agoCorrection: Finding the optimal algorithm (minimal number of operations) for computing a Jacobian is NP-complete, but evaluating it in a multiple of the cost of a forward evaluation is standard. Also, many optimizers that are popular in ML only need gradients (in which case the Jacobian is just the gradient vector). Second order methods are important in applications with ill-conditioning (such as bundle adjustment or large-scale GPR), but they have lots of exploitable structure/sparsity. The situation is not nearly as dire as you suggest.
- davedx 7y agoA quick Google revealed DVC (Data Version Control): https://dvc.org/ https://dvc.org/ Smashing your ML models into git or other text-based VCS probably isn't the best way to do it
- jakear 7y agoIts not clear how thats different from git's LFS.
- arethuza 7y agoTo be fair they do have a page giving a comparison with git LFS and other related approaches to storing large files: https://dvc.org/doc/understanding-dvc/related-technologies#git-lfs-large-file-storage https://dvc.org/doc/understanding-dvc/related-technologies#g... Mind you - DVC seems to be a platform on top of git rather than a replacement for git. So I'd argue that it's not really a new revision control system.
- dchichkov 7y agoIt is different. DVC is a server-less management tool that helps you organize and link to your storage backends. And move data from these backends to your workspace. Git LFS requires a dedicated server. And you can store data only on that server, instead of moving data between a multitude of storage backends (like Google Drive, S3, GCP, local drive).
- fathead_glacier 7y agoYou can refer to a talk about Differentiable Programming here: https://www.youtube.com/watch?v=Sv3d0k7wWHk https://www.youtube.com/watch?v=Sv3d0k7wWHk the talk is for Julia, albeit the principles are general. If you skip to https://youtu.be/Sv3d0k7wWHk?t=2877 https://youtu.be/Sv3d0k7wWHk?t=2877 you can see a question asked: why would you need this. In essence there are cases outside the well developed uses (CNN, LSTM etc.) such as Neural ODEs where you need to mix different tools (ODE solvers and neural networks) and the ability to do Differentiable Programming is helpful otherwise it is harder the get gradients. The way I can see it being useful is that it helps speed up development work so we can explore more architectures, again Neural ODEs being a great example.
- pabl0rg 7y agoIs it very different from probabilistic programming (a term that is both older and easier to understand)? Erik Meijer gave two great talks on the concept. https://m.youtube.com/watch?v=NKeHrApPWlo https://m.youtube.com/watch?v=NKeHrApPWlo https://m.youtube.com/watch?v=13eYMhuvmXE https://m.youtube.com/watch?v=13eYMhuvmXE
- zeroxfe 7y agoYes, these are two very different things. Differential programming is about building software that is differentiable end-to-end, so that optimal solutions can be calculated with gradient descent. Probabilistic programming (which is a bit more vague) is about specifying probabilistic models in an elegant and consistent way (which than then be used for training and inference.) So, you can build some kinds of probabilistic programs with differential programming languages, but not vice versa.
- MiroF 7y agoBe hard on the article, but easier on the concept - I think there is a lot of potential for differential programming w/ Swift, but this article is not a good advocate.
- jasode 7y ago>The name is pretentious, If we already have useful phrases like "embedded programming", "numerical programming", "systems programming", or "CRUD programming", etc, I'm not seeing the pretentiousness of "differentiable programming". If you program embedded chips, we often call it "embedded programming"; likewise, if you write programs where differentials are a 1st-class syntax concept, I'm not seeing the harm in calling it "differentiable programming" -- because that basically describes the specialization. >the quotes are ridiculous ("Deep Learning est mort. Vive Differentiable Programming."; Fyi, it's a joking type of rhetorical technique called a "snowclone". Previous comment about a similar phrase: https://news.ycombinator.com/item?id=11455219 https://news.ycombinator.com/item?id=11455219
- BoorishBears 7y agoI feel like calling it a snowclone is a stretch. The whole point is you’re supposed to have the same “something” on both sides (X is dead, long live X), to indicate it’s not a totally new thing but a significant shift in how it’s done The most well known one: > The King is dead. Long live the King! If you change one side, you’re removing the tongue-in-cheek nature of it, and it does sound pretty pretentious.
- Allezxandre 7y agoNot really. The sentence with the King was used to mean that the new King immediately got into function. > Le Roi (Louis ##) est mort. Vive le Roi (Louis ## + 1) ! Using the sentence with Deep Learning and Differential Learning just suggests that Differential Learning is the heir/successor/evolution of Deep Learning. It does not imply that they are the same thing. As a French person used to the saying, Le Cun probably meant that.
- BoorishBears 7y ago> Using the sentence with Deep Learning and Differential Learning just shows how Differential Learning is an evolution. It does not imply that they are the same thing. ... where did I imply it means they're the same thing? From my comment: > indicate it’s not a totally new thing but a significant shift in how it’s done You could say, an evolution? - A snowclone is a statement in a certain form. The relevant form that English speakers use (I'm not a French person, and this is an English article) is "X is dead, long live X", where both are X. That's where the "joking" the above comment is referring to comes from, it sounds "nonsensical" if you take it literally. If you change one X to Y, suddenly there's no tongue-in-cheek aspect, you're just saying "that thing sucks, this is the new hotness". I suspect the author just missed that nuance or got caught up in their excitement, but the whole point of a snowclone is it has a formula, and by customizing the variable parts of that formula, you add a new subtle meaning or tint to the statement.
- tchaffee 7y agoThe article gives specific and reasonable motivations for why you might want a new language for machine learning. There are already new tools. Like Pytorch. If you have never used Jupyter notebooks instead of an IDE, give it a try for a few weeks. It was the biggest boost I've seen to my coding productivity in literally decades. A new Git? I don't quite get that one. But considering the author arguably already got three out of his four claims right, maybe there is some reasoning behind that Git claim too.
- BiteCode_dev 7y agoMaybe versioning for models?
- lonesword 7y ago> If you have never used Jupyter notebooks instead of an IDE, give it a try for a few weeks. It was the biggest boost I've seen to my coding productivity in literally decades Really? I seem to shoot myself in the foot a lot with jupyter notebook. I can't count the number of times my snippet was not working and it was because I was reusing some variable name from some other cell that no longer exists. The amount of bugs I get in a notebook is ridiculous. Of course, I'm probably using it wrong
- TACIXAT 7y agoI could say the same about Python, honestly. Nonexistent variables, incorrectly typed variables, everything-is-function-scoped variables.
- tchaffee 7y agoYeah really, but I'm a very experienced dev so what I'm getting from it is likely very different from your experience. Consider looking into git or some other version control. If you are deleting stuff that breaks your code, you want to be able to go back to a working version, or at least look at the code from the last working version so you can see what how you broke it.
- FridgeSeal 7y agoImo people’s love of Jupyter notebooks is another one of those “this is My Thing and I love it despite the flaws” situations. Jupyter notebooks are painful to read, the allow you to do silly stuff all too easily, they’re nightmarish to debug, don’t play well with git, and almost every. Single. One. I’ve ever seen my teammates write eschewed almost every software engineering principle possible. You’re not using them wrong, they shepard you to working very “fast and loose” and that’s a knife edge that you have to hope gets you to your destination before everything falls apart at the seams.
- solinent 7y agoDifferentiable programming allows one to specify any parametrized function and allows one to use optimization to learn the objective. There is definitely some need for an EDSL of some sort, but I think a general method is pretty useless. Being able to arbitrarily come up with automatic jacobians for a function isn't really language specific, and usually much better results are obtained using manually calculated jacobians. By starting from scratch you lose all the language theory poured into all the pre-existing languages. I'm sure there'll be a nice haskell version that works in a much simpler manner. Here's a good start: https://github.com/hasktorch/hasktorch/blob/master/examples/image-processing/Main.hs https://github.com/hasktorch/hasktorch/blob/master/examples/... I think it's pretty trivial to generalize and extend it beyond multilinear functions.
- abra1010 7y agoYou're right, it sounds like BS because it's BS. Swift was railroaded in Google by Chris Lattner, who has since left Google and S4TF is on death watch. No one is really using it and it hasn't delivered anything useful in 2.5 years
- MiroF 7y agoSometimes things take a little longer to develop. I don't know who will create it, but from my perspective, the need for a statically typed "differentiable" language is extremely high and C++ is not it.
- abra1010 7y ago> the need for a statically typed "differentiable" language is extremely high This is not what Google has found, actually. Teams who wanted to use this for research found that a static language is not flexible enough when they want to generate graphs at runtime. This is apparently pretty common these days, and obviously Python allows it. Especially with JAX that traces code for autodiff
- MiroF 7y ago> This is not what Google has found, actually. Or has at least found that existing solutions for statically typed "differentiable" programming are ineffective, and I'd agree. But having some way to check types/properties of tensors that you are doing operations to would really help to make sure you don't get your one hidden dimension accidentally switched with the other or something. Some of these problems are silent and need something other than dynamic runtime checking to find them, even if it's just a bolt-on type checker to python. There are a lot of issues with our current approach of just using memory and indexed dimensions. [0] [0]: http://nlp.seas.harvard.edu/NamedTensor http://nlp.seas.harvard.edu/NamedTensor
- danielscrubs 7y agoFlexible enough or just... former statisticians have enough on their plate than learning programming so lets use the simplest popular language in existence?
- blackandblue 7y agohow do you get formatting to work in hn?
- todd8 7y agoThere a just a few simple rules, see https://news.ycombinator.com/formatdoc https://news.ycombinator.com/formatdoc I would add that verbatim text is sometimes hard to read because it doesn't wrap long lines and small screens require the reader to scroll horizontally so try not to use it for large/wide blocks of text. Also bullet lists are usually written as separate, ordinary paragraphs for each item with an asterisk or dash as the paragraphs first character.
- blackandblue 7y agoappreciated!
- hinkley 7y agoA particularly salty developer I knew ages and ages ago once said (apocryphally, apparently) that there was an old Inuit proverb, “everyone likes the smell of their own farts.” Google is still taking themselves very seriously while everyone else is starting to get bored. The problem with being 25 is that you have about 8 years ahead of you before you figure out how full of shit everyone is in their twenties, and maybe another 8 before you figure out that everyone is full of shit and stop worrying quite so much about it.
- dreamcompiler 7y agoSturgeon's law applies both spatially and temporally.
- MaysonL 7y ago"Point of view is worth 80 IQ points." Make that ±80…
- lamontcg 7y agoThis doesn't seem to be crap, but it does seem to be hype. You can use CasADI to do automatic differentiation in C++, Python and Matlab today: https://web.casadi.org/ https://web.casadi.org/ Tight integration with the language may be beneficial in making it simpler to write, but its not like you can't do this already in other languages. Baking it into the language might be useful to make it more popular. Nobody should be doing the chain rule by hand in the 21st century.
- iainmerrick 7y agoA version of that saying I like is: “success is like a fart; only your own smells good.”
- Upvoter33 7y agoWho in the world thinks their own farts smell good? Yeesh.