7 ms·

Differentiable Programming Mega-Proposal

162 points by xmmrm 7y ago

taliesinb 7y ago
Capitalizing on the presence of people who might be new to Automatic Differentiation and want a deeper understanding of how it works, here is an interactive Colab notebook I wrote about this topic entitled “Build your Own TensorFlow” for the Deep Learning Indaba that just happened in Kenya: https://colab.research.google.com/drive/14GeXkFd5pQKKNIJ7BMswP0ihlYg5nIbS#forceEdit=true&offline=true&sandboxMode=true https://colab.research.google.com/drive/14GeXkFd5pQKKNIJ7BMs...
- Donald 7y ago
  That's an excellent tutorial - more are available on http://www.deeplearningindaba.com/practicals-2019.html http://www.deeplearningindaba.com/practicals-2019.html for those interested.
wenc 7y ago
I'm curious (for folks in the know): how does differentiable programming handle non-differentiable points? Can it detect non-differentiable/non-smooth functions? Non-smooth functions like abs(), max(), min() have points where derivatives do not exist. ReLU functions are non-differentiable at their hinge points. Disjoint IF-THEN-ELSE conditions are discontinuities in the function space, and are traditionally handled in optimization with mixed-integer formulations (i.e. split up the space and do something clever like branch-and-bound to find the optimum).
- abeppu 7y ago
  My understanding is you don't explicitly handle that. For whichever place you're evaluating, you have some computation graph, and you use the elementary operators on that graph. So, if you have defined ReLU with an `if x >= 0 return x` clause, then if you evaluate the derivative at 0, that's the branch you go down, and you say the answer is 1.
- taliesinb 7y ago
  While the gradients don’t technically exist, you usually use the limiting value from one side (e.g for the commonly used ReLU activation, you can use either 0 or 1). This is justified by the observation that randomly initialized networks are extremely unlikely to encounter these particular values without some kinder of deeper conspiracy happening. You could put this on a mathematical footing by saying the set of non differentiable points has measure zero, for example.
  - abeppu 7y ago
    > randomly initialized networks But "networks" here, you're thinking of ANNs, yes? But in the context of proposing differential programming as an addition to a general purpose language (and where the proposal explicitly brings up a bunch of cases outside of deep learning), is it fair to justify behavior based on what makes sense in a popular but narrow application?
    - taliesinb 7y ago
      It’s a good question what the plans are for DP languages to handle situations where non-differentiability shouldn’t be ignored. For sensitivity analysis it might be disastrous to conclude that an output is sensitive to an input when it is actually not, merely because an intermediary ReLU hit 0, for example. A conservative approach could be to define versions of the relevant functions that threw exceptions at such points, or that also calculated the trusted margin of the resulting gradients; non-differentiability would then produce a zero trust margin.
    - mlevental 7y ago
      he/she addressed that - the points at which the function isn't differentiable has measure zero. besides this isn't some kind of new hack - one sided limits (and therefore derivatives) were invented exactly for such cases (min, max, abs) and have been used by mathematicians probably since just about when calculus was invented.
- deleted 7y ago
  [deleted]
- chillee 7y ago
  Other people have answered what they do, but this is the big gap between people talking about 'differentiable programming' in theory, and having it actually work in practice. It's true that once you have control flow, the gradient quickly becomes meaningless. I posted an example here: https://news.ycombinator.com/item?id=20892287 https://news.ycombinator.com/item?id=20892287 That's also the biggest reason I tend to find much of this "differentiable programming" stuff to be overhyped. It's hard to reformulate programs in a way s.t. the derivative can mean something meaningful. And I'm not convinced traditional languages will benefit. That's not to say that there isn't cases where your program can be formulated to have a meaningful derivative. See this differentiable ray tracer: https://people.csail.mit.edu/tzumao/diffrt/ https://people.csail.mit.edu/tzumao/diffrt/
  - taliesinb 7y ago
    > It's hard to reformulate programs in a way s.t. the derivative can mean something meaningful. Really? The gradients computed by AD are the exact answer to the following question: if I were to change this input or parameter an infinitesimal amount, how much would it change the output of my function? That is always meaningful (when it is defined), and means what I just said. You can easily make functions where it is not defined, of course, just like you can make a sphere into two spheres with the Banach-Tarski theorem! But there are vast, vast forests of numerical computation employed in industry, science, finance, engineering, where it is almost always defined. And even for more “chunky” computations where the non-differentiability is more severe, there are algorithms like REINFORCE that you can use to estimate gradients through these parts.
    - chillee 7y ago
      It's not meaningful in the sense that it won't correspond with your intuition of "will increasing the input increase my output". Presumably the point of differentiable programming isn't just getting the derivative for fun, it's for optimizing some quantity. For example, take this code. x,y for (int i=0; i<x; i++) y += 1 return y It's technically true that the gradient of 0 is correct (modulo boundaries). But if someone was trying to optimize this function, that's not very helpful. I believe REINFORCE is not of much help either - it's not magic. I'm not aware of any stochastic gradient estimators that are helpful in this case (although if there is a method I'd like to hear about it).
- JustFinishedBSG 7y ago
  > on-smooth functions like abs(), max(), min() where derivatives do not exist While these functions are not differentiable they are sub-differentiable. Which is an extension of differentiability. Subderivatives are set, if the set only contain one point the function is differentiable. Otherwise it doesn't matter for gradient descent you can just use any element of the set. > Disjoint IF-THEN-ELSE conditions are discontinuities in the function space Same, it doesn't matter, they mathematically are equivalent to indicator functions and are subdifferentiable. https://en.wikipedia.org/wiki/Subderivative https://en.wikipedia.org/wiki/Subderivative
hsaliak 7y ago
How does this compare to Jax? https://github.com/google/jax https://github.com/google/jax Why do this in the language instead of a library?
- taliesinb 7y ago
  Probably the main justification is that the analysis and transformation steps needed to compute the vjp and jvp pullbacks of a function (which correspond to reverse- and forward-mode automatic differentiation) require enough of the other machinery of a compiler that they are best done WITHIN a compiler. Then other things become quite natural, too, like producing the tangent vector versions of data structures like tuples and maps! Moreover, a statically typed language like Swift is a much better starting point for this kind of effort than Python. Array shapes and dimensions are already a type system - you might as well go the whole distance and get all the other safety, readability, and efficiency benefits! PS shout out for named array axes as the future of array-based (and hence differentiable) programming... see http://nlp.seas.harvard.edu/NamedTensor http://nlp.seas.harvard.edu/NamedTensor for a good rationale
  - mlevental 7y ago
    is there something in the proposal that addresses named tensors? ctrlf doesn't find anything.
  - hsaliak 7y ago
    jax does have a just in time compiler that lets you compile your python functions to XLA-optimized kernels (through llvmlite under the hood?). The fact that you can jit compile and gain the benefits of "doing this within the compiler" is one of its main selling points.
- optevo 7y ago
  If you were targeting a Lisp you could do this as a (macro) library
  - deleted 7y ago
    [deleted]
__erik 7y ago
I'm really excited for this. I'm unaware of any mainstream language with first class support for differentiation, I think its going to be really interesting to see what people use it for out side of ML.
- adamnemecek 7y ago
  Julia is in the same space.
  - __erik 7y ago
    Julia is great but it doesnt play in the domain of apps and servers
    - ddragon 7y ago
      It does have a ton of numerical/data science/general science libraries to be differentiated and to add value to the differentiable code though.
    - byt143 7y ago
      Yes it does. See genie.jl for a full mvc framework. Also there's mux.jl and http.jl.
    - mark_l_watson 7y ago
      It does. I spent several evenings writing little bits of Julia code that do “non numeric” stuff like querying RDF data stores, text processing, etc. I think Julia is a reasonable general purpose language.
  - taliesinb 7y ago
    Yup, work is continuing apace with Julia’s next-gen Zygote project. Also, from the GP’s thought about applications beyond DL, my favorite examples so far are for model-based RL [1] and Neural ODEs [2] [1] https://fluxml.ai/2019/03/05/dp-vs-rl.html https://fluxml.ai/2019/03/05/dp-vs-rl.html [2] https://julialang.org/blog/2019/01/fluxdiffeq https://julialang.org/blog/2019/01/fluxdiffeq
sethryclaus 7y ago
Well done Swift. Insightful language design.
thedataangel 7y ago
This is actually huge. I saw a proof of concept of something like this in Haskell a few years back, but it's amazing it see it (probably) making it into the core of a mainstream language. This may let them capture a large chunk of the ML market from Python - and hopefully greatly improve ML apis while they're at it.
- one-punch 7y ago
  > I saw a proof of concept of something like this in Haskell a few years back You mean the automatic differentiation package `ad`? https://hackage.haskell.org/package/ad https://hackage.haskell.org/package/ad
- krapht 7y ago
  Huh? Nobody is writing numerically intensive libraries in Python. Clearly this language proposal is taking aim at C++ and Fortran. Even if this caused TensorFlow & others to rewrite everything in Swift, people would write Python bindings to it and keep using Python. I'll get excited if Apple actually merges this into Swift. It's a niche feature that their compiler team will need to maintain forever. I actually have been working on algorithmic differentiation in C++, so it's not even that I wouldn't want to try Swift out if it actually made it in. However, because this sort of thing is of such narrow interest I believe the future will stay with embedded DSLs / libraries / ugly macro/template hackery.
  - mlevental 7y ago
    lol this is literally by the group that's rewriting tensorflow in Swift https://www.tensorflow.org/swift https://www.tensorflow.org/swift so you're off on the intention here in that it is exactly taking aim at python as the main data ecosystem language.
- seanmcdirmid 7y ago
  > I saw a proof of concept of something like this in Haskell a few years back, but it's amazing it see it (probably) making it into the core of a mainstream language. Probably Conal Elliott’s work, eg in Vertigo (http://conal.net/Vertigo/ http://conal.net/Vertigo/, circa 2005)? There he was using it for normal computations used in pixel shading, pretty cool stuff. He is still active in this field, and has a lot of new papers that are more ML focused. I do wonder if “general” AD support will be useful for computer graphics as well as ML?
adamnemecek 7y ago
Now do automatic integration.
cs702 7y ago
How does this relate to all the work that Christ Lattner et al have been doing at Google with Swift, MLIR, etc.?[a] Is this... a separate, parallel, more encompassing proposal? Is there any coordination between these two groups? -- [a] https://www.youtube.com/watch?v=yCd3CzGSte8 https://www.youtube.com/watch?v=yCd3CzGSte8
- adamnemecek 7y ago
  It's the same group.
- samtheprogram 7y ago
  I don't know if "Christ" Lattner was intentional (humorous) or an accident, but I chuckled.
  - cs702 7y ago
    Accident.
oyashius 7y ago
This is super exciting
abeppu 7y ago
I'm not a Swift programmer, so perhaps my confusion is just a symptom of broader ignorance, but I find two things unclear here: - What does 'first-class' mean, really? - Which of these benefits are unique to integrating notions of derivatives into the language, and which could be enjoyed well-written libraries? The mega-proposal links out to a separate doc on embedded DSLs, with broad statements about what's "typical" or "often" true of existing DSLs -- but which of those issues are insurmountable? That section mentions Swift's limited metaprogramming facilities. Why choose to carve out this added support for a single family of algorithms rather than add in some more general metaprogramming abilities that enable better EDSLs?
- thedataangel 7y ago
  My understanding of automatic differentiation (AD) is that it's only really possible at the compiler level, since you need the ability to interpret and manipulate function definitions themselves. Certainly, no library would be able to offer the same level of guarantees telling you if you've done it wrong, nor the same opportunities for optimisation.
  - abeppu 7y ago
    It's certainly not the case that autodiff is only possible at the compiler level. I've implemented forward mode (via dual numbers) and reverse mode (via tapes / wengert nodes) autodiff in libraries before.
    - mlevental 7y ago
      notice the qualifier "really". obviously you can implement autodiff kind of outside the complier since pytorch and tensor flow exist. but those implementations constrain you to a select few compositions (please no comments on Turing completeness with just loops and conditionals). so for example if statements in pytorch are not differentiable (they might have piece wise continuous derivates) because pytorch doesn't actually trace the ast. I'm not a languages expert but outside of implementing in the compiler I imagine you'd need a homoiconic language to implement as a library.
taliesinb 7y ago
If you are interested in following along, the Swift for TensorFlow team has a design meeting every Friday. The meetings are live-streamed and anyone can join. I recommend them if you are curious and want to hear more about the challenges and opportunities! A recent talk had Jeremy Howard (of fasti.ai fame) present the full-featured deep learning API he has designed on top of Swift for TensorFlow. You can find out more on the mailing list https://groups.google.com/a/tensorflow.org/forum/m/#!forum/swift https://groups.google.com/a/tensorflow.org/forum/m/#!forum/s...
chriscaruso 7y ago
I don't see why a well written library could not serve the same purpose. It seems like a lot of cruft. I doubt, for example, Python would ever consider adding this and it's the defacto language that would benefit the most from something like this - due to the existing tools and communities. It just seems so narrow and not at the same level of abstraction that languages typically sit at. I could see the language supporting higher level functionality so a library could do this without a bunch of extra work (such as by some reflection).
- AbrahamParangi 7y ago
  I would counter that differentiable programming should perhaps rise to the level of baseline functionality that most languages should offer. I think the applications for automatic differentiation and gradient optimization well exceed what we think of as ML and data science today.
antpls 7y ago
Can "Differentiable Programming" be related to "Differentiable Privacy", or have we now one word (and acronym!) to describe two different things?
- omaranto 7y ago
  I think you meant "differential privacy", and no, it's not closely related to differentiable programming. [1] https://en.wikipedia.org/wiki/Differential_privacy https://en.wikipedia.org/wiki/Differential_privacy
akhilcacharya 7y ago
I really like the idea of building AD directly into the language and compiler infrastructure itself, but the challenge of upstreaming it in a language built for very specific tasks makes me concerned. Is Latner just going to end up making a GSwift? Will it just be forked?
layoutIfNeeded 7y ago
Ew, that’s really not something I would do at the language level...
mark_l_watson 7y ago
This will certainly help people who work on new deep learning theories and model architectures, but not so much the large crowd of deep learning practitioners. In his excellent interview with Lex Fridman, Yann LeCun was critical of any approach to AI that was not differentiable, even constraint satisfaction, and other solid optimization techniques. In the context of scaling to very large problems or models with many billions of parameters, he is probably correct. I have had problems with the Swift and TensorFlow code drops. Sometimes they work for me and sometime they don’t. So, very good technology but perhaps wait for it to mature. I read that some students for the fast.ai course using Swift have also had some setup difficulties. EDIT: you might also want to look at Julia for differentiable programming and Julia with deep learning libraries like Flux is also a ‘turtles all the way down’ system, where unlike TensorFlow where the guts are implemented in C++, for Swift and Julia the entire stack can be implemented in a single language.

162 points by xmmrm 7y ago

Capitalizing on the presence of people who might be new to Automatic Differentiation and want a deeper understanding of how it works, here is an interactive Colab notebook I wrote about this topic entitled “Build your Own TensorFlow” for the Deep Learning Indaba that just happened in Kenya: https://colab.research.google.com/drive/14GeXkFd5pQKKNIJ7BMswP0ihlYg5nIbS#forceEdit=true&offline=true&sandboxMode=true https://colab.research.google.com/drive/14GeXkFd5pQKKNIJ7BMs...

Donald 7y ago
That's an excellent tutorial - more are available on http://www.deeplearningindaba.com/practicals-2019.html http://www.deeplearningindaba.com/practicals-2019.html for those interested.

wenc 7y ago

I'm curious (for folks in the know): how does differentiable programming handle non-differentiable points? Can it detect non-differentiable/non-smooth functions? Non-smooth functions like abs(), max(), min() have points where derivatives do not exist. ReLU functions are non-differentiable at their hinge points. Disjoint IF-THEN-ELSE conditions are discontinuities in the function space, and are traditionally handled in optimization with mixed-integer formulations (i.e. split up the space and do something clever like branch-and-bound to find the optimum).

abeppu 7y ago
My understanding is you don't explicitly handle that. For whichever place you're evaluating, you have some computation graph, and you use the elementary operators on that graph. So, if you have defined ReLU with an `if x >= 0 return x` clause, then if you evaluate the derivative at 0, that's the branch you go down, and you say the answer is 1.
taliesinb 7y ago
While the gradients don’t technically exist, you usually use the limiting value from one side (e.g for the commonly used ReLU activation, you can use either 0 or 1). This is justified by the observation that randomly initialized networks are extremely unlikely to encounter these particular values without some kinder of deeper conspiracy happening. You could put this on a mathematical footing by saying the set of non differentiable points has measure zero, for example.
- abeppu 7y ago
  > randomly initialized networks But "networks" here, you're thinking of ANNs, yes? But in the context of proposing differential programming as an addition to a general purpose language (and where the proposal explicitly brings up a bunch of cases outside of deep learning), is it fair to justify behavior based on what makes sense in a popular but narrow application?
  - taliesinb 7y ago
    It’s a good question what the plans are for DP languages to handle situations where non-differentiability shouldn’t be ignored. For sensitivity analysis it might be disastrous to conclude that an output is sensitive to an input when it is actually not, merely because an intermediary ReLU hit 0, for example. A conservative approach could be to define versions of the relevant functions that threw exceptions at such points, or that also calculated the trusted margin of the resulting gradients; non-differentiability would then produce a zero trust margin.
  - mlevental 7y ago
    he/she addressed that - the points at which the function isn't differentiable has measure zero. besides this isn't some kind of new hack - one sided limits (and therefore derivatives) were invented exactly for such cases (min, max, abs) and have been used by mathematicians probably since just about when calculus was invented.
deleted 7y ago
[deleted]
chillee 7y ago
Other people have answered what they do, but this is the big gap between people talking about 'differentiable programming' in theory, and having it actually work in practice. It's true that once you have control flow, the gradient quickly becomes meaningless. I posted an example here: https://news.ycombinator.com/item?id=20892287 https://news.ycombinator.com/item?id=20892287 That's also the biggest reason I tend to find much of this "differentiable programming" stuff to be overhyped. It's hard to reformulate programs in a way s.t. the derivative can mean something meaningful. And I'm not convinced traditional languages will benefit. That's not to say that there isn't cases where your program can be formulated to have a meaningful derivative. See this differentiable ray tracer: https://people.csail.mit.edu/tzumao/diffrt/ https://people.csail.mit.edu/tzumao/diffrt/
- taliesinb 7y ago
  > It's hard to reformulate programs in a way s.t. the derivative can mean something meaningful. Really? The gradients computed by AD are the exact answer to the following question: if I were to change this input or parameter an infinitesimal amount, how much would it change the output of my function? That is always meaningful (when it is defined), and means what I just said. You can easily make functions where it is not defined, of course, just like you can make a sphere into two spheres with the Banach-Tarski theorem! But there are vast, vast forests of numerical computation employed in industry, science, finance, engineering, where it is almost always defined. And even for more “chunky” computations where the non-differentiability is more severe, there are algorithms like REINFORCE that you can use to estimate gradients through these parts.
  - chillee 7y ago
    It's not meaningful in the sense that it won't correspond with your intuition of "will increasing the input increase my output". Presumably the point of differentiable programming isn't just getting the derivative for fun, it's for optimizing some quantity. For example, take this code. x,y for (int i=0; i<x; i++) y += 1 return y It's technically true that the gradient of 0 is correct (modulo boundaries). But if someone was trying to optimize this function, that's not very helpful. I believe REINFORCE is not of much help either - it's not magic. I'm not aware of any stochastic gradient estimators that are helpful in this case (although if there is a method I'd like to hear about it).
JustFinishedBSG 7y ago
> on-smooth functions like abs(), max(), min() where derivatives do not exist While these functions are not differentiable they are sub-differentiable. Which is an extension of differentiability. Subderivatives are set, if the set only contain one point the function is differentiable. Otherwise it doesn't matter for gradient descent you can just use any element of the set. > Disjoint IF-THEN-ELSE conditions are discontinuities in the function space Same, it doesn't matter, they mathematically are equivalent to indicator functions and are subdifferentiable. https://en.wikipedia.org/wiki/Subderivative https://en.wikipedia.org/wiki/Subderivative

hsaliak 7y ago

How does this compare to Jax? https://github.com/google/jax https://github.com/google/jax Why do this in the language instead of a library?

taliesinb 7y ago
Probably the main justification is that the analysis and transformation steps needed to compute the vjp and jvp pullbacks of a function (which correspond to reverse- and forward-mode automatic differentiation) require enough of the other machinery of a compiler that they are best done WITHIN a compiler. Then other things become quite natural, too, like producing the tangent vector versions of data structures like tuples and maps! Moreover, a statically typed language like Swift is a much better starting point for this kind of effort than Python. Array shapes and dimensions are already a type system - you might as well go the whole distance and get all the other safety, readability, and efficiency benefits! PS shout out for named array axes as the future of array-based (and hence differentiable) programming... see http://nlp.seas.harvard.edu/NamedTensor http://nlp.seas.harvard.edu/NamedTensor for a good rationale
- mlevental 7y ago
  is there something in the proposal that addresses named tensors? ctrlf doesn't find anything.
- hsaliak 7y ago
  jax does have a just in time compiler that lets you compile your python functions to XLA-optimized kernels (through llvmlite under the hood?). The fact that you can jit compile and gain the benefits of "doing this within the compiler" is one of its main selling points.
optevo 7y ago
If you were targeting a Lisp you could do this as a (macro) library
- deleted 7y ago
  [deleted]

__erik 7y ago

I'm really excited for this. I'm unaware of any mainstream language with first class support for differentiation, I think its going to be really interesting to see what people use it for out side of ML.

adamnemecek 7y ago
Julia is in the same space.
- __erik 7y ago
  Julia is great but it doesnt play in the domain of apps and servers
  - ddragon 7y ago
    It does have a ton of numerical/data science/general science libraries to be differentiated and to add value to the differentiable code though.
  - byt143 7y ago
    Yes it does. See genie.jl for a full mvc framework. Also there's mux.jl and http.jl.
  - mark_l_watson 7y ago
    It does. I spent several evenings writing little bits of Julia code that do “non numeric” stuff like querying RDF data stores, text processing, etc. I think Julia is a reasonable general purpose language.
- taliesinb 7y ago
  Yup, work is continuing apace with Julia’s next-gen Zygote project. Also, from the GP’s thought about applications beyond DL, my favorite examples so far are for model-based RL [1] and Neural ODEs [2] [1] https://fluxml.ai/2019/03/05/dp-vs-rl.html https://fluxml.ai/2019/03/05/dp-vs-rl.html [2] https://julialang.org/blog/2019/01/fluxdiffeq https://julialang.org/blog/2019/01/fluxdiffeq

sethryclaus 7y ago

Well done Swift. Insightful language design.

thedataangel 7y ago

This is actually huge. I saw a proof of concept of something like this in Haskell a few years back, but it's amazing it see it (probably) making it into the core of a mainstream language. This may let them capture a large chunk of the ML market from Python - and hopefully greatly improve ML apis while they're at it.

one-punch 7y ago
> I saw a proof of concept of something like this in Haskell a few years back You mean the automatic differentiation package `ad`? https://hackage.haskell.org/package/ad https://hackage.haskell.org/package/ad
krapht 7y ago
Huh? Nobody is writing numerically intensive libraries in Python. Clearly this language proposal is taking aim at C++ and Fortran. Even if this caused TensorFlow & others to rewrite everything in Swift, people would write Python bindings to it and keep using Python. I'll get excited if Apple actually merges this into Swift. It's a niche feature that their compiler team will need to maintain forever. I actually have been working on algorithmic differentiation in C++, so it's not even that I wouldn't want to try Swift out if it actually made it in. However, because this sort of thing is of such narrow interest I believe the future will stay with embedded DSLs / libraries / ugly macro/template hackery.
- mlevental 7y ago
  lol this is literally by the group that's rewriting tensorflow in Swift https://www.tensorflow.org/swift https://www.tensorflow.org/swift so you're off on the intention here in that it is exactly taking aim at python as the main data ecosystem language.
seanmcdirmid 7y ago
> I saw a proof of concept of something like this in Haskell a few years back, but it's amazing it see it (probably) making it into the core of a mainstream language. Probably Conal Elliott’s work, eg in Vertigo (http://conal.net/Vertigo/ http://conal.net/Vertigo/, circa 2005)? There he was using it for normal computations used in pixel shading, pretty cool stuff. He is still active in this field, and has a lot of new papers that are more ML focused. I do wonder if “general” AD support will be useful for computer graphics as well as ML?

adamnemecek 7y ago

Now do automatic integration.

cs702 7y ago

How does this relate to all the work that Christ Lattner et al have been doing at Google with Swift, MLIR, etc.?[a] Is this... a separate, parallel, more encompassing proposal? Is there any coordination between these two groups? -- [a] https://www.youtube.com/watch?v=yCd3CzGSte8 https://www.youtube.com/watch?v=yCd3CzGSte8

adamnemecek 7y ago
It's the same group.
samtheprogram 7y ago
I don't know if "Christ" Lattner was intentional (humorous) or an accident, but I chuckled.
- cs702 7y ago
  Accident.

oyashius 7y ago

This is super exciting

abeppu 7y ago

I'm not a Swift programmer, so perhaps my confusion is just a symptom of broader ignorance, but I find two things unclear here: - What does 'first-class' mean, really? - Which of these benefits are unique to integrating notions of derivatives into the language, and which could be enjoyed well-written libraries? The mega-proposal links out to a separate doc on embedded DSLs, with broad statements about what's "typical" or "often" true of existing DSLs -- but which of those issues are insurmountable? That section mentions Swift's limited metaprogramming facilities. Why choose to carve out this added support for a single family of algorithms rather than add in some more general metaprogramming abilities that enable better EDSLs?

thedataangel 7y ago
My understanding of automatic differentiation (AD) is that it's only really possible at the compiler level, since you need the ability to interpret and manipulate function definitions themselves. Certainly, no library would be able to offer the same level of guarantees telling you if you've done it wrong, nor the same opportunities for optimisation.
- abeppu 7y ago
  It's certainly not the case that autodiff is only possible at the compiler level. I've implemented forward mode (via dual numbers) and reverse mode (via tapes / wengert nodes) autodiff in libraries before.
  - mlevental 7y ago
    notice the qualifier "really". obviously you can implement autodiff kind of outside the complier since pytorch and tensor flow exist. but those implementations constrain you to a select few compositions (please no comments on Turing completeness with just loops and conditionals). so for example if statements in pytorch are not differentiable (they might have piece wise continuous derivates) because pytorch doesn't actually trace the ast. I'm not a languages expert but outside of implementing in the compiler I imagine you'd need a homoiconic language to implement as a library.

taliesinb 7y ago

If you are interested in following along, the Swift for TensorFlow team has a design meeting every Friday. The meetings are live-streamed and anyone can join. I recommend them if you are curious and want to hear more about the challenges and opportunities! A recent talk had Jeremy Howard (of fasti.ai fame) present the full-featured deep learning API he has designed on top of Swift for TensorFlow. You can find out more on the mailing list https://groups.google.com/a/tensorflow.org/forum/m/#!forum/swift https://groups.google.com/a/tensorflow.org/forum/m/#!forum/s...

chriscaruso 7y ago

I don't see why a well written library could not serve the same purpose. It seems like a lot of cruft. I doubt, for example, Python would ever consider adding this and it's the defacto language that would benefit the most from something like this - due to the existing tools and communities. It just seems so narrow and not at the same level of abstraction that languages typically sit at. I could see the language supporting higher level functionality so a library could do this without a bunch of extra work (such as by some reflection).

AbrahamParangi 7y ago
I would counter that differentiable programming should perhaps rise to the level of baseline functionality that most languages should offer. I think the applications for automatic differentiation and gradient optimization well exceed what we think of as ML and data science today.

antpls 7y ago

Can "Differentiable Programming" be related to "Differentiable Privacy", or have we now one word (and acronym!) to describe two different things?

omaranto 7y ago
I think you meant "differential privacy", and no, it's not closely related to differentiable programming. [1] https://en.wikipedia.org/wiki/Differential_privacy https://en.wikipedia.org/wiki/Differential_privacy

akhilcacharya 7y ago

I really like the idea of building AD directly into the language and compiler infrastructure itself, but the challenge of upstreaming it in a language built for very specific tasks makes me concerned. Is Latner just going to end up making a GSwift? Will it just be forked?

layoutIfNeeded 7y ago

Ew, that’s really not something I would do at the language level...

mark_l_watson 7y ago

This will certainly help people who work on new deep learning theories and model architectures, but not so much the large crowd of deep learning practitioners. In his excellent interview with Lex Fridman, Yann LeCun was critical of any approach to AI that was not differentiable, even constraint satisfaction, and other solid optimization techniques. In the context of scaling to very large problems or models with many billions of parameters, he is probably correct. I have had problems with the Swift and TensorFlow code drops. Sometimes they work for me and sometime they don’t. So, very good technology but perhaps wait for it to mature. I read that some students for the fast.ai course using Swift have also had some setup difficulties. EDIT: you might also want to look at Julia for differentiable programming and Julia with deep learning libraries like Flux is also a ‘turtles all the way down’ system, where unlike TensorFlow where the guts are implemented in C++, for Swift and Julia the entire stack can be implemented in a single language.