7 ms·

No surprises there. There were maybe a total of 5 people excited about adding automatic differentiation to Swift. Too bad they didn't try improving Julia instea

by ipsum2 6y ago

No surprises there. There were maybe a total of 5 people excited about adding automatic differentiation to Swift. Too bad they didn't try improving Julia instead.

_Wintermute 6y ago
I remember their terrible article trying to justify why Swift was the right choice out of a load of more appropriate languages. I'm still annoyed they didn't go with Julia.
- cglong 6y ago
  Source? Curious to read that :D
  - _Wintermute 6y ago
    Here you are: https://github.com/tensorflow/swift/blob/main/docs/WhySwiftForTensorFlow.md#which-languages-fit-our-project-requirements https://github.com/tensorflow/swift/blob/main/docs/WhySwiftF...
    - sanderjd 6y ago
      I always figured they would have chosen rust if it weren't for Lattner's sway (real or perceived) in the Swift compiler development community. One underestimated advantage of rust over swift in my view is that it has interested practitioners across a much more diverse range of problems spaces. I think it was always going to be hard to convince people to use a language for for ML when most of its practitioners are solely mobile developers. But people already used rust for lots of different kinds of data processing heavy use cases.
      - johnnycerberus 6y ago
        You can't be serious. Rust would just replace C++ in the current formula, you can't use Rust as if it was Python. ML people don't need to waste time proving memory properties at compile time, cmon! Indeed, there were alternatives. Two already come to my mind, one is Scala (which syntax-wise is strikingly similar to Python) and Julia. Both having features like type-safety, macros, JIT, FFIs, etc.
        sanderjd 6y ago
        None of this is less true for swift. The whole idea is that there may be an advantage to better static analysis that overcomes the perceived downside of using a language with a strict compiler and lots of rules. That idea may or may not be worthwhile, but it is equally applicable to swift and rust. Julia I agree but Scala strikes me as a bit of a non sequitur. It is much more complex and foot-gunny (and thus difficult for scientists who only know python to get right) than either swift or rust, and much more difficult for static analyzers.
        pjmlp 6y ago
        Scala already has a foothold on the same research domain due to large data frameworks like Hadoop and Spark, with NVidia caring for polyglot support on CUDA, including JVM based languages.
        sanderjd 6y ago
        That is true, and certainly relevant, but I believe it is also true that Scala does not meet the criteria for what that were trying to do, because it is not easy (perhaps not possible) to do the kind of static analysis necessary to achieve their goals. For a generic "what language with a good type system should we use with ML?" project, I would totally agree with you that Scala is an obvious choice. But this project was attempting a more specific thing, for which Scala is a less good fit.
        johnnycerberus 6y ago
        I really don't see how Scala is not fit for this. Swift for Tensorflow was mainly revolving around the Differentiable protocol. There is a paper published in 2019 from Purdue University where they show how AD can be combined with multi-stage programming in Scala. [1] DeepLearning.scala that does compile-time reverse AD. [2] Compute.scala for operator fusion at runtime (JITifying) [3] Here [4] is a Pre-SIP to add reifiable control flow to Scala compiler so AD and operator fusion could happen at compile-time. Moreover, Kotlin also has a library that does type-safe AD. [5] nVidia and Oracle want better interopabiltiy between CUDA and GraalVM through grCuda. [6] [1] Demystifying Differentiable Programming: Shift/Reset the Penultimate Backpropagator - https://arxiv.org/pdf/1803.10228.pdf https://arxiv.org/pdf/1803.10228.pdf [2] https://github.com/ThoughtWorksInc/DeepLearning.scala https://github.com/ThoughtWorksInc/DeepLearning.scala [3] https://github.com/ThoughtWorksInc/Compute.scala https://github.com/ThoughtWorksInc/Compute.scala [4] https://contributors.scala-lang.org/t/pre-sip-for-with-control-flow-an-alternative-to-scala-async-scala-continuations-and-scala-virtualized/4423 https://contributors.scala-lang.org/t/pre-sip-for-with-contr... [5] https://github.com/breandan/kotlingrad https://github.com/breandan/kotlingrad [6] https://developer.nvidia.com/blog/grcuda-a-polyglot-language-binding-for-cuda-in-graalvm/ https://developer.nvidia.com/blog/grcuda-a-polyglot-language...
remexre 6y ago
Well, Julia already got it via https://github.com/FluxML/Zygote.jl https://github.com/FluxML/Zygote.jl
xiphias2 6y ago
While I love using Julia for many things, I prefer having a Python monoculture for AI to be able to mix and match different algorithms. At the end the engine that compiles the mathematical expression to hardware is what matters, and I don't think that LLVM IR that uses is the best IR for the optimizations.
- PartiallyTyped 6y ago
  > I prefer having a Python monoculture for AI to be able to mix and match different algorithms. I do not. Unfortunately, high performance and python do not go hand in hand. Yes, I know the heavy lifting is done by C/C++/Rust/Cuda/Blas/Numba and so on, but, when you run simulations for millions of steps, you end up with billions of python function calls. Afaik only Jax actually performs any optimizations because it constructs an analytical gradient. Zygote seems to be able to that and more on the LLVM IR level which, I think should enable more optimizations.
  - swagonomixxx 6y ago
    I'm not sure I understand what your getting at. If Python's performance in the ML space is not sufficient, then the community would have quickly moved on from it and built something better. And that something better is certainly not Julia and it's definitely not Swift. > Yes, I know the heavy lifting is done by C/C++/Rust/Cuda/Blas/Numba and so on, but, when you run simulations for millions of steps, you end up with billions of python function calls. For ML purposes, if your model isn't running on the GPU, it doesn't matter if you're using Swift, Rust, or whatever, your stuff is gonna be slow. Like it or not, Python is one of the best glue languages out there, and the reason why libraries like TensorFlow and Torch are not used in their native languages (C++) [0] is because they're significantly simpler to use in Python and the performance overhead is usually one function call (e.g run_inference(...)) and not billions. If you find yourself writing a simulation, and you need to optimize away billions of function calls, you can use the C++ API provided by TensorFlow. [0] https://www.tensorflow.org/api_docs/cc https://www.tensorflow.org/api_docs/cc
    - 6y ago
favorited 6y ago
> adding automatic differentiation to Swift That work is still happening (as noted in the link). One of the primary engineers who was working on it at Google works at Apple now.
- ncmncm 6y ago
  Speaking of shuffling, Dave Abrahams, (who was?) lead on the Swift standard library, moved to Google sometime back. Is he still doing Swift there?

by ipsum2 6y ago

No surprises there. There were maybe a total of 5 people excited about adding automatic differentiation to Swift. Too bad they didn't try improving Julia instead.

_Wintermute 6y ago

I remember their terrible article trying to justify why Swift was the right choice out of a load of more appropriate languages. I'm still annoyed they didn't go with Julia.

cglong 6y ago
Source? Curious to read that :D
- _Wintermute 6y ago
  Here you are: https://github.com/tensorflow/swift/blob/main/docs/WhySwiftForTensorFlow.md#which-languages-fit-our-project-requirements https://github.com/tensorflow/swift/blob/main/docs/WhySwiftF...
  - sanderjd 6y ago
    I always figured they would have chosen rust if it weren't for Lattner's sway (real or perceived) in the Swift compiler development community. One underestimated advantage of rust over swift in my view is that it has interested practitioners across a much more diverse range of problems spaces. I think it was always going to be hard to convince people to use a language for for ML when most of its practitioners are solely mobile developers. But people already used rust for lots of different kinds of data processing heavy use cases.
    - johnnycerberus 6y ago
      You can't be serious. Rust would just replace C++ in the current formula, you can't use Rust as if it was Python. ML people don't need to waste time proving memory properties at compile time, cmon! Indeed, there were alternatives. Two already come to my mind, one is Scala (which syntax-wise is strikingly similar to Python) and Julia. Both having features like type-safety, macros, JIT, FFIs, etc.
      - sanderjd 6y ago
        None of this is less true for swift. The whole idea is that there may be an advantage to better static analysis that overcomes the perceived downside of using a language with a strict compiler and lots of rules. That idea may or may not be worthwhile, but it is equally applicable to swift and rust. Julia I agree but Scala strikes me as a bit of a non sequitur. It is much more complex and foot-gunny (and thus difficult for scientists who only know python to get right) than either swift or rust, and much more difficult for static analyzers.
        pjmlp 6y ago
        Scala already has a foothold on the same research domain due to large data frameworks like Hadoop and Spark, with NVidia caring for polyglot support on CUDA, including JVM based languages.
        sanderjd 6y ago
        That is true, and certainly relevant, but I believe it is also true that Scala does not meet the criteria for what that were trying to do, because it is not easy (perhaps not possible) to do the kind of static analysis necessary to achieve their goals. For a generic "what language with a good type system should we use with ML?" project, I would totally agree with you that Scala is an obvious choice. But this project was attempting a more specific thing, for which Scala is a less good fit.
        johnnycerberus 6y ago
        I really don't see how Scala is not fit for this. Swift for Tensorflow was mainly revolving around the Differentiable protocol. There is a paper published in 2019 from Purdue University where they show how AD can be combined with multi-stage programming in Scala. [1] DeepLearning.scala that does compile-time reverse AD. [2] Compute.scala for operator fusion at runtime (JITifying) [3] Here [4] is a Pre-SIP to add reifiable control flow to Scala compiler so AD and operator fusion could happen at compile-time. Moreover, Kotlin also has a library that does type-safe AD. [5] nVidia and Oracle want better interopabiltiy between CUDA and GraalVM through grCuda. [6] [1] Demystifying Differentiable Programming: Shift/Reset the Penultimate Backpropagator - https://arxiv.org/pdf/1803.10228.pdf https://arxiv.org/pdf/1803.10228.pdf [2] https://github.com/ThoughtWorksInc/DeepLearning.scala https://github.com/ThoughtWorksInc/DeepLearning.scala [3] https://github.com/ThoughtWorksInc/Compute.scala https://github.com/ThoughtWorksInc/Compute.scala [4] https://contributors.scala-lang.org/t/pre-sip-for-with-control-flow-an-alternative-to-scala-async-scala-continuations-and-scala-virtualized/4423 https://contributors.scala-lang.org/t/pre-sip-for-with-contr... [5] https://github.com/breandan/kotlingrad https://github.com/breandan/kotlingrad [6] https://developer.nvidia.com/blog/grcuda-a-polyglot-language-binding-for-cuda-in-graalvm/ https://developer.nvidia.com/blog/grcuda-a-polyglot-language...

remexre 6y ago

Well, Julia already got it via https://github.com/FluxML/Zygote.jl https://github.com/FluxML/Zygote.jl

xiphias2 6y ago

While I love using Julia for many things, I prefer having a Python monoculture for AI to be able to mix and match different algorithms. At the end the engine that compiles the mathematical expression to hardware is what matters, and I don't think that LLVM IR that uses is the best IR for the optimizations.

PartiallyTyped 6y ago
> I prefer having a Python monoculture for AI to be able to mix and match different algorithms. I do not. Unfortunately, high performance and python do not go hand in hand. Yes, I know the heavy lifting is done by C/C++/Rust/Cuda/Blas/Numba and so on, but, when you run simulations for millions of steps, you end up with billions of python function calls. Afaik only Jax actually performs any optimizations because it constructs an analytical gradient. Zygote seems to be able to that and more on the LLVM IR level which, I think should enable more optimizations.
- swagonomixxx 6y ago
  I'm not sure I understand what your getting at. If Python's performance in the ML space is not sufficient, then the community would have quickly moved on from it and built something better. And that something better is certainly not Julia and it's definitely not Swift. > Yes, I know the heavy lifting is done by C/C++/Rust/Cuda/Blas/Numba and so on, but, when you run simulations for millions of steps, you end up with billions of python function calls. For ML purposes, if your model isn't running on the GPU, it doesn't matter if you're using Swift, Rust, or whatever, your stuff is gonna be slow. Like it or not, Python is one of the best glue languages out there, and the reason why libraries like TensorFlow and Torch are not used in their native languages (C++) [0] is because they're significantly simpler to use in Python and the performance overhead is usually one function call (e.g run_inference(...)) and not billions. If you find yourself writing a simulation, and you need to optimize away billions of function calls, you can use the C++ API provided by TensorFlow. [0] https://www.tensorflow.org/api_docs/cc https://www.tensorflow.org/api_docs/cc
  - 6y ago

favorited 6y ago

> adding automatic differentiation to Swift That work is still happening (as noted in the link). One of the primary engineers who was working on it at Google works at Apple now.

ncmncm 6y ago
Speaking of shuffling, Dave Abrahams, (who was?) lead on the Swift standard library, moved to Google sometime back. Is he still doing Swift there?