13 ms·

Swift for TensorFlow – A system for deep learning and differentiable computing

135 points by OMGCable 6y ago

cube2222 6y ago
I think the title is cut off at the end.
VHRanger 6y ago
Who would want to use an Apple-centric language for ML, seriously? Apple hardware is outright incompatible to the kind of hardware we use daily in machine learning workstations.
- maxerickson 6y ago
  People that make apps for Apple hardware? Seems kind of obvious.
- xiaodai 6y ago
  The appeal isn't broad enough
- deleted 6y ago
  [deleted]
- iddan 6y ago
  Swift is gaining momentum outside the Apple ecosystem. Web servers are being built and the Tensorflow team explains in length it’s advantages for developing ML models with it.
  - pjmlp 6y ago
    What momentum? Even IBM gave up on it.
    - rvz 6y ago
      Amazon? [0][1][2] [0] https://swift.org/blog/aws-lambda-runtime/ https://swift.org/blog/aws-lambda-runtime/ [1] https://aws.amazon.com/blogs/opensource/continuous-delivery-with-server-side-swift-on-amazon-linux-2/ https://aws.amazon.com/blogs/opensource/continuous-delivery-... [2] https://github.com/amzn/smoke-aws https://github.com/amzn/smoke-aws
      - pjmlp 6y ago
        Numbers?
    - rudedogg 6y ago
      I don't have any inside knowledge, but as a developer using Swift (that followed the Swift web frameworks closely): I think Kitura only existed as a long shot for kicking off their cloud offering. They were hoping iOS developers would code their backends in Swift/Kitura and host on IBMs cloud. But that never happened - so Kitura was killed. Swift on the server is still niche. And the companies that used a Swift backend overwhelmingly went with https://vapor.codes https://vapor.codes instead. It is faster, and has a nicer API.
- YetAnotherNick 6y ago
  I find swift really good in terms of ease of coding and speed. But yeah, I would have preferred Julia or something than swift.
- jasode 6y ago
  >Who would want to use an Apple-centric language for ML, seriously? Apple hardware is outright incompatible Based on how you wrote your comment, I'm guessing you may not know this S4TF is a Google initiative. Yes, Chris Lattner used to work for Apple but he was at Google Brain during the start of this project. When his team wanted to create a language where automatic differentiation and gradient descent was a 1st-class concept in the core syntax (i.e. without libraries) of the programming language , they looked at Rust, Julia, Swift, etc[1]. They ended up choosing Swift as the base language to extend with new ML syntax. Also, Chris has said in previous interviews that he thought of Swift as general purpose language and that hoped it would be used outside of Apple's ecosystem. EDIT to address the confusion where some assume Apple hardware dictates the direction of S4TF: For those not aware, Google's hardware TPU (Tensor Processing Unit)[2] is built with custom ASIC chips and not ARM nor Apple Silicon chips. Presumably, S4TF would run natively against Google's TPU. In other words, the goals and execution targets of S4TF are not restricted by Apple's ecosystem of macOS/iOS/Macbooks/iMacs/iPads in any way. Yes, the SwiftUI framework is Apple-specific but Swift-the-core-language-syntax[3] is not. [to downvoters: if I wrote inaccuracies, please correct me.] [1] https://github.com/tensorflow/swift/blob/master/docs/WhySwiftForTensorFlow.md https://github.com/tensorflow/swift/blob/master/docs/WhySwif... [2] https://en.wikipedia.org/wiki/Tensor_Processing_Unit https://en.wikipedia.org/wiki/Tensor_Processing_Unit [3] https://docs.swift.org/swift-book/ReferenceManual/zzSummaryOfTheGrammar.html https://docs.swift.org/swift-book/ReferenceManual/zzSummaryO...
  - nsonha 6y ago
    For answering a rhetorical question? Yes of course that guy who came up with this also roots for swift to gain more traction outside Apple ecosystem but the reality hasn't going in that direction so far. There are plenty of cool languages out there like f#, julia, mathematica etc and the ML mob settled for python which is an average language so there is no reason to believe they are attracted to swift. And you just ignored the hardware comment
  - dunefox 6y ago
    What a coincidence that someone would choose his own programming language in an evaluation. There was no good reason not to choose Julia.
- ericmay 6y ago
  Working on expanding it outside of the Apple/iOS ecosystem but many (probably most) developers use Mac. You can write Swift code in Linux right now. I’d also say that it appears that Swift is going to be a great language for machine learning. Things like calculating gradients are built in to the language and you can import Python to fill in gaps until Swift libraries are ready. And Swift is quite fast.
  - pjmlp 6y ago
    Try to use Framework code on Linux.
  - MiroF 6y ago
    I tried to use Swift recently while avoiding Xcode crud. It's a nightmare, definitely not going to be adopted until Apple goes more hands off.
- vimy 6y ago
  That might change with Apple Silicon and Apple’s new ML Compute framework.
  - rudedogg 6y ago
    It seems like they'll be making more consumer focused GPUs/ML chips to me. I think Apple Silicon is exciting, but I imagine for serious work you'll still need an external enclosure and a dedicated GPU from AMD. Or Linux/Windows :(. Losing Nvidia/CUDA kind of killed ML on macOS overnight.
- yunohn 6y ago
  I agree on the hardware part. MacOS only supports AMD GPUs which are unsupported by Tensor flow...
  - c16a 6y ago
    Tensorflow can work with ROCM which is equivalent to Nvidia's CUDA.
    - yunohn 6y ago
      There is no official support, only AMD's custom TF branch. Further ROCM itself works only on Linux. I believe most people writing Swift are using MacOS.
      - amkkma 6y ago
        And Julia can do custom ROCM codegen, either by itself or through array and kernel abstractions :) https://github.com/JuliaGPU/AMDGPU.jl https://github.com/JuliaGPU/AMDGPU.jl https://juliagpu.gitlab.io/KernelAbstractions.jl/ https://juliagpu.gitlab.io/KernelAbstractions.jl/ https://github.com/JuliaGPU/GPUArrays.jl/pulse https://github.com/JuliaGPU/GPUArrays.jl/pulse
        yunohn 6y ago
        Indeed, the Julia scene looks enticing right now. If only AMD ROCM supported MacOS... :/
xvilka 6y ago
Should have used Julia instead. The choice was only because of the Chris Lattner who already left.
- marcinzm 6y ago
  They presumably wanted a semi-popular statically typed language as the gains of Julia over Python aren't enough to be worth it (and Julia isn't popular enough).
  - pjmlp 6y ago
    On HN circles maybe. https://juliacomputing.com/case-studies/ https://juliacomputing.com/case-studies/
    - marcinzm 6y ago
      There's a difference between popular and "not a toy language." I'm not arguing Julia isn't used, I'm arguing it's not used often enough to be a merit irrespective of other reasons.
      - pjmlp 6y ago
        Interesting given some of the renowned names using it, probably with more revenue than plenty of Rust unicorns.
        MiroF 6y ago
        Apple, Microsoft, Google? This website isn't counting "Julia only" stacks, it's just companies that have used Julia for one project or another. If you really want to compare that to rust, julia is again going to fall short.
        pjmlp 6y ago
        Those were not the companies I had in mind with my Rust remark, but if you so wish, Swift, Kotlin/Native, Go, C++, .NET Native, Verona, Checked C, Objective-C. It remains to be seen how much Rust they will actually make into tier 1 OS SDKs for userspace applications. In fact, currently it looks more they are bringing their experience with Rust into their platform languages than anything else. Swift memory ownership, Verona, C++ Core Guidelines checker, Kotlin/Native ownership rules. Beware wishing for Julia's downfall with glass ceilings.
adonese 6y ago
Is this (future of tf is in swift) still the case? Because there have been speculation esp. when lattner left google. Also projects like Jax has gained good momentum. I guess it's really hard to convince scientist to move away off of their python legacy. And python scientific stack is really incredible.
throw6606 6y ago
Careful, folks. S4TF is pretty much dead on arrival. It was pushed aggressively by Chris Lattner (for obvious reasons) but he left Google a while ago and since then most internal users lost interest. There's nothing in Swift that's inherently suitable for ML and building the ecosystem is a ton of work; without all the political pushing, it went nowhere and is close to a "semi-abandoned research project" phase.
- rtorr 6y ago
  “Stillborn” is a pretty awful term to use for software.
  - throw6606 6y ago
    Mea culpa. Reworded.
  - StavrosK 6y ago
    Why is that? It conveys the DOA meaning pretty well.
    - MiroF 6y ago
      Generally, your metaphors should not rely on comparison to a pretty traumatic event that has happened to quite a few people, many of whom might be around you without you knowing.
      - lehi 6y ago
        I know anorexics for whom any mention of food can be a trigger. Should metaphors involving consumption be verboten in public discourse because they might be read by someone with an eating disorder?
      - layoutIfNeeded 6y ago
        Sure. I guess then you shouldn’t even use the word “dead”, as plenty of people have lost close relatives which is a pretty traumatic event.
6d65 6y ago
Last time I looked the automatic differentiation was in a compiler branch with no immediate plans to merge in master. But overall it is promising. I even installed Swift on Linux to play with it, didn't get to ML as I have an AMD GPU and this is a can of worms. Hope it's finished one day. I would prefer for Julia ml libraries to become mainstream. But, it is what it is. Also, the ideal for me would be Rust for tensorflow, but the slower compile times (didn't compare with Swift) are an impediment for an iterative workflow such as tweaking models.
- turbinerneiter 6y ago
  I recently spend some days playing with differentiable programming in Swift on Linux: * as you said, auto-diff is in a branch or the Google fork of the project * the only pre-built images are for Ubuntu 18.04 * on Linux, the REPL seems to somewhat broken * many libraries are assuming OSX or iOS I don't feel a lot of hope for adoption of Swift on Linux. Apple obviously is not working on that (fair, they have no reason to do so) and the Swift community also has no focus on Linux, since ... they are in the Apple ecosystem. Meanwhile, the Open Source community is much more interested in Rust than Swift. For the differentiable programming - this is what got me excited, but after trying, I was a bit underwhelmed. Not that it isn't great technology, its just not figured out yet. I tried to come up with a use case outside of ML and the one I tried wasn't really applicable. I do feel however that someone will come up with something and that it will have quite some impact.
  - 6d65 6y ago
    Yep. It's difficult to find usages for autodiff, and it looks like a very niche thing to add to a language. But, it's still cool. In a language with extensible syntax(ex: proc macros), this would sit in a library. And I think you're right, there is a low chance that this gets traction. Especially since Chris Lattner is at SiFive, they probably will have some kind ofAI accelerators, but TF is for training rather then execution. So not sure they'll find a reason to push it. Jeremy Howard from fast ai might be able to convince people to give it a try. But without people working on it full time, the chances are not great. Especially with a compiler fork that requires constant merges/rebases. But, who knows.
YetAnotherNick 6y ago
I am not getting the exact usecase. If the speed is the issue, then I think tf.function solves that. For most of the operation we need to use python via pythonkit, so the things which can't be sped up using tf.function, can't be sped up using swift. Also if we need to use python everywhere, type safety is also very minor.
- ddragon 6y ago
  The unique property is the ability to just pick any code or library that is unaware of the differentiation library (unlike tf.function as it needs to specifically use tf methods) and get the gradient. In a language like Julia this is immediately useful as it has a massive ecosystem of numerical code that make sense to get gradients (like differential equations and the SciML project [1], or less conventional stuff like raytracers), but in a language like Swift (as there is no meaning to gradient of GUI libraries or frontend stuff) it is more of a "if you build they'll come" faith from Google. But regardless unique features, it's a have cake and eat it too type of interface. You don't need to learn a second language within the language like tensorflow's tf.* making it even more natural and flexiblethan pytorch, including all debug mechanisms of the host language itself, but you still get compile time graph creation like tensorflow, including all kinds of optimizations. It makes other approaches seem primitive by comparison, but creating it is much more complex, and the main audience is already more than used to using language within language solutions (like numpy) which can provide something almost as good even if less elegantly, so it's not easy to convince people as well (when it involves changing programming languages). [1] https://sciml.ai/ https://sciml.ai/
- MiroF 6y ago
  > exact usecase Static type checking?
pjmlp 6y ago
Still no Windows support at the level of DirectML, or Julia.
bodono 6y ago
I'm not sure this is really going to take off, it seems that most people who are abandoning TF are moving to Jax or pytorch. My own experience with Jax is that it is much easier to use then TF, just an all round more pleasant experience. It would be interesting to try this, but at this point I'm not really willing to learn 'yet another deep learning framework' and the extreme anti-user problems that TF had make me loath to give it another shot, even with a presumably better frontend. Moreover, I think that python is just a better all-round ML/data science language at this point. Has anyone tried both Jax and this and would be willing to give us their thoughts on strengths and weaknesses of each?
- alpineidyll3 6y ago
  The subtext is Google would love even more Google projects to be ml prerequisites.
- gas9S9zw3P9c 6y ago
  I'm skeptical of JAX. It feels good right now, but when the first TF beta version came out it was very much like that too - clean, simple, minimal, and just a better version of Theano. Then the "crossing the chasm" effort started and everyone at Google wanted to be part of it, making TF the big complex mess it is today. It's a great example of Conway's Law. I'm not convinced the same won't happen to JAX as it catches on. PyTorch has already stood the test of time and proven that its development is led by a competent team.
  - bodono 6y ago
    I know where you're coming from, but TF in my opinion was very user-hostile even on arrival. I can't tell you how much hair-pulling I did over tf.conds, tf.while_loops and the whole gather / scatter paradigm for simple indexing into arrays. I really think the people working on it wanted users to write TF code in a certain, particular way and made it really difficult to use it in other ways. Just thinking back on that time still raises my blood pressure! So far Jax is much better and I'm cautiously optimistic they have learned lessons from TF.
Nelkins 6y ago
There's some cool autodiff work going on in F#/.NET now too[0]. From a usage/API perspective they look kind of similar. [0] https://diffsharp.github.io/ https://diffsharp.github.io/
tanilama 6y ago
I still think it is lacking in this proposal to justify why picking Swift as the host language
suyash 6y ago
For those who prefer an alternate language here is Java for TensorFlow project https://www.tensorflow.org/install/lang_java https://www.tensorflow.org/install/lang_java
Razengan 6y ago
Related: Why Swift for TensorFlow: https://github.com/tensorflow/swift/blob/master/docs/WhySwiftForTensorFlow.md https://github.com/tensorflow/swift/blob/master/docs/WhySwif...
- dunefox 6y ago
  If you're dead set on using a particular language you can find reasons why it appears suitable even if they don't make any sense.
  - Razengan 6y ago
    That logic works both ways.
hedgehog 6y ago
The name is a bit confusing. Swift for TensorFlow combines a bunch of things: - adding autodiff to Swift language & compiler - neural net construction & training API in Swift - low-friction Python bindings - low-friction C++ interop - ability to run neural nets TensorFlow using the C++ interop - ability to alternately run neural nets directly on XLA ("X10") The Swift changes are supposed to get mainlined and I think at least some of the stuff related to Python and C++ already are. The idea is that Swift is nice enough to cover experiment + train + embed as library even on mobile platforms. It's a big engineering project and I hope Google keeps funding the work.
mark_l_watson 6y ago
I question the long term viability of Swift for TensorFlow. I hope that I don’t sound like I am whining but I invested a fair amount of time with Swift from last fall through this February because I wanted: a better faster language to work in for DL than Python; I was also interested in trying some iOS, iPadOS, and macOS development; Swift looked interesting also for Linux server side work. For Apple development I have found Flutter+Dart to be more pleasant. For DL I decided to stick with taking advantage of my 5+ years experience with TF in Python. Off topic, but Julia and Flux are really worth checking out for DL. Some advice: if you want to experiment with Swift for TensorFlow use Google’s colab and make your life easier. I wish I could get back the install times on my Linux GPU laptop and on macOS. If you pay for colab like I do, you get good GPU/TPU resources, and it is simply an easy and fun way to go.
joaogui1 6y ago
I really don't understand why they didn't go with Julia, it fits the purpose much better than Swift, already having a ML ecosystem and having great interop with Python, C, C++, R and Matlab. Heck JAX, where a lot of TF refugees are going, is pretty similar to Zygote
- dunefox 6y ago
  It's because they were set on Swift and the 'language evaluation' was pointless. Julia would have been the natural choice.
socialdemocrat 6y ago
I am a fan of both Swift and Julia but I honestly don’t see the point of Swift in scientific computing. Julia is just a way better fit. However for App development Swift of course has a far more impressive stack.

135 points by OMGCable 6y ago

I think the title is cut off at the end.

Who would want to use an Apple-centric language for ML, seriously? Apple hardware is outright incompatible to the kind of hardware we use daily in machine learning workstations.

maxerickson 6y ago
People that make apps for Apple hardware? Seems kind of obvious.
xiaodai 6y ago
The appeal isn't broad enough
deleted 6y ago
[deleted]
iddan 6y ago
Swift is gaining momentum outside the Apple ecosystem. Web servers are being built and the Tensorflow team explains in length it’s advantages for developing ML models with it.
- pjmlp 6y ago
  What momentum? Even IBM gave up on it.
  - rvz 6y ago
    Amazon? [0][1][2] [0] https://swift.org/blog/aws-lambda-runtime/ https://swift.org/blog/aws-lambda-runtime/ [1] https://aws.amazon.com/blogs/opensource/continuous-delivery-with-server-side-swift-on-amazon-linux-2/ https://aws.amazon.com/blogs/opensource/continuous-delivery-... [2] https://github.com/amzn/smoke-aws https://github.com/amzn/smoke-aws
    - pjmlp 6y ago
      Numbers?
  - rudedogg 6y ago
    I don't have any inside knowledge, but as a developer using Swift (that followed the Swift web frameworks closely): I think Kitura only existed as a long shot for kicking off their cloud offering. They were hoping iOS developers would code their backends in Swift/Kitura and host on IBMs cloud. But that never happened - so Kitura was killed. Swift on the server is still niche. And the companies that used a Swift backend overwhelmingly went with https://vapor.codes https://vapor.codes instead. It is faster, and has a nicer API.
YetAnotherNick 6y ago
I find swift really good in terms of ease of coding and speed. But yeah, I would have preferred Julia or something than swift.
jasode 6y ago
>Who would want to use an Apple-centric language for ML, seriously? Apple hardware is outright incompatible Based on how you wrote your comment, I'm guessing you may not know this S4TF is a Google initiative. Yes, Chris Lattner used to work for Apple but he was at Google Brain during the start of this project. When his team wanted to create a language where automatic differentiation and gradient descent was a 1st-class concept in the core syntax (i.e. without libraries) of the programming language , they looked at Rust, Julia, Swift, etc[1]. They ended up choosing Swift as the base language to extend with new ML syntax. Also, Chris has said in previous interviews that he thought of Swift as general purpose language and that hoped it would be used outside of Apple's ecosystem. EDIT to address the confusion where some assume Apple hardware dictates the direction of S4TF: For those not aware, Google's hardware TPU (Tensor Processing Unit)[2] is built with custom ASIC chips and not ARM nor Apple Silicon chips. Presumably, S4TF would run natively against Google's TPU. In other words, the goals and execution targets of S4TF are not restricted by Apple's ecosystem of macOS/iOS/Macbooks/iMacs/iPads in any way. Yes, the SwiftUI framework is Apple-specific but Swift-the-core-language-syntax[3] is not. [to downvoters: if I wrote inaccuracies, please correct me.] [1] https://github.com/tensorflow/swift/blob/master/docs/WhySwiftForTensorFlow.md https://github.com/tensorflow/swift/blob/master/docs/WhySwif... [2] https://en.wikipedia.org/wiki/Tensor_Processing_Unit https://en.wikipedia.org/wiki/Tensor_Processing_Unit [3] https://docs.swift.org/swift-book/ReferenceManual/zzSummaryOfTheGrammar.html https://docs.swift.org/swift-book/ReferenceManual/zzSummaryO...
- nsonha 6y ago
  For answering a rhetorical question? Yes of course that guy who came up with this also roots for swift to gain more traction outside Apple ecosystem but the reality hasn't going in that direction so far. There are plenty of cool languages out there like f#, julia, mathematica etc and the ML mob settled for python which is an average language so there is no reason to believe they are attracted to swift. And you just ignored the hardware comment
- dunefox 6y ago
  What a coincidence that someone would choose his own programming language in an evaluation. There was no good reason not to choose Julia.
ericmay 6y ago
Working on expanding it outside of the Apple/iOS ecosystem but many (probably most) developers use Mac. You can write Swift code in Linux right now. I’d also say that it appears that Swift is going to be a great language for machine learning. Things like calculating gradients are built in to the language and you can import Python to fill in gaps until Swift libraries are ready. And Swift is quite fast.
- pjmlp 6y ago
  Try to use Framework code on Linux.
- MiroF 6y ago
  I tried to use Swift recently while avoiding Xcode crud. It's a nightmare, definitely not going to be adopted until Apple goes more hands off.
vimy 6y ago
That might change with Apple Silicon and Apple’s new ML Compute framework.
- rudedogg 6y ago
  It seems like they'll be making more consumer focused GPUs/ML chips to me. I think Apple Silicon is exciting, but I imagine for serious work you'll still need an external enclosure and a dedicated GPU from AMD. Or Linux/Windows :(. Losing Nvidia/CUDA kind of killed ML on macOS overnight.
yunohn 6y ago
I agree on the hardware part. MacOS only supports AMD GPUs which are unsupported by Tensor flow...
- c16a 6y ago
  Tensorflow can work with ROCM which is equivalent to Nvidia's CUDA.
  - yunohn 6y ago
    There is no official support, only AMD's custom TF branch. Further ROCM itself works only on Linux. I believe most people writing Swift are using MacOS.
    - amkkma 6y ago
      And Julia can do custom ROCM codegen, either by itself or through array and kernel abstractions :) https://github.com/JuliaGPU/AMDGPU.jl https://github.com/JuliaGPU/AMDGPU.jl https://juliagpu.gitlab.io/KernelAbstractions.jl/ https://juliagpu.gitlab.io/KernelAbstractions.jl/ https://github.com/JuliaGPU/GPUArrays.jl/pulse https://github.com/JuliaGPU/GPUArrays.jl/pulse
      - yunohn 6y ago
        Indeed, the Julia scene looks enticing right now. If only AMD ROCM supported MacOS... :/

xvilka 6y ago

Should have used Julia instead. The choice was only because of the Chris Lattner who already left.

marcinzm 6y ago
They presumably wanted a semi-popular statically typed language as the gains of Julia over Python aren't enough to be worth it (and Julia isn't popular enough).
- pjmlp 6y ago
  On HN circles maybe. https://juliacomputing.com/case-studies/ https://juliacomputing.com/case-studies/
  - marcinzm 6y ago
    There's a difference between popular and "not a toy language." I'm not arguing Julia isn't used, I'm arguing it's not used often enough to be a merit irrespective of other reasons.
    - pjmlp 6y ago
      Interesting given some of the renowned names using it, probably with more revenue than plenty of Rust unicorns.
      - MiroF 6y ago
        Apple, Microsoft, Google? This website isn't counting "Julia only" stacks, it's just companies that have used Julia for one project or another. If you really want to compare that to rust, julia is again going to fall short.
        pjmlp 6y ago
        Those were not the companies I had in mind with my Rust remark, but if you so wish, Swift, Kotlin/Native, Go, C++, .NET Native, Verona, Checked C, Objective-C. It remains to be seen how much Rust they will actually make into tier 1 OS SDKs for userspace applications. In fact, currently it looks more they are bringing their experience with Rust into their platform languages than anything else. Swift memory ownership, Verona, C++ Core Guidelines checker, Kotlin/Native ownership rules. Beware wishing for Julia's downfall with glass ceilings.

adonese 6y ago

Is this (future of tf is in swift) still the case? Because there have been speculation esp. when lattner left google. Also projects like Jax has gained good momentum. I guess it's really hard to convince scientist to move away off of their python legacy. And python scientific stack is really incredible.

throw6606 6y ago

Careful, folks. S4TF is pretty much dead on arrival. It was pushed aggressively by Chris Lattner (for obvious reasons) but he left Google a while ago and since then most internal users lost interest. There's nothing in Swift that's inherently suitable for ML and building the ecosystem is a ton of work; without all the political pushing, it went nowhere and is close to a "semi-abandoned research project" phase.

rtorr 6y ago
“Stillborn” is a pretty awful term to use for software.
- throw6606 6y ago
  Mea culpa. Reworded.
- StavrosK 6y ago
  Why is that? It conveys the DOA meaning pretty well.
  - MiroF 6y ago
    Generally, your metaphors should not rely on comparison to a pretty traumatic event that has happened to quite a few people, many of whom might be around you without you knowing.
    - lehi 6y ago
      I know anorexics for whom any mention of food can be a trigger. Should metaphors involving consumption be verboten in public discourse because they might be read by someone with an eating disorder?
    - layoutIfNeeded 6y ago
      Sure. I guess then you shouldn’t even use the word “dead”, as plenty of people have lost close relatives which is a pretty traumatic event.

6d65 6y ago

Last time I looked the automatic differentiation was in a compiler branch with no immediate plans to merge in master. But overall it is promising. I even installed Swift on Linux to play with it, didn't get to ML as I have an AMD GPU and this is a can of worms. Hope it's finished one day. I would prefer for Julia ml libraries to become mainstream. But, it is what it is. Also, the ideal for me would be Rust for tensorflow, but the slower compile times (didn't compare with Swift) are an impediment for an iterative workflow such as tweaking models.

turbinerneiter 6y ago
I recently spend some days playing with differentiable programming in Swift on Linux: * as you said, auto-diff is in a branch or the Google fork of the project * the only pre-built images are for Ubuntu 18.04 * on Linux, the REPL seems to somewhat broken * many libraries are assuming OSX or iOS I don't feel a lot of hope for adoption of Swift on Linux. Apple obviously is not working on that (fair, they have no reason to do so) and the Swift community also has no focus on Linux, since ... they are in the Apple ecosystem. Meanwhile, the Open Source community is much more interested in Rust than Swift. For the differentiable programming - this is what got me excited, but after trying, I was a bit underwhelmed. Not that it isn't great technology, its just not figured out yet. I tried to come up with a use case outside of ML and the one I tried wasn't really applicable. I do feel however that someone will come up with something and that it will have quite some impact.
- 6d65 6y ago
  Yep. It's difficult to find usages for autodiff, and it looks like a very niche thing to add to a language. But, it's still cool. In a language with extensible syntax(ex: proc macros), this would sit in a library. And I think you're right, there is a low chance that this gets traction. Especially since Chris Lattner is at SiFive, they probably will have some kind ofAI accelerators, but TF is for training rather then execution. So not sure they'll find a reason to push it. Jeremy Howard from fast ai might be able to convince people to give it a try. But without people working on it full time, the chances are not great. Especially with a compiler fork that requires constant merges/rebases. But, who knows.

YetAnotherNick 6y ago

I am not getting the exact usecase. If the speed is the issue, then I think tf.function solves that. For most of the operation we need to use python via pythonkit, so the things which can't be sped up using tf.function, can't be sped up using swift. Also if we need to use python everywhere, type safety is also very minor.

ddragon 6y ago
The unique property is the ability to just pick any code or library that is unaware of the differentiation library (unlike tf.function as it needs to specifically use tf methods) and get the gradient. In a language like Julia this is immediately useful as it has a massive ecosystem of numerical code that make sense to get gradients (like differential equations and the SciML project [1], or less conventional stuff like raytracers), but in a language like Swift (as there is no meaning to gradient of GUI libraries or frontend stuff) it is more of a "if you build they'll come" faith from Google. But regardless unique features, it's a have cake and eat it too type of interface. You don't need to learn a second language within the language like tensorflow's tf.* making it even more natural and flexiblethan pytorch, including all debug mechanisms of the host language itself, but you still get compile time graph creation like tensorflow, including all kinds of optimizations. It makes other approaches seem primitive by comparison, but creating it is much more complex, and the main audience is already more than used to using language within language solutions (like numpy) which can provide something almost as good even if less elegantly, so it's not easy to convince people as well (when it involves changing programming languages). [1] https://sciml.ai/ https://sciml.ai/
MiroF 6y ago
> exact usecase Static type checking?

pjmlp 6y ago

Still no Windows support at the level of DirectML, or Julia.

bodono 6y ago

I'm not sure this is really going to take off, it seems that most people who are abandoning TF are moving to Jax or pytorch. My own experience with Jax is that it is much easier to use then TF, just an all round more pleasant experience. It would be interesting to try this, but at this point I'm not really willing to learn 'yet another deep learning framework' and the extreme anti-user problems that TF had make me loath to give it another shot, even with a presumably better frontend. Moreover, I think that python is just a better all-round ML/data science language at this point. Has anyone tried both Jax and this and would be willing to give us their thoughts on strengths and weaknesses of each?

alpineidyll3 6y ago
The subtext is Google would love even more Google projects to be ml prerequisites.
gas9S9zw3P9c 6y ago
I'm skeptical of JAX. It feels good right now, but when the first TF beta version came out it was very much like that too - clean, simple, minimal, and just a better version of Theano. Then the "crossing the chasm" effort started and everyone at Google wanted to be part of it, making TF the big complex mess it is today. It's a great example of Conway's Law. I'm not convinced the same won't happen to JAX as it catches on. PyTorch has already stood the test of time and proven that its development is led by a competent team.
- bodono 6y ago
  I know where you're coming from, but TF in my opinion was very user-hostile even on arrival. I can't tell you how much hair-pulling I did over tf.conds, tf.while_loops and the whole gather / scatter paradigm for simple indexing into arrays. I really think the people working on it wanted users to write TF code in a certain, particular way and made it really difficult to use it in other ways. Just thinking back on that time still raises my blood pressure! So far Jax is much better and I'm cautiously optimistic they have learned lessons from TF.

Nelkins 6y ago

There's some cool autodiff work going on in F#/.NET now too[0]. From a usage/API perspective they look kind of similar. [0] https://diffsharp.github.io/ https://diffsharp.github.io/

tanilama 6y ago

I still think it is lacking in this proposal to justify why picking Swift as the host language

suyash 6y ago

For those who prefer an alternate language here is Java for TensorFlow project https://www.tensorflow.org/install/lang_java https://www.tensorflow.org/install/lang_java

Razengan 6y ago

Related: Why Swift for TensorFlow: https://github.com/tensorflow/swift/blob/master/docs/WhySwiftForTensorFlow.md https://github.com/tensorflow/swift/blob/master/docs/WhySwif...

dunefox 6y ago
If you're dead set on using a particular language you can find reasons why it appears suitable even if they don't make any sense.
- Razengan 6y ago
  That logic works both ways.

hedgehog 6y ago

The name is a bit confusing. Swift for TensorFlow combines a bunch of things: - adding autodiff to Swift language & compiler - neural net construction & training API in Swift - low-friction Python bindings - low-friction C++ interop - ability to run neural nets TensorFlow using the C++ interop - ability to alternately run neural nets directly on XLA ("X10") The Swift changes are supposed to get mainlined and I think at least some of the stuff related to Python and C++ already are. The idea is that Swift is nice enough to cover experiment + train + embed as library even on mobile platforms. It's a big engineering project and I hope Google keeps funding the work.

mark_l_watson 6y ago

I question the long term viability of Swift for TensorFlow. I hope that I don’t sound like I am whining but I invested a fair amount of time with Swift from last fall through this February because I wanted: a better faster language to work in for DL than Python; I was also interested in trying some iOS, iPadOS, and macOS development; Swift looked interesting also for Linux server side work. For Apple development I have found Flutter+Dart to be more pleasant. For DL I decided to stick with taking advantage of my 5+ years experience with TF in Python. Off topic, but Julia and Flux are really worth checking out for DL. Some advice: if you want to experiment with Swift for TensorFlow use Google’s colab and make your life easier. I wish I could get back the install times on my Linux GPU laptop and on macOS. If you pay for colab like I do, you get good GPU/TPU resources, and it is simply an easy and fun way to go.

joaogui1 6y ago

I really don't understand why they didn't go with Julia, it fits the purpose much better than Swift, already having a ML ecosystem and having great interop with Python, C, C++, R and Matlab. Heck JAX, where a lot of TF refugees are going, is pretty similar to Zygote

dunefox 6y ago
It's because they were set on Swift and the 'language evaluation' was pointless. Julia would have been the natural choice.

socialdemocrat 6y ago

I am a fan of both Swift and Julia but I honestly don’t see the point of Swift in scientific computing. Julia is just a way better fit. However for App development Swift of course has a far more impressive stack.