5 ms·

Last time I looked the automatic differentiation was in a compiler branch with no immediate plans to merge in master. But overall it is promising. I even insta

by 6d65 6y ago

Last time I looked the automatic differentiation was in a compiler branch with no immediate plans to merge in master. But overall it is promising. I even installed Swift on Linux to play with it, didn't get to ML as I have an AMD GPU and this is a can of worms. Hope it's finished one day. I would prefer for Julia ml libraries to become mainstream. But, it is what it is. Also, the ideal for me would be Rust for tensorflow, but the slower compile times (didn't compare with Swift) are an impediment for an iterative workflow such as tweaking models.

turbinerneiter 6y ago
I recently spend some days playing with differentiable programming in Swift on Linux: * as you said, auto-diff is in a branch or the Google fork of the project * the only pre-built images are for Ubuntu 18.04 * on Linux, the REPL seems to somewhat broken * many libraries are assuming OSX or iOS I don't feel a lot of hope for adoption of Swift on Linux. Apple obviously is not working on that (fair, they have no reason to do so) and the Swift community also has no focus on Linux, since ... they are in the Apple ecosystem. Meanwhile, the Open Source community is much more interested in Rust than Swift. For the differentiable programming - this is what got me excited, but after trying, I was a bit underwhelmed. Not that it isn't great technology, its just not figured out yet. I tried to come up with a use case outside of ML and the one I tried wasn't really applicable. I do feel however that someone will come up with something and that it will have quite some impact.
- 6d65 6y ago
  Yep. It's difficult to find usages for autodiff, and it looks like a very niche thing to add to a language. But, it's still cool. In a language with extensible syntax(ex: proc macros), this would sit in a library. And I think you're right, there is a low chance that this gets traction. Especially since Chris Lattner is at SiFive, they probably will have some kind ofAI accelerators, but TF is for training rather then execution. So not sure they'll find a reason to push it. Jeremy Howard from fast ai might be able to convince people to give it a try. But without people working on it full time, the chances are not great. Especially with a compiler fork that requires constant merges/rebases. But, who knows.
  - improbable22 6y ago
    > It's difficult to find usages for autodiff I guess there are lots of uses in optimisation problems, and in sampling algorithms for statistics. I don't know how easy it will be to sell Swift to people who now use Stan or Stata (or R) and don't think of themselves as programmers. > In a language with extensible syntax(ex: proc macros), this would sit in a library. And this would allow easier iteration of different designs.
    - Scipio_Afri 6y ago
      Does autodiff exist in R? Then why not use R since it seems it also has much ML algo support due to its focus on statistics - its used by many statisticians. Is the idea here that Swift is a more approachable language and thus this is to lower the barrier of entry to TF?
      - 6d65 6y ago
        I think the main selling points of Swift to tensorflow are: * Speed, ML learning pipelines are often bottlenecked by the data load and transformation. TF had new mechanisms (the last I've seen was TF.data). But a language compiled to native is much more flexible in that regard. * Type safety. Sometimes issues with the models can pop long after they have been running. The hope is that typed API's will show simple errors at compile times * Auto differentiation built into the language. If I'm not mistaken, this is more powerful than backpropagation in TF, wich also has autodiff. The idea is that this would allow for more custom models without a performance penalty. My knowledge here is limited, since it's been over 2 years since I've implemented back propagation. I've successfully forgotten most of the things I knew about ML/DL. I don't have any experience with R, but from what I've heard, it was known to be slow. But that might have changed or I may have misunderstood the situation.
        socialdemocrat 6y ago
        Does not need to be built into the language. Julia does it great without special built in support. But Julia is an extremely flexible language. Number types are first class which is a big part of it and you can add extra custom passes to the JIT compiler in regular library code.
- dsabanin 6y ago
  Apple I believe has a reason to work on Swift for Linux, and that reason is their considerable cloud infrastructure and various backend services. I’m sure being able to share domain specific Swift code between client apps and backend would be pretty high on their list of wants. Also, little clues like the way Xcode generates SwiftPM packages in a Linux-ready fashion out of the box shows that they care at least a bit. Having a lot of interest in the programming languages, my opinion is that Swift is a damn good one. It’s very high level, supports FP deep enough, has a great type system (that is getting better with every release), great OOP support, native performance characteristics and it still lets you get to a really low level when you need it. I also like how they took great ideas from Haskell, Scala, Smalltalk, C# and others. I code daily in Scala and Swift, previously had done Erlang, Clojure, Common LISP, TypeScript, Ruby, Python, Haskell, OCaml, Java, PHP, C, Smalltalk and some others. In this list, Swift is now almost at the top. They need to get Higher-Kinded Types and then it’s going to win the world (just kidding, JavaScript gets to win the world, unfortunately) :)
  - threatofrain 6y ago
    If Apple has intrinsic care for Swift on Linux then let it be evident in their story or direction.
    - dsabanin 6y ago
      I can't speak for Apple, of course, but some indications of their seriousness are there. SwiftNIO is a cross-platform asynchronous event-driven network application framework for rapid development of maintainable high performance protocol servers & clients. (https://github.com/apple/swift-nio https://github.com/apple/swift-nio) Distributed Membership Protocol implementations in Swift: https://github.com/apple/swift-cluster-membership https://github.com/apple/swift-cluster-membership Docker Official Image packaging for Swift: https://github.com/apple/swift-docker https://github.com/apple/swift-docker Also, on official https://swift.org/download https://swift.org/download all releases and snapshots are automatically available for: Ubuntu 16.04, Ubuntu 18.04, Ubuntu 20.04, CentOS 7, CentOS 8, Amazon Linux 2 Official Swift Server work group is a steering team that promotes the use of Swift for developing and deploying server applications: https://swift.org/server/ https://swift.org/server/ Swift AWS Lambda Runtime: https://github.com/swift-server/swift-aws-lambda-runtime https://github.com/swift-server/swift-aws-lambda-runtime Official Swift 5.3 release goals have stated a major goal of "expanding the number of platforms where Swift is available and supported, notably adding support for Windows and additional Linux distributions." (https://swift.org/blog/5-3-release-process/ https://swift.org/blog/5-3-release-process/)
- shadowfiend 6y ago
  > Apple obviously is not working on that (fair, they have no reason to do so) and the Swift community also has no focus on Linux, since ... they are in the Apple ecosystem. Neither of these things are true, fwiw. Strong Linux support is an explicit current goal for Swift, though it is admittedly not all the way there yet. See this post regarding Swift 6 and its goals for wider support, for example: https://forums.swift.org/t/on-the-road-to-swift-6/32862 https://forums.swift.org/t/on-the-road-to-swift-6/32862
cs702 6y ago
A Rust for TensorFlow (and/or a "RustTorch") would be awesome. I hope all the work being done on improving incremental compilation[a] and developing interactive Rust REPLs like evcxr[b] makes using Rust for AI a practical reality. [a] https://doc.rust-lang.org/edition-guide/rust-2018/the-compiler/incremental-compilation-for-faster-compiles.html https://doc.rust-lang.org/edition-guide/rust-2018/the-compil... [b] https://github.com/google/evcxr https://github.com/google/evcxr
- 6d65 6y ago
  I think there are official tensorflow bindings for Rust, add well as for pytorch C++ API. But, adding auto differentiation to match the Swift for tensorflow behaviour, sounds like a serious undertaking, and I doubt is on anyone's radar. But yeah, I've been wanting this for a while. Shoehorning Rust everywhere is the endgame.
- anoncept 6y ago
  As a starting point, maybe take a look at the 'tch' crate [1] and the 'rust-bert' crate [2] built on top of it? [1]: https://github.com/LaurentMazare/tch-rs https://github.com/LaurentMazare/tch-rs -> https://crates.io/crates/tch https://crates.io/crates/tch [2]: https://github.com/guillaume-be/rust-bert https://github.com/guillaume-be/rust-bert -> https://crates.io/crates/rust-bert https://crates.io/crates/rust-bert
  - danieldk 6y ago
    tch-rs is really nice. I have built a Rust sequence labeler + dependency parser + lemmatizer on top of it. It supports multi-task learning, finetuning of BERT-like models, model distillation, etc.: https://github.com/stickeritis/sticker2/ https://github.com/stickeritis/sticker2/ Unfortunately, when I started this guillaume-be's rust-bert crate wasn't around yet (I think), so I also ported some of the models from Huggingface transformers: https://github.com/stickeritis/sticker-transformers/ https://github.com/stickeritis/sticker-transformers/ At any rate, I can highly recommend the tch crate if you are looking to build neural networks in Rust.
- adamnemecek 6y ago

by 6d65 6y ago

turbinerneiter 6y ago

I recently spend some days playing with differentiable programming in Swift on Linux: * as you said, auto-diff is in a branch or the Google fork of the project * the only pre-built images are for Ubuntu 18.04 * on Linux, the REPL seems to somewhat broken * many libraries are assuming OSX or iOS I don't feel a lot of hope for adoption of Swift on Linux. Apple obviously is not working on that (fair, they have no reason to do so) and the Swift community also has no focus on Linux, since ... they are in the Apple ecosystem. Meanwhile, the Open Source community is much more interested in Rust than Swift. For the differentiable programming - this is what got me excited, but after trying, I was a bit underwhelmed. Not that it isn't great technology, its just not figured out yet. I tried to come up with a use case outside of ML and the one I tried wasn't really applicable. I do feel however that someone will come up with something and that it will have quite some impact.

6d65 6y ago
Yep. It's difficult to find usages for autodiff, and it looks like a very niche thing to add to a language. But, it's still cool. In a language with extensible syntax(ex: proc macros), this would sit in a library. And I think you're right, there is a low chance that this gets traction. Especially since Chris Lattner is at SiFive, they probably will have some kind ofAI accelerators, but TF is for training rather then execution. So not sure they'll find a reason to push it. Jeremy Howard from fast ai might be able to convince people to give it a try. But without people working on it full time, the chances are not great. Especially with a compiler fork that requires constant merges/rebases. But, who knows.
- improbable22 6y ago
  > It's difficult to find usages for autodiff I guess there are lots of uses in optimisation problems, and in sampling algorithms for statistics. I don't know how easy it will be to sell Swift to people who now use Stan or Stata (or R) and don't think of themselves as programmers. > In a language with extensible syntax(ex: proc macros), this would sit in a library. And this would allow easier iteration of different designs.
  - Scipio_Afri 6y ago
    Does autodiff exist in R? Then why not use R since it seems it also has much ML algo support due to its focus on statistics - its used by many statisticians. Is the idea here that Swift is a more approachable language and thus this is to lower the barrier of entry to TF?
    - 6d65 6y ago
      I think the main selling points of Swift to tensorflow are: * Speed, ML learning pipelines are often bottlenecked by the data load and transformation. TF had new mechanisms (the last I've seen was TF.data). But a language compiled to native is much more flexible in that regard. * Type safety. Sometimes issues with the models can pop long after they have been running. The hope is that typed API's will show simple errors at compile times * Auto differentiation built into the language. If I'm not mistaken, this is more powerful than backpropagation in TF, wich also has autodiff. The idea is that this would allow for more custom models without a performance penalty. My knowledge here is limited, since it's been over 2 years since I've implemented back propagation. I've successfully forgotten most of the things I knew about ML/DL. I don't have any experience with R, but from what I've heard, it was known to be slow. But that might have changed or I may have misunderstood the situation.
      - socialdemocrat 6y ago
        Does not need to be built into the language. Julia does it great without special built in support. But Julia is an extremely flexible language. Number types are first class which is a big part of it and you can add extra custom passes to the JIT compiler in regular library code.
dsabanin 6y ago
Apple I believe has a reason to work on Swift for Linux, and that reason is their considerable cloud infrastructure and various backend services. I’m sure being able to share domain specific Swift code between client apps and backend would be pretty high on their list of wants. Also, little clues like the way Xcode generates SwiftPM packages in a Linux-ready fashion out of the box shows that they care at least a bit. Having a lot of interest in the programming languages, my opinion is that Swift is a damn good one. It’s very high level, supports FP deep enough, has a great type system (that is getting better with every release), great OOP support, native performance characteristics and it still lets you get to a really low level when you need it. I also like how they took great ideas from Haskell, Scala, Smalltalk, C# and others. I code daily in Scala and Swift, previously had done Erlang, Clojure, Common LISP, TypeScript, Ruby, Python, Haskell, OCaml, Java, PHP, C, Smalltalk and some others. In this list, Swift is now almost at the top. They need to get Higher-Kinded Types and then it’s going to win the world (just kidding, JavaScript gets to win the world, unfortunately) :)
- threatofrain 6y ago
  If Apple has intrinsic care for Swift on Linux then let it be evident in their story or direction.
  - dsabanin 6y ago
    I can't speak for Apple, of course, but some indications of their seriousness are there. SwiftNIO is a cross-platform asynchronous event-driven network application framework for rapid development of maintainable high performance protocol servers & clients. (https://github.com/apple/swift-nio https://github.com/apple/swift-nio) Distributed Membership Protocol implementations in Swift: https://github.com/apple/swift-cluster-membership https://github.com/apple/swift-cluster-membership Docker Official Image packaging for Swift: https://github.com/apple/swift-docker https://github.com/apple/swift-docker Also, on official https://swift.org/download https://swift.org/download all releases and snapshots are automatically available for: Ubuntu 16.04, Ubuntu 18.04, Ubuntu 20.04, CentOS 7, CentOS 8, Amazon Linux 2 Official Swift Server work group is a steering team that promotes the use of Swift for developing and deploying server applications: https://swift.org/server/ https://swift.org/server/ Swift AWS Lambda Runtime: https://github.com/swift-server/swift-aws-lambda-runtime https://github.com/swift-server/swift-aws-lambda-runtime Official Swift 5.3 release goals have stated a major goal of "expanding the number of platforms where Swift is available and supported, notably adding support for Windows and additional Linux distributions." (https://swift.org/blog/5-3-release-process/ https://swift.org/blog/5-3-release-process/)
shadowfiend 6y ago
> Apple obviously is not working on that (fair, they have no reason to do so) and the Swift community also has no focus on Linux, since ... they are in the Apple ecosystem. Neither of these things are true, fwiw. Strong Linux support is an explicit current goal for Swift, though it is admittedly not all the way there yet. See this post regarding Swift 6 and its goals for wider support, for example: https://forums.swift.org/t/on-the-road-to-swift-6/32862 https://forums.swift.org/t/on-the-road-to-swift-6/32862

cs702 6y ago

A Rust for TensorFlow (and/or a "RustTorch") would be awesome. I hope all the work being done on improving incremental compilation[a] and developing interactive Rust REPLs like evcxr[b] makes using Rust for AI a practical reality. [a] https://doc.rust-lang.org/edition-guide/rust-2018/the-compiler/incremental-compilation-for-faster-compiles.html https://doc.rust-lang.org/edition-guide/rust-2018/the-compil... [b] https://github.com/google/evcxr https://github.com/google/evcxr

6d65 6y ago
I think there are official tensorflow bindings for Rust, add well as for pytorch C++ API. But, adding auto differentiation to match the Swift for tensorflow behaviour, sounds like a serious undertaking, and I doubt is on anyone's radar. But yeah, I've been wanting this for a while. Shoehorning Rust everywhere is the endgame.
anoncept 6y ago
As a starting point, maybe take a look at the 'tch' crate [1] and the 'rust-bert' crate [2] built on top of it? [1]: https://github.com/LaurentMazare/tch-rs https://github.com/LaurentMazare/tch-rs -> https://crates.io/crates/tch https://crates.io/crates/tch [2]: https://github.com/guillaume-be/rust-bert https://github.com/guillaume-be/rust-bert -> https://crates.io/crates/rust-bert https://crates.io/crates/rust-bert
- danieldk 6y ago
  tch-rs is really nice. I have built a Rust sequence labeler + dependency parser + lemmatizer on top of it. It supports multi-task learning, finetuning of BERT-like models, model distillation, etc.: https://github.com/stickeritis/sticker2/ https://github.com/stickeritis/sticker2/ Unfortunately, when I started this guillaume-be's rust-bert crate wasn't around yet (I think), so I also ported some of the models from Huggingface transformers: https://github.com/stickeritis/sticker-transformers/ https://github.com/stickeritis/sticker-transformers/ At any rate, I can highly recommend the tch crate if you are looking to build neural networks in Rust.
adamnemecek 6y ago