5 ms·

It's interesting how it's going to play out. On one hand side, Swift is a pleasant language to work with (despite its infancy). But on the other, having a Tenso

by 3jckd 8y ago

It's interesting how it's going to play out. On one hand side, Swift is a pleasant language to work with (despite its infancy). But on the other, having a Tensorflow API doesn't suddenly give it a bunch of libraries for statistics, comp. vision, modeling, visualisation, etc. that Python/R/Julia coughMATLABcough have. Nowadays, it's difficult enough to convince people to drop e.g. MATLAB for R or Python for Julia (let's assume that there's some merit to it), despite them having excellent counterparts for almost everything. Swift's success in this domain depends solely on the adoption by developers/researchers/engineers. Unless they're just going to mostly use it internally (as Google is known to). Which brings me to the last point - why on Earth would they pick Swift (apart from Christ Lattner being involved) when Julia was on the table? It ticks all their boxes and has more mature ecosystem for all things "data". Provided rationale is hardly convincing.

marmaduke 8y ago
Julia doesn't tick the "compile to .o/.h" box. As far as I can tell, the use case for AOT Julia is avoiding package compilation overhead, not delivery of standalone code objects. _edit_ seems JuliaC does support this sort of thing: https://juliacomputing.com/blog/2016/02/09/static-julia.html https://juliacomputing.com/blog/2016/02/09/static-julia.html
- StefanKarpinski 8y ago
  Julia does already support this kind of thing. Moreover with a minuscule fraction of the money that’s being poured into making Swift usable for data science and machine learning, truly top notch support for generating standalone binaries from Julia could readily be developed. Which is kind of frustrating but what can you do?
- celrod 8y ago
  If you'd like to track the latest developments: https://github.com/JuliaLang/PackageCompiler.jl https://github.com/JuliaLang/PackageCompiler.jl
aschampion 8y ago
Yeah, I don't buy the justification versus Julia because of community size either, given most of Swift's community has little to do with data science. The document even says as much, contradicting that rationale, later on. As someone who uses TF heavily, I would be much more excited about this project if they'd chosen Julia. Swift's tooling isn't great, and I already have a foot in one language with an immature data science ecosystem (Rust).
- tomjakubowski 8y ago
  How is Rust-Julia interop?
  - celrod 8y ago
    Julia's `ccall` is great in terms of overhead[0], so calling Rust shared libraries is not a problem. On the Rust side, it took me a while to figure out passing in pointers, and then constructing slices via from_raw_parts[_mut], so that I can transition to safe Rust. Perhaps that is obvious to more experienced Rust programmers, but I was left with the impression that receiving pointers and crunching numbers is not yet a common application for Rust (unlike C, C++, or Fortran). Meaning there is not a lot of introductory material coming from that angle at the moment. Additionally, to get good vectorization seems to require nightly and a fastfloat[1] library. In particular, you'd want associative math / fp-contract for fma and SIMD instructions, and perhaps fno-math-errno to turn off branches in functions like sqrt. I imagine calling Rust from Julia will be much more common than calling Julia from Rust. I know approximately nothing about this, but there are plenty of questions about embedding Julia into C/C++[2][3]. May be similar for Rust. [0] https://github.com/dyu/ffi-overhead https://github.com/dyu/ffi-overhead [1] https://github.com/robsmith11/fastfloat https://github.com/robsmith11/fastfloat [2] https://discourse.julialang.org/t/support-with-embedding-julia-in-c/5054 https://discourse.julialang.org/t/support-with-embedding-jul... [3] https://discourse.julialang.org/t/api-reference-for-julia-embedding-in-c/3963/10 https://discourse.julialang.org/t/api-reference-for-julia-em...
microcolonel 8y ago
I feel like GraalVM has a chance to solve some of this at least. I wonder if anyone will make an Octave GraalVM frontend, they already have one for R.
- c-cube 8y ago
  Isn't Graal an Oracle thing? I don't understand why anyone would want to touch that even with a 10-foot pole.
  - microcolonel 8y ago
    Yeah, I would tend to agree. If Google successfully appeals Oracle's suit against them for implementing Java, I might consider using it in a product, but Oracle makes it legally risky to use any of their products.
ychen306 8y ago
I will say this in the risk of talking out of my ass as I have no experience in either language :). Having a statically typed language greatly simplifies the tooling because static analysis is much easier; graph program extraction involves one such analysis. When you have to deploy the trained model in production one would hope not to use Python or Julia. I'd like to add that, with my limited experience in prototyping some of my ML models, having a static checker to check that your tensors have the right shape is much better than having to run your code.
- dnautics 8y ago
  what's the problem with deploying julia in production in inference? Some occasional piece of data that looks wrong in an unanticipated way causes a runtime type fault? People deploy high uptime websites with django - how do they do it? Well you use kubernetes (or, gasp, systemd) and have restart and load balancing logic. Even if you were typecheck-compiled, you can't guarantee some other developer logic or system error, or an errant bit flip from a cosmic ray, won't take your setup down. Static checker doesn't really matter. If you're at the point where you're ready to deploy, you're probably good for at least 95-99% of the data you'll ingest. The rest of the gap can be closed using rolling update.
  - shele 8y ago
    On the other hand, Julia can do the right thing dynamically. Your matrix happens to be symmetric? Julia will chose an appropriate factorisation and will propagate that knowledge through dynamic dispatch.
    - skohan 8y ago
      "Automagical" solutions tend to work until they don't. I would much rather have my solution be provably correct at compilation time than to depend on high-overhead runtime systems.
      - dnautics 8y ago
cageface 8y ago
Swift is a nice language but its reliance on reference counting means you have to work a lot harder to avoid retain cycles than you do in a garbage collected language. That might have been the right choice for Apple’s uses of Swift where GC pauses affect the user experience but for most other use cases it’s too much of a cognitive burden IMO.
- saagarjha 8y ago
  Personally, I find that this only really comes up rarely. Most of the time strong references are fine.
  - cageface 8y ago
    My iOS code is loaded with weak/strong ref handling logic. It comes up all the time when using closures in UIKit.
    - saagarjha 8y ago
      UIKit is kind of annoying because it’s really not “idiomatic” Swift: it’s a wrapper around Objective-C (albeit, a very nice one) that happens to bring along with it a bunch of decisions that lead to having to deal with reference lifetimes.
      - cageface 8y ago
        That could be. I'll admit I haven't used Swift at all outside of coding Appkit/UIKit apps.
    - dep_b 8y ago
      iOS Swift is usually the front-end code which means that you're doing a lot of connections to data sources elsewhere. Non-blocking connections always need a weak/strong dance. If you're using Swift for logic most of that is gone.
plg 8y ago
> why on Earth would they pick Swift Because of iOS?
- phillipcarter 8y ago
  iOS isn't really relevant for this. You would certainly deploy a model into an app, but that is likely to be using ONNX: https://medium.com/@alexiscreuzot/building-a-neural-style-transfer-app-on-ios-with-pytorch-and-coreml-76e00cd14b28 https://medium.com/@alexiscreuzot/building-a-neural-style-tr...
agibsonccc 8y ago
I'd argue in general that outside of python, there's not really much of a focus from google itself. They are largely leaving other language bindings to other people (see what's currently going on with tensorflow 2.0). Their focus is more on the c bindings and allowing other people to build what they want on top of that. Other language bindings aren't generally going to be used for anything more than inference. First class actual data science work isn't going to happen in other languages anytime soon (at least outside of julia and R which are at least trying to compete in this niche).
newen 8y ago
Hah...it's funny how they pretend to give objective rationales for choosing Swift when it's pretty clear the decision was made long before.
skohan 8y ago
> Tensorflow API doesn't suddenly give it a bunch of libraries for statistics, comp. vision, modeling, visualisation, etc. that Python/R/Julia coughMATLABcough have. Actually in this case it does. Swift for Tensorflow includes python interop out of the box: https://www.tensorflow.org/swift/api_docs/Global-Variables#/s:10TensorFlow6PythonAA0C9InterfaceVvp https://www.tensorflow.org/swift/api_docs/Global-Variables#/... The supported use-case would be to do your ML work in Swift, and then call numby etc. from Python.

by 3jckd 8y ago

marmaduke 8y ago

Julia doesn't tick the "compile to .o/.h" box. As far as I can tell, the use case for AOT Julia is avoiding package compilation overhead, not delivery of standalone code objects. _edit_ seems JuliaC does support this sort of thing: https://juliacomputing.com/blog/2016/02/09/static-julia.html https://juliacomputing.com/blog/2016/02/09/static-julia.html

StefanKarpinski 8y ago
Julia does already support this kind of thing. Moreover with a minuscule fraction of the money that’s being poured into making Swift usable for data science and machine learning, truly top notch support for generating standalone binaries from Julia could readily be developed. Which is kind of frustrating but what can you do?
celrod 8y ago
If you'd like to track the latest developments: https://github.com/JuliaLang/PackageCompiler.jl https://github.com/JuliaLang/PackageCompiler.jl

aschampion 8y ago

Yeah, I don't buy the justification versus Julia because of community size either, given most of Swift's community has little to do with data science. The document even says as much, contradicting that rationale, later on. As someone who uses TF heavily, I would be much more excited about this project if they'd chosen Julia. Swift's tooling isn't great, and I already have a foot in one language with an immature data science ecosystem (Rust).

tomjakubowski 8y ago
How is Rust-Julia interop?
- celrod 8y ago
  Julia's `ccall` is great in terms of overhead[0], so calling Rust shared libraries is not a problem. On the Rust side, it took me a while to figure out passing in pointers, and then constructing slices via from_raw_parts[_mut], so that I can transition to safe Rust. Perhaps that is obvious to more experienced Rust programmers, but I was left with the impression that receiving pointers and crunching numbers is not yet a common application for Rust (unlike C, C++, or Fortran). Meaning there is not a lot of introductory material coming from that angle at the moment. Additionally, to get good vectorization seems to require nightly and a fastfloat[1] library. In particular, you'd want associative math / fp-contract for fma and SIMD instructions, and perhaps fno-math-errno to turn off branches in functions like sqrt. I imagine calling Rust from Julia will be much more common than calling Julia from Rust. I know approximately nothing about this, but there are plenty of questions about embedding Julia into C/C++[2][3]. May be similar for Rust. [0] https://github.com/dyu/ffi-overhead https://github.com/dyu/ffi-overhead [1] https://github.com/robsmith11/fastfloat https://github.com/robsmith11/fastfloat [2] https://discourse.julialang.org/t/support-with-embedding-julia-in-c/5054 https://discourse.julialang.org/t/support-with-embedding-jul... [3] https://discourse.julialang.org/t/api-reference-for-julia-embedding-in-c/3963/10 https://discourse.julialang.org/t/api-reference-for-julia-em...

microcolonel 8y ago

I feel like GraalVM has a chance to solve some of this at least. I wonder if anyone will make an Octave GraalVM frontend, they already have one for R.

c-cube 8y ago
Isn't Graal an Oracle thing? I don't understand why anyone would want to touch that even with a 10-foot pole.
- microcolonel 8y ago
  Yeah, I would tend to agree. If Google successfully appeals Oracle's suit against them for implementing Java, I might consider using it in a product, but Oracle makes it legally risky to use any of their products.

ychen306 8y ago

I will say this in the risk of talking out of my ass as I have no experience in either language :). Having a statically typed language greatly simplifies the tooling because static analysis is much easier; graph program extraction involves one such analysis. When you have to deploy the trained model in production one would hope not to use Python or Julia. I'd like to add that, with my limited experience in prototyping some of my ML models, having a static checker to check that your tensors have the right shape is much better than having to run your code.

dnautics 8y ago
what's the problem with deploying julia in production in inference? Some occasional piece of data that looks wrong in an unanticipated way causes a runtime type fault? People deploy high uptime websites with django - how do they do it? Well you use kubernetes (or, gasp, systemd) and have restart and load balancing logic. Even if you were typecheck-compiled, you can't guarantee some other developer logic or system error, or an errant bit flip from a cosmic ray, won't take your setup down. Static checker doesn't really matter. If you're at the point where you're ready to deploy, you're probably good for at least 95-99% of the data you'll ingest. The rest of the gap can be closed using rolling update.
- shele 8y ago
  On the other hand, Julia can do the right thing dynamically. Your matrix happens to be symmetric? Julia will chose an appropriate factorisation and will propagate that knowledge through dynamic dispatch.
  - skohan 8y ago
    "Automagical" solutions tend to work until they don't. I would much rather have my solution be provably correct at compilation time than to depend on high-overhead runtime systems.
    - dnautics 8y ago

cageface 8y ago

Swift is a nice language but its reliance on reference counting means you have to work a lot harder to avoid retain cycles than you do in a garbage collected language. That might have been the right choice for Apple’s uses of Swift where GC pauses affect the user experience but for most other use cases it’s too much of a cognitive burden IMO.

saagarjha 8y ago
Personally, I find that this only really comes up rarely. Most of the time strong references are fine.
- cageface 8y ago
  My iOS code is loaded with weak/strong ref handling logic. It comes up all the time when using closures in UIKit.
  - saagarjha 8y ago
    UIKit is kind of annoying because it’s really not “idiomatic” Swift: it’s a wrapper around Objective-C (albeit, a very nice one) that happens to bring along with it a bunch of decisions that lead to having to deal with reference lifetimes.
    - cageface 8y ago
      That could be. I'll admit I haven't used Swift at all outside of coding Appkit/UIKit apps.
  - dep_b 8y ago
    iOS Swift is usually the front-end code which means that you're doing a lot of connections to data sources elsewhere. Non-blocking connections always need a weak/strong dance. If you're using Swift for logic most of that is gone.

plg 8y ago

> why on Earth would they pick Swift Because of iOS?

phillipcarter 8y ago
iOS isn't really relevant for this. You would certainly deploy a model into an app, but that is likely to be using ONNX: https://medium.com/@alexiscreuzot/building-a-neural-style-transfer-app-on-ios-with-pytorch-and-coreml-76e00cd14b28 https://medium.com/@alexiscreuzot/building-a-neural-style-tr...

agibsonccc 8y ago

I'd argue in general that outside of python, there's not really much of a focus from google itself. They are largely leaving other language bindings to other people (see what's currently going on with tensorflow 2.0). Their focus is more on the c bindings and allowing other people to build what they want on top of that. Other language bindings aren't generally going to be used for anything more than inference. First class actual data science work isn't going to happen in other languages anytime soon (at least outside of julia and R which are at least trying to compete in this niche).

newen 8y ago

Hah...it's funny how they pretend to give objective rationales for choosing Swift when it's pretty clear the decision was made long before.

skohan 8y ago

> Tensorflow API doesn't suddenly give it a bunch of libraries for statistics, comp. vision, modeling, visualisation, etc. that Python/R/Julia coughMATLABcough have. Actually in this case it does. Swift for Tensorflow includes python interop out of the box: https://www.tensorflow.org/swift/api_docs/Global-Variables#/s:10TensorFlow6PythonAA0C9InterfaceVvp https://www.tensorflow.org/swift/api_docs/Global-Variables#/... The supported use-case would be to do your ML work in Swift, and then call numby etc. from Python.