4 ms·

> There's nothing in Swift that's inherently suitable for ML The type system?

by MiroF 6y ago

mkolodny 6y ago
Speed, too. For PyTorch to train models and run inference quickly, your Python code gets translated to C++/CUDA. Part of the idea with S4TF is to be able to write ML code in a single, fast language.
- MiroF 6y ago
  I am skeptical that there are no calls to underlying C or CUDA libraries occurring. Swift doesn’t naively beat BLAS
- socialdemocrat 6y ago
  You can already do that in Julia and unlike Swift that is a language with lots of scientific and machine learning libraries. I like Swift but looking at the code examples I got to say preparing data and setting up a model is 10x easier in Julia.
- dklend122 6y ago
  Well, it's not happening in swift yet. S4TF still requires either c cuda kernels or XLA. Julia on the other hand has JIT GPU codegen and its CPU codegen has been benchmarked to beat openblas
  - MiroF 6y ago
    > CPU codegen has been benchmarked to beat openblas Source on this claim?
    - ddragon 6y ago
      He probably means Tullio.jl (which also seems to integrate with Julia's source to source differentiation library Zygote.jl, the main competitor to Swift for Tensorflow): https://discourse.julialang.org/t/realistically-how-close-is-gaius-jl-to-becoming-a-full-replacement-for-blas-in-julia/42780/5 https://discourse.julialang.org/t/realistically-how-close-is... https://github.com/mcabbott/Tullio.jl https://github.com/mcabbott/Tullio.jl Regardless if it can consistently beat Fortran/BLAS in every area, in general JIT languages have more opportunities for optimizations than AoT languages, so it's interesting to see what comes out of a language that focuses on leveraging this to get the most performance.
      - MiroF 6y ago
        I'm surprised re: blas - that is closer than I thought! Still a huge gap on the GPU, but still impressive. > in general JIT languages have more opportunities for optimizations than AoT languages I'm not sure I agree with this. I would say the opportunities for any non-GC mature AoT language (C++, Rust, etc.) is going to be pretty much the same, since you can just attach a JIT to most AoT langs.
        amkkma 6y ago
        The GPU gap is only if written in the high level index or loop style. There is little to no gap if done either using array abstractions (broadcast, map etc) or at a level similar to Cuda C (though with nicer Julia abstractions and syntax): https://juliagpu.org/cuda/ https://juliagpu.org/cuda/ The Julialab at MIT is working on making the higher level codegen faster
        MiroF 6y ago
        I guess that makes sense to me.. you can just automatically convert the C in BLAS to Julia and then if they're both being converted to llvm ir by clang anyways than i guess it'll be about as fast!
        amkkma 6y ago
        That's not at all what Julia is doing. It's much more sophisticated in that it has very low level intrinsic primitives that can compose and it optimizes the IR to make it fast and then compiles it to CUDA. These all map to Julia constructs.
        ddragon 6y ago
        Sure, if you give static language a JIT they'll be able to get the advantages of having JIT, though language semantics still matter. A language built for JITs like Julia or Common Lisp have native ways of interfacing with the compiler, and programs are built without worry of exponential explosion of implementations during method monomorphization (as you'll only compile the optimal versions that you'll actually use, based on runtime information, without having to be pessimist as any overspecialization can be fixed on demand). AoT languages would probably need a compiler pragma or type similar to a dynamic boxing but for delayed monomorphization/compilation for methods when you want to avoid compiling all paths AoT (which might be a way to allow for example tensor specialization on sizes, similar to StaticArrays on Julia).

> There's nothing in Swift that's inherently suitable for ML The type system?

by MiroF 6y ago

> There's nothing in Swift that's inherently suitable for ML The type system?

Speed, too. For PyTorch to train models and run inference quickly, your Python code gets translated to C++/CUDA. Part of the idea with S4TF is to be able to write ML code in a single, fast language.

MiroF 6y ago
I am skeptical that there are no calls to underlying C or CUDA libraries occurring. Swift doesn’t naively beat BLAS
socialdemocrat 6y ago
You can already do that in Julia and unlike Swift that is a language with lots of scientific and machine learning libraries. I like Swift but looking at the code examples I got to say preparing data and setting up a model is 10x easier in Julia.
dklend122 6y ago
Well, it's not happening in swift yet. S4TF still requires either c cuda kernels or XLA. Julia on the other hand has JIT GPU codegen and its CPU codegen has been benchmarked to beat openblas
- MiroF 6y ago
  > CPU codegen has been benchmarked to beat openblas Source on this claim?
  - ddragon 6y ago
    He probably means Tullio.jl (which also seems to integrate with Julia's source to source differentiation library Zygote.jl, the main competitor to Swift for Tensorflow): https://discourse.julialang.org/t/realistically-how-close-is-gaius-jl-to-becoming-a-full-replacement-for-blas-in-julia/42780/5 https://discourse.julialang.org/t/realistically-how-close-is... https://github.com/mcabbott/Tullio.jl https://github.com/mcabbott/Tullio.jl Regardless if it can consistently beat Fortran/BLAS in every area, in general JIT languages have more opportunities for optimizations than AoT languages, so it's interesting to see what comes out of a language that focuses on leveraging this to get the most performance.
    - MiroF 6y ago
      I'm surprised re: blas - that is closer than I thought! Still a huge gap on the GPU, but still impressive. > in general JIT languages have more opportunities for optimizations than AoT languages I'm not sure I agree with this. I would say the opportunities for any non-GC mature AoT language (C++, Rust, etc.) is going to be pretty much the same, since you can just attach a JIT to most AoT langs.
      - amkkma 6y ago
        The GPU gap is only if written in the high level index or loop style. There is little to no gap if done either using array abstractions (broadcast, map etc) or at a level similar to Cuda C (though with nicer Julia abstractions and syntax): https://juliagpu.org/cuda/ https://juliagpu.org/cuda/ The Julialab at MIT is working on making the higher level codegen faster
        MiroF 6y ago
        I guess that makes sense to me.. you can just automatically convert the C in BLAS to Julia and then if they're both being converted to llvm ir by clang anyways than i guess it'll be about as fast!
        amkkma 6y ago
        That's not at all what Julia is doing. It's much more sophisticated in that it has very low level intrinsic primitives that can compose and it optimizes the IR to make it fast and then compiles it to CUDA. These all map to Julia constructs.
      - ddragon 6y ago
        Sure, if you give static language a JIT they'll be able to get the advantages of having JIT, though language semantics still matter. A language built for JITs like Julia or Common Lisp have native ways of interfacing with the compiler, and programs are built without worry of exponential explosion of implementations during method monomorphization (as you'll only compile the optimal versions that you'll actually use, based on runtime information, without having to be pessimist as any overspecialization can be fixed on demand). AoT languages would probably need a compiler pragma or type similar to a dynamic boxing but for delayed monomorphization/compilation for methods when you want to avoid compiling all paths AoT (which might be a way to allow for example tensor specialization on sizes, similar to StaticArrays on Julia).