7 ms·

Is this somehow similar to the new Swift Automatic Differentiation feature? https://github.com/tensorflow/swift/blob/master/docs/AutomaticDifferentiation.md htt

Is this somehow similar to the new Swift Automatic Differentiation feature? https://github.com/tensorflow/swift/blob/master/docs/AutomaticDifferentiation.md https://github.com/tensorflow/swift/blob/master/docs/Automat...

melling 7y ago
Does Swift have an equivalent to Python‘s numpy? It seems to lack a nice way of doing vector and matrix operations.
- akhilcacharya 7y ago
  They built in Python interop so that you can use numpy in TFSwift
  - jefft255 7y ago
    Doesn’t that introduce serious overhead compared to first class native arrays as in Julia or FORTRAN?
    - superlopuh 7y ago
      My impression is that this is to bootstrap adoption, there is a Swift-native TensorFlow package being developed also.
  - jjoonathan 7y ago
    Bridges always have trolls under them. If you don't see the troll, that doesn't mean it isn't there, it's just waiting for you to cross.
  - tel 7y ago
    Bridging into NumPy isn’t going to be useful unless it covers some AD system too (XLA?).
    - StefanKarpinski 7y ago
      Correct: you can call Python from Swift OR you can use AD but not both at the same time.
    - shoyer 7y ago
      XLA is a compiler for array code. It doesn't come with AD -- you need a wrapper like TF or JAX (or Swift, I guess) for that.
      - tel 7y ago
        I was imagining the FFI emitting XLA code needed to implement the TF/JAX routines behind the scenes, but it's wild speculation.
sadlion 7y ago
Thanks for linking this. I have been curious how auto diff gets implemented.
StefanKarpinski 7y ago
Yes, it's the same idea. But Julia's differentiable programming capabilities are far more advanced and mature than Swift's. As far as I'm aware, Swift still doesn't support differentiating code with control flow (branches or loops), which, needless to say, eliminates pretty much all non-trivial programs. Compare that to the situation in Julia: ∂P works today on arbitrary programs—like the ray tracer and other examples in this paper—programs which are highly non-trivial and use iteration, recursion, mutation and global state. All of which Julia's ∂P can take derivatives through. When you additionally consider Swift's essentially non-existent computational and data science ecosystem, it's a bit hard (for me at least) to rationalize the Swift ∂P effort. (Are we going to differentiate iPhone apps?) They're attempting to bootstrap their computational/data ecosystem by allowing calling Python code, but as soon as you call into Python, you lose all ability to take derivatives which only works for pure Swift code. So any program which relies on Python to do some of the computation you want to take a derivative of won't be differentiable, which kind of defeats the point of having ∂P in the first place. We'll see how it pans out but the Swift effort has considerable technical and social challenges to overcome.
- billconan 7y ago
  How do if conditions handled? They are not differentiatable right? So are loops?
  - StefanKarpinski 7y ago
    There's a whole paper explaining how it all works :)

melling 7y ago

Does Swift have an equivalent to Python‘s numpy? It seems to lack a nice way of doing vector and matrix operations.

akhilcacharya 7y ago
They built in Python interop so that you can use numpy in TFSwift
- jefft255 7y ago
  Doesn’t that introduce serious overhead compared to first class native arrays as in Julia or FORTRAN?
  - superlopuh 7y ago
    My impression is that this is to bootstrap adoption, there is a Swift-native TensorFlow package being developed also.
- jjoonathan 7y ago
  Bridges always have trolls under them. If you don't see the troll, that doesn't mean it isn't there, it's just waiting for you to cross.
- tel 7y ago
  Bridging into NumPy isn’t going to be useful unless it covers some AD system too (XLA?).
  - StefanKarpinski 7y ago
    Correct: you can call Python from Swift OR you can use AD but not both at the same time.
  - shoyer 7y ago
    XLA is a compiler for array code. It doesn't come with AD -- you need a wrapper like TF or JAX (or Swift, I guess) for that.
    - tel 7y ago
      I was imagining the FFI emitting XLA code needed to implement the TF/JAX routines behind the scenes, but it's wild speculation.

sadlion 7y ago

Thanks for linking this. I have been curious how auto diff gets implemented.

StefanKarpinski 7y ago

Yes, it's the same idea. But Julia's differentiable programming capabilities are far more advanced and mature than Swift's. As far as I'm aware, Swift still doesn't support differentiating code with control flow (branches or loops), which, needless to say, eliminates pretty much all non-trivial programs. Compare that to the situation in Julia: ∂P works today on arbitrary programs—like the ray tracer and other examples in this paper—programs which are highly non-trivial and use iteration, recursion, mutation and global state. All of which Julia's ∂P can take derivatives through. When you additionally consider Swift's essentially non-existent computational and data science ecosystem, it's a bit hard (for me at least) to rationalize the Swift ∂P effort. (Are we going to differentiate iPhone apps?) They're attempting to bootstrap their computational/data ecosystem by allowing calling Python code, but as soon as you call into Python, you lose all ability to take derivatives which only works for pure Swift code. So any program which relies on Python to do some of the computation you want to take a derivative of won't be differentiable, which kind of defeats the point of having ∂P in the first place. We'll see how it pans out but the Swift effort has considerable technical and social challenges to overcome.

billconan 7y ago
How do if conditions handled? They are not differentiatable right? So are loops?
- StefanKarpinski 7y ago
  There's a whole paper explaining how it all works :)