7 ms·

Does anyone here use Swift for Deep Learning work ? I'm yet to see any real advantage, and even if there were or were to be added, I think Julia would almost al

Does anyone here use Swift for Deep Learning work ? I'm yet to see any real advantage, and even if there were or were to be added, I think Julia would almost always be a better choice here.

gaogao 7y ago
“Why didn’t they choose Julia"
- gaogao 7y ago
  I'm quoting the article on how the first HN comment will be this.
- threw4234324 7y ago
  Well, why though ? Julia has plenty of numerical libraries going for it, and all of them can be used in a differentiable manner (see, Neural ODE etc.). While Julia was built by a mix of people doing Numerical Linear Algebra and PLs (MIT, UCSD..), Swift seems to be entirely the output of the latter group.
  - choeger 7y ago
    Maybe because swift is statically typed, whereas Julia is not?
    - pjmlp 7y ago
      Julia supports type annotations though.
      - choeger 7y ago
        So does python. That does not make it a typed language.
        ddragon 7y ago
        Julia is compiled unlike Python though (at least by default), the moment you call Zygote it will have to run through the entire program you want to differentiate and fail immediately if any type does not match (without the need of any type annotation besides in the method arguments for the multiple dispatch and structs, as Julia has type inference). As some people say, Julia will fail "Just Ahead of Time", and if you're experimenting with live code in the REPL or Jupyter (the recommended workflow) it could very well be just as good. That said, for machine learning a static type system (as they are now) is definitely not as much of a boom as in most other areas. Most of it involves tensor operations, and embedding things like it's shape in the generics will often lead to an exponential explosion of possible monomorphizations that the compiler will be forced to create. Even JIT languages like Julia will have trouble even though it only needs to compile when it's used (for example StaticArrays will get to a point when the compiler will take so much time that it's not worth it anymore). And even then I feel like most of the issues that actually take time are deeper than stuff like shapes and not naming dimensions, like numerical instabilities, hyperparameter tuning, architecture search and other stuff that gets more benefit of a language allowing quicker exploration.
        choeger 7y ago
        Python is also compiled to bytecode in the mainstream implementation. But that is an implementation detail and has nothing to do with the (lack of a) type system.
        ddragon 7y ago
        Julia is also "compiled" down to it's (untyped) IR before even starting (when all macros are expanded), but since it's a dynamic language like Python, it can't know types at compile-time, so this step can only know parser errors. It could only type check at this step if the type was in the definition and not in the variable inside (if it was static). And both Julia and Python are also strongly typed (there isn't really a language that lacks a type system). The difference is that whenever a function in called in Julia, based on the type of the arguments the compiler can infer every type of every variable and arguments of every subsequent function, immediately compiling them down to machine code (unlike Python there is no such thing as a Julia Virtual Machine or even a Julia Interpreter outside of the debugger). Whenever you enter a function in Julia it becomes a static program, with all the properties of a static language (for example, you can't redefine types, you can define functions but the program can't see them since they are not part of the compiled code, you can't import libraries, and all types were already checked before running). That's why Julia is fast, and the language was entirely designed for working this way (there are lots of things you can do in Python that you can't in Julia to make this work).
  - random_moonwalk 7y ago
    I wonder how important more superficial elements will be in terms of future adoption. As a Python programmer Swift looks much more familiar and if you can easily import familiar packages from Python then maybe this will help overcome the inertia required to switch, especially in industry.
    - s1t5 7y ago
      Even if Swift wins over Julia it certainly won't be because of these two points (syntax and Python interop). Julia probably has the best Python interop out of any other language and the syntax is fairly simple (although a bit ugly with the 'end' blocks and the allowed unicode characters).
      - ziotom78 7y ago
        One of the things that people in my field (physics) love when I showcase Julia is the ability to use greek letters as variables: this makes mathematical formulae so clean to read!
      - chappi42 7y ago
        'end' is subjective (I like it). The unicode characters allow to write syntax which is very close to mathematical formulae, this can help a lot with reading/understanding in some situations.
      - OkayPhysicist 7y ago
        Don't you dare talk shit about my Unicode characters. Any langauge written after Unicode's adoption has no excuse not to use them. They're wonderful. They're magical. They let you say exactly what you mean, and keep it condensed enought that you can read it at a glance. If you're in a field where symbols have existing meanings, it's asinine to make your code clunky and harder to read by not using those existing meanings.
    - mft_ 7y ago
      Looks, or is similar to use? Python is my main language, and I struggled mightily to get my head around some aspects of Swift (e.g. randomly sticking question marks in different places until it was happy). As an aside, I also found the API incredibly verbose, the documentation poor, and Xcode to be a bad IDE. In contrast, I enjoyed Julia - while the documentation isn't great either, after a single afternoon of learning (with plenty of Googling) I was able to port code over from Python and have it run perfectly.
JustFinishedBSG 7y ago
Because Chris Lattner wanted to use Swift. Doesn't matter anyway, it's a google project, it will be abandoned before 2022
- 72deluxe 7y ago
  Sad but true. The adoption of a rapidly-changing language (Swift) mostly under the control of a company that doesn't mind breaking things from one year to the next (Apple) being used by a company that abandons projects monthly (Google) sounds like a waste of time and effort for developer investment.
- TheOtherHobbes 7y ago
  Chris will leave 2023/2024 and then a related project will iterate to Julia around 2025. If people start working with Julia now, they'll be able to pick up that wave when it happens. (I honestly can't tell if this is snark or a realistic and plausible prediction.)
  - p_l 7y ago
    He already left...
spinningslate 7y ago
As an outsider, I was disappointed they didn't pick Julia too. It seems very thoughtfully put together, has a good community and some high quality libs. And it's fast. As the article points out, Google did consider it. IIRC it came down to Julia and Swift in the end. And, given Chris Lattner was leading the effort, there was only really going to be one answer. There's clearly some merit to that: they were expecting to make changes in the compiler (again, if I remember, for e.g. optimising GPU code). If you're going to change the compiler, it's pretty compelling to opt for the language that one of your team designed. And it's not clear (to me) what the implications of commits into the Julia master tree would have been. That doesn't generate a community though. It's yet to be seen whether that will happen. It would take the level of resource that very few firms can afford to dedicate. Google is one of them: though its patchy record on committing to long term endeavours means it's definitely not a slam dunk. And Lattner leaving further detracts from that confidence. I'll be interested to see what they come up with. Still think it's a pity they didn't choose Julia. But it's not my project so I don't get to choose.
- roca 7y ago
  > If you're going to change the compiler, it's pretty compelling to opt for the language that one of your team designed. It might be compelling if that person sticks around. But given Chris Lattner already moved on, it doesn't seem all that compelling after all.
- ksec 7y ago
  >I'll be interested to see what they come up with. Still think it's a pity they didn't choose Julia. But it's not my project so I don't get to choose. I still think it is not too late to admit that was a mistake, and change course to Julia. Especially Chris Lattner has moved on.
pjmlp 7y ago
1 - Chris Lattner was there 2 - S4TF is only usable in Google Cloud notebooks

Does anyone here use Swift for Deep Learning work ? I'm yet to see any real advantage, and even if there were or were to be added, I think Julia would almost always be a better choice here.

gaogao 7y ago

“Why didn’t they choose Julia"

gaogao 7y ago
I'm quoting the article on how the first HN comment will be this.
threw4234324 7y ago
Well, why though ? Julia has plenty of numerical libraries going for it, and all of them can be used in a differentiable manner (see, Neural ODE etc.). While Julia was built by a mix of people doing Numerical Linear Algebra and PLs (MIT, UCSD..), Swift seems to be entirely the output of the latter group.
- choeger 7y ago
  Maybe because swift is statically typed, whereas Julia is not?
  - pjmlp 7y ago
    Julia supports type annotations though.
    - choeger 7y ago
      So does python. That does not make it a typed language.
      - ddragon 7y ago
        Julia is compiled unlike Python though (at least by default), the moment you call Zygote it will have to run through the entire program you want to differentiate and fail immediately if any type does not match (without the need of any type annotation besides in the method arguments for the multiple dispatch and structs, as Julia has type inference). As some people say, Julia will fail "Just Ahead of Time", and if you're experimenting with live code in the REPL or Jupyter (the recommended workflow) it could very well be just as good. That said, for machine learning a static type system (as they are now) is definitely not as much of a boom as in most other areas. Most of it involves tensor operations, and embedding things like it's shape in the generics will often lead to an exponential explosion of possible monomorphizations that the compiler will be forced to create. Even JIT languages like Julia will have trouble even though it only needs to compile when it's used (for example StaticArrays will get to a point when the compiler will take so much time that it's not worth it anymore). And even then I feel like most of the issues that actually take time are deeper than stuff like shapes and not naming dimensions, like numerical instabilities, hyperparameter tuning, architecture search and other stuff that gets more benefit of a language allowing quicker exploration.
        choeger 7y ago
        Python is also compiled to bytecode in the mainstream implementation. But that is an implementation detail and has nothing to do with the (lack of a) type system.
        ddragon 7y ago
        Julia is also "compiled" down to it's (untyped) IR before even starting (when all macros are expanded), but since it's a dynamic language like Python, it can't know types at compile-time, so this step can only know parser errors. It could only type check at this step if the type was in the definition and not in the variable inside (if it was static). And both Julia and Python are also strongly typed (there isn't really a language that lacks a type system). The difference is that whenever a function in called in Julia, based on the type of the arguments the compiler can infer every type of every variable and arguments of every subsequent function, immediately compiling them down to machine code (unlike Python there is no such thing as a Julia Virtual Machine or even a Julia Interpreter outside of the debugger). Whenever you enter a function in Julia it becomes a static program, with all the properties of a static language (for example, you can't redefine types, you can define functions but the program can't see them since they are not part of the compiled code, you can't import libraries, and all types were already checked before running). That's why Julia is fast, and the language was entirely designed for working this way (there are lots of things you can do in Python that you can't in Julia to make this work).
- random_moonwalk 7y ago
  I wonder how important more superficial elements will be in terms of future adoption. As a Python programmer Swift looks much more familiar and if you can easily import familiar packages from Python then maybe this will help overcome the inertia required to switch, especially in industry.
  - s1t5 7y ago
    Even if Swift wins over Julia it certainly won't be because of these two points (syntax and Python interop). Julia probably has the best Python interop out of any other language and the syntax is fairly simple (although a bit ugly with the 'end' blocks and the allowed unicode characters).
    - ziotom78 7y ago
      One of the things that people in my field (physics) love when I showcase Julia is the ability to use greek letters as variables: this makes mathematical formulae so clean to read!
    - chappi42 7y ago
      'end' is subjective (I like it). The unicode characters allow to write syntax which is very close to mathematical formulae, this can help a lot with reading/understanding in some situations.
    - OkayPhysicist 7y ago
      Don't you dare talk shit about my Unicode characters. Any langauge written after Unicode's adoption has no excuse not to use them. They're wonderful. They're magical. They let you say exactly what you mean, and keep it condensed enought that you can read it at a glance. If you're in a field where symbols have existing meanings, it's asinine to make your code clunky and harder to read by not using those existing meanings.
  - mft_ 7y ago
    Looks, or is similar to use? Python is my main language, and I struggled mightily to get my head around some aspects of Swift (e.g. randomly sticking question marks in different places until it was happy). As an aside, I also found the API incredibly verbose, the documentation poor, and Xcode to be a bad IDE. In contrast, I enjoyed Julia - while the documentation isn't great either, after a single afternoon of learning (with plenty of Googling) I was able to port code over from Python and have it run perfectly.

JustFinishedBSG 7y ago

Because Chris Lattner wanted to use Swift. Doesn't matter anyway, it's a google project, it will be abandoned before 2022

72deluxe 7y ago
Sad but true. The adoption of a rapidly-changing language (Swift) mostly under the control of a company that doesn't mind breaking things from one year to the next (Apple) being used by a company that abandons projects monthly (Google) sounds like a waste of time and effort for developer investment.
TheOtherHobbes 7y ago
Chris will leave 2023/2024 and then a related project will iterate to Julia around 2025. If people start working with Julia now, they'll be able to pick up that wave when it happens. (I honestly can't tell if this is snark or a realistic and plausible prediction.)
- p_l 7y ago
  He already left...

spinningslate 7y ago

As an outsider, I was disappointed they didn't pick Julia too. It seems very thoughtfully put together, has a good community and some high quality libs. And it's fast. As the article points out, Google did consider it. IIRC it came down to Julia and Swift in the end. And, given Chris Lattner was leading the effort, there was only really going to be one answer. There's clearly some merit to that: they were expecting to make changes in the compiler (again, if I remember, for e.g. optimising GPU code). If you're going to change the compiler, it's pretty compelling to opt for the language that one of your team designed. And it's not clear (to me) what the implications of commits into the Julia master tree would have been. That doesn't generate a community though. It's yet to be seen whether that will happen. It would take the level of resource that very few firms can afford to dedicate. Google is one of them: though its patchy record on committing to long term endeavours means it's definitely not a slam dunk. And Lattner leaving further detracts from that confidence. I'll be interested to see what they come up with. Still think it's a pity they didn't choose Julia. But it's not my project so I don't get to choose.

roca 7y ago
> If you're going to change the compiler, it's pretty compelling to opt for the language that one of your team designed. It might be compelling if that person sticks around. But given Chris Lattner already moved on, it doesn't seem all that compelling after all.
ksec 7y ago
>I'll be interested to see what they come up with. Still think it's a pity they didn't choose Julia. But it's not my project so I don't get to choose. I still think it is not too late to admit that was a mistake, and change course to Julia. Especially Chris Lattner has moved on.

pjmlp 7y ago

1 - Chris Lattner was there 2 - S4TF is only usable in Google Cloud notebooks