5 ms·

> Companies such as Google and Facebook should have supported Julia for machine learning, in my opinion. Given that ML is becoming so important and Julia has e

> Companies such as Google and Facebook should have supported Julia for machine learning, in my opinion. Given that ML is becoming so important and Julia has ergonomic handling of numerical code, why don't they explicitly support it? Is it inertia?

Joel_Mckay 3y ago
You didn't look hard enough. ML on CUDA is very much part of Julia already, and Meta seems into that lib as well. =)
- jjoonathan 3y ago
  Is it good? PyCUDA was so bad (as of 2018) that after 3 major bugfixes from me and no end in sight it still almost sank a major project until I gave up on it and went with native CUDA.
  - empyrrhicist 3y ago
    Anecdotally, the Julia approach certainly seems... nicer via e.g. KernelAbstractions etc. I don't know if the performance/flexibility is quite there vs. native CUDA, but going from zero to GPU kernel programming in Julia is as close to painless as I've ever seen, especially given how modular everything is (for example, you can use OffsetArrays.jl directly in GPU kernel code).
  - Joel_Mckay 3y ago
    "Is it good?" Compared to the CUDA dumpster fire at a cat food factory, it is often trivial and nearly transparent syntax for the users familiar with ML. Really depends on the use-case =) The conventional options are fairly well documented: https://sciml.ai/ https://sciml.ai/ https://fluxml.ai/Flux.jl/stable/gpu/ https://fluxml.ai/Flux.jl/stable/gpu/ https://github.com/SciML/DiffEqFlux.jl https://github.com/SciML/DiffEqFlux.jl https://github.com/alan-turing-institute/MLJ.jl https://github.com/alan-turing-institute/MLJ.jl
bmitc 3y ago
They have a 100,000 Python, C++, and Java devs. My assumption is they don't care, i.e., inertia like you said. The big mistake was Microsoft not having .NET cross-platform from the beginning and not pitching F# as a Python alternative.
- soulbadguy 3y ago
  > F# as a Python alternative To play devil advocate, i think when F# was out (and still being developed), the Zeitgeist was really enamored with the idea of dynamic language, and the envisioned productivity gain from them. So F# was really too early to the dance. Since then, things have change quite a bit with most major dynamic language adding a proto-type system. Right now, i think we understand better what made language such as python and ruby so popular and are able to design better statically typed version with better tooling around them ( rust, zig, julia etc...)
chriskanan 3y ago
The developers of Pytorch put out an article about this a few years ago and they admitted Julia would make sense from a pure ML perspective but the weight of the Python ecosystem was too critical. I wonder if GPT-4 could be used to efficiently start porting more stuff over to Julia. I feel like Python is really a poor fit for deep learning. Julia is much more pleasant.
- disgruntledphd2 3y ago
  The only reason PyTorch exists is that Yann LeCunn couldn't convince people at Facebook to use the original Lua implementation. So yeah, I'd be surprised if FB ever moved away from Python.
roflmaostc 3y ago
From my point, there is still too much change around automatic differentiation libraries. There is Zygote.jl, which is used in Flux.jl. However, it's more in maintenance mode. At some point, Diffractor.jl was hyped but it didn't take off yet. And then there is Enzyme.jl which people hype now. But for me as a user, it's not clear what I should really do to make my code well differentiable for those libraries. If you stick with torch, jax or tensorflow, everything seems to work better regarding AD. [0]: https://github.com/FluxML/Flux.jl https://github.com/FluxML/Flux.jl [1]: https://github.com/FluxML/Zygote.jl https://github.com/FluxML/Zygote.jl [2]: https://github.com/JuliaDiff/Diffractor.jl https://github.com/JuliaDiff/Diffractor.jl [3]: https://github.com/EnzymeAD/Enzyme.jl https://github.com/EnzymeAD/Enzyme.jl
alechammond 3y ago
Note that Meta recently added a Julia ruleset to its buck2 prelude set. An important step toward working with Julia on prod environments. https://github.com/facebook/buck2/tree/main/prelude/julia https://github.com/facebook/buck2/tree/main/prelude/julia
_Wintermute 3y ago
Instead we got Tensorflow for ... Swift, which nobody asked for.
- raccoonDivider 3y ago
  Apple probably asked and paid for it. Isn't there potential in accelerating LLMs and transformers on smartphones? Most of what they are good at fits the smartphone use case well.
  - pjmlp 3y ago
    Not at all, it was more related to well known Swift folks at Google.
raccoonDivider 3y ago
I tried using the latest stable Flux.jl with the latest stable Julia interpreter about a year ago, and most of it didn't work. Some of the simplest model usage examples led to a traceback with dozens of frames, ending in an incomprehensible internal error that I couldn't possibly understand as a beginner. It seems to me that Julia is still moving too fast for its own good. At some point, something needs to stabilize for it to be worth adopting.

Joel_Mckay 3y ago

You didn't look hard enough. ML on CUDA is very much part of Julia already, and Meta seems into that lib as well. =)

jjoonathan 3y ago
Is it good? PyCUDA was so bad (as of 2018) that after 3 major bugfixes from me and no end in sight it still almost sank a major project until I gave up on it and went with native CUDA.
- empyrrhicist 3y ago
  Anecdotally, the Julia approach certainly seems... nicer via e.g. KernelAbstractions etc. I don't know if the performance/flexibility is quite there vs. native CUDA, but going from zero to GPU kernel programming in Julia is as close to painless as I've ever seen, especially given how modular everything is (for example, you can use OffsetArrays.jl directly in GPU kernel code).
- Joel_Mckay 3y ago
  "Is it good?" Compared to the CUDA dumpster fire at a cat food factory, it is often trivial and nearly transparent syntax for the users familiar with ML. Really depends on the use-case =) The conventional options are fairly well documented: https://sciml.ai/ https://sciml.ai/ https://fluxml.ai/Flux.jl/stable/gpu/ https://fluxml.ai/Flux.jl/stable/gpu/ https://github.com/SciML/DiffEqFlux.jl https://github.com/SciML/DiffEqFlux.jl https://github.com/alan-turing-institute/MLJ.jl https://github.com/alan-turing-institute/MLJ.jl

bmitc 3y ago

They have a 100,000 Python, C++, and Java devs. My assumption is they don't care, i.e., inertia like you said. The big mistake was Microsoft not having .NET cross-platform from the beginning and not pitching F# as a Python alternative.

soulbadguy 3y ago
> F# as a Python alternative To play devil advocate, i think when F# was out (and still being developed), the Zeitgeist was really enamored with the idea of dynamic language, and the envisioned productivity gain from them. So F# was really too early to the dance. Since then, things have change quite a bit with most major dynamic language adding a proto-type system. Right now, i think we understand better what made language such as python and ruby so popular and are able to design better statically typed version with better tooling around them ( rust, zig, julia etc...)

chriskanan 3y ago

The developers of Pytorch put out an article about this a few years ago and they admitted Julia would make sense from a pure ML perspective but the weight of the Python ecosystem was too critical. I wonder if GPT-4 could be used to efficiently start porting more stuff over to Julia. I feel like Python is really a poor fit for deep learning. Julia is much more pleasant.

disgruntledphd2 3y ago
The only reason PyTorch exists is that Yann LeCunn couldn't convince people at Facebook to use the original Lua implementation. So yeah, I'd be surprised if FB ever moved away from Python.

roflmaostc 3y ago

From my point, there is still too much change around automatic differentiation libraries. There is Zygote.jl, which is used in Flux.jl. However, it's more in maintenance mode. At some point, Diffractor.jl was hyped but it didn't take off yet. And then there is Enzyme.jl which people hype now. But for me as a user, it's not clear what I should really do to make my code well differentiable for those libraries. If you stick with torch, jax or tensorflow, everything seems to work better regarding AD. [0]: https://github.com/FluxML/Flux.jl https://github.com/FluxML/Flux.jl [1]: https://github.com/FluxML/Zygote.jl https://github.com/FluxML/Zygote.jl [2]: https://github.com/JuliaDiff/Diffractor.jl https://github.com/JuliaDiff/Diffractor.jl [3]: https://github.com/EnzymeAD/Enzyme.jl https://github.com/EnzymeAD/Enzyme.jl

alechammond 3y ago

Note that Meta recently added a Julia ruleset to its buck2 prelude set. An important step toward working with Julia on prod environments. https://github.com/facebook/buck2/tree/main/prelude/julia https://github.com/facebook/buck2/tree/main/prelude/julia

_Wintermute 3y ago

Instead we got Tensorflow for ... Swift, which nobody asked for.

raccoonDivider 3y ago
Apple probably asked and paid for it. Isn't there potential in accelerating LLMs and transformers on smartphones? Most of what they are good at fits the smartphone use case well.
- pjmlp 3y ago
  Not at all, it was more related to well known Swift folks at Google.

raccoonDivider 3y ago

I tried using the latest stable Flux.jl with the latest stable Julia interpreter about a year ago, and most of it didn't work. Some of the simplest model usage examples led to a traceback with dozens of frames, ending in an incomprehensible internal error that I couldn't possibly understand as a beginner. It seems to me that Julia is still moving too fast for its own good. At some point, something needs to stabilize for it to be worth adopting.