6 ms·

Micrograd.jl

huqedato 2y ago
Julia is a splendid, high performance language. And the most overlooked. Such a huge pity and shame that the entire current AI ecosystem is build on Python/Pytorch. Python - not a real programming language, let alone is interpreted... such a huge loss of performance besides Julia.
- FranzFerdiNaN 2y ago
  > not a real programming language Really? Why do you feel the need to say this? Not liking Python, sure, but this kind of comments is just stupid elitism. What's next, the only REAL programmers are the ones that make their own punch cards?
  - chasd00 2y ago
    Theyre just trolling for a reaction. It is indeed a ridiculous statement.
- brrrrrm 2y ago
  It’s all about the kernels tho. The language doesn’t matter much. For the things that matter, everything is a dispatch to some cuda graph I’m not really a fan of this convergence but the old school imperative CPU way of thinking about things is dead in this space
  - adgjlsfhk1 2y ago
    One of the really nice things about Julia for GPU programming is that you can write your own kernels. CUDA.jl isn't just C kernels. This is why (for example) DiffEQGPU.jl is able to be a lot faster than the other GPU based ODE solvers (see https://arxiv.org/abs/2304.06835 https://arxiv.org/abs/2304.06835 for details).
    - almostgotcaught 2y ago
      > One of the really nice things about Julia for GPU programming is that you can write your own kernels. CUDA.jl isn't just C kernels. Do y'all like understand that this isn't special to Julia and it isn't special to any language and it's 100% due to the fact that nvcc has an open-source counterpart in LLVM? Like literally there are now dozens of languages that be used to author CUDA kernels because they all just target nvcc's flavor of ll. Eg you can do it in Python like 13 different ways - numba, taichi, etc all do it. You can even use the MLIR python bindings to directly emit nvgpu dialect and then lower to target specific llvmir.
- deleted 2y ago
  [deleted]
- atoav 2y ago
  Python is not a real programing language? That must come as a shocking revalation to the many thousand people running it successfully in production. /s As someone who programs C/C++/Python/Rust/JS you had me curious in the first half of the post. But that comment makes me wonder about the quality of the rest of what you're saying.
- Y_Y 2y ago
  I recognize the use of "not a real language" as traditional hyperbole[0]. I have my own gripes with python, even though it pays the bills, but this is just going to set off a load of people and is probably bad for discussion quality. Ironically it's very hard to write actual low-level parallel code (like CUDA) through python, there's really no choice but to call out to Fortran and C libraries for the likes of pytorch. [0] https://en.wikipedia.org/wiki/Real_Programmers_Don't_Use_Pascal https://en.wikipedia.org/wiki/Real_Programmers_Don't_Use_Pas...
- xaellison 2y ago
  as a major julia enthusiast I gotta say this is not how you get people to check it out buddy
xiaodai 2y ago
I kinda gave up on Julia for deep learning since it’s so buggy. I am using PyTorch now. Not great but at least it works!
- enkursigilo 2y ago
  Can you elaborate a bit more?
- moelf 2y ago
  what is so buggy, Julia the language or the deep learning libraries in Julia? in either case it would be good to have some examples.
  - xiaodai 2y ago
    the deep learning libraries. can't figure out why one of my gradient didn't work so i switch implementation to pytorch and it worked perfectly fine.
  - catgary 2y ago
    This is an old-ish article about Julia, but from what I can tell the core issues with autograd were never fixed: https://kidger.site/thoughts/jax-vs-julia/ https://kidger.site/thoughts/jax-vs-julia/
  - currymj 2y ago
    julia the language is really good. but a lot of core infrastructure julia libraries are maintained by some overworked grad student. sometimes that grad student is a brilliantly productive programmer + the libraries reach escape velocity and build a community, and then you get areas where Julia is state of the art like in differential equation solving, or generally other areas of "classical" scientific computing. in other cases the grad student is merely a very good programmer, and they just sort of float along being "almost but not quite there" for a long time, maybe abandoned depending on the maintainer's career path. the latter case is pretty common in the machine learning ecosystem. a lot of people get excited about using a fast language for ML, see that Julia can do what they want in a really cool way, and then run into some breaking problem or missing feature ("will be fixed eventually") after investing some time in a project.
xyproto 2y ago
Why did Julia select a package naming convention that makes every project name look like a filename?
- eigenspace 2y ago
  It makes a julia package name very recognizable and easily searchable. It's actually something I really miss when I'm trying to look up packages in other languages.
- NeuroCoder 2y ago
  I thought that was weird too but then I realized it was on of the most useful tools for searching stuff online and getting exactly what I wanted.
- infogulch 2y ago
  Imagine searching for Plots.jl, Symbolics.jl, CUDA.jl if they didn't have the ".jl". I wish more package ecosystems used a convention like this.
- stackghost 2y ago
  Presumably for similar reasons to JavaScript, with names like Next.js, etc.
anon389r58r58 2y ago
Almost feels like a fallacy of Julia at this point, on the one hand Julia really needs a stable, high-performance AD-engine, but on the other hand it seems to be fairly easy to get a minimal AD-package off the ground. And so the perennial cycle continues and another Julia AD-package emerges, and ignores all/most previous work in order to claim novelty. Without a claim for a complete list: ReverseDiff.jl, ForwardDiff.jl, Zygote.jl, Enzyme.jl, Tangent.jl, Diffractor.jl, and many more whose name has disappeared in the short history of Julia...
- 0cf8612b2e1e 2y ago
  I do not think this is meant to be a “real” library, but a didactic exercise inspired by Andrej Kaparthy‘s Python implementation.
  - nextos 2y ago
    This is a didactic exercise. Julia is fantastic, but lacks funding to develop a differentiable programming ecosystem that can compete with Torch or Jax. These two have corporate juggernauts backing them. Still, it is quite remarkable how far Julia has got with few resources. Having an alternative to Python would benefit the ML ecosystem, which is too much of a monoculture right now. Julia has some really interesting statistics, probabilistic programming and physics-informed ML packages.
    - anon389r58r58 2y ago
      I think you are asking an ill-posed question in parts. Julia has a lot of great things, and needs to continue evolving to find an even better fit amongst the many programming languages available today and sustain itself long-term. Emulating Python's ML ecosystem is not going to be a viable strategy. The investment into the Python-based standard is just too large. What I could see happening though is that the continuous evolution of the ML ecosystem will further abstract components of the software stack down to an MLIR/LLVM abstraction level at which point something like Julia could also use this componentry. Sort of a continuum of components, where the "frontend" language and associated programming style is the remaining choice for the user to make.
thetwentyone 2y ago
Odd that the author excluded ForwardDiff.jl and Zygote.jl, both of which get a lot of mileage in the Julia AD world. Nonetheless, awesome tutorial and great to see more Julia content like this!
fithisux 2y ago
Another testament to the awesomeness of Julia

Julia is a splendid, high performance language. And the most overlooked. Such a huge pity and shame that the entire current AI ecosystem is build on Python/Pytorch. Python - not a real programming language, let alone is interpreted... such a huge loss of performance besides Julia.

FranzFerdiNaN 2y ago
> not a real programming language Really? Why do you feel the need to say this? Not liking Python, sure, but this kind of comments is just stupid elitism. What's next, the only REAL programmers are the ones that make their own punch cards?
- chasd00 2y ago
  Theyre just trolling for a reaction. It is indeed a ridiculous statement.
brrrrrm 2y ago
It’s all about the kernels tho. The language doesn’t matter much. For the things that matter, everything is a dispatch to some cuda graph I’m not really a fan of this convergence but the old school imperative CPU way of thinking about things is dead in this space
- adgjlsfhk1 2y ago
  One of the really nice things about Julia for GPU programming is that you can write your own kernels. CUDA.jl isn't just C kernels. This is why (for example) DiffEQGPU.jl is able to be a lot faster than the other GPU based ODE solvers (see https://arxiv.org/abs/2304.06835 https://arxiv.org/abs/2304.06835 for details).
  - almostgotcaught 2y ago
    > One of the really nice things about Julia for GPU programming is that you can write your own kernels. CUDA.jl isn't just C kernels. Do y'all like understand that this isn't special to Julia and it isn't special to any language and it's 100% due to the fact that nvcc has an open-source counterpart in LLVM? Like literally there are now dozens of languages that be used to author CUDA kernels because they all just target nvcc's flavor of ll. Eg you can do it in Python like 13 different ways - numba, taichi, etc all do it. You can even use the MLIR python bindings to directly emit nvgpu dialect and then lower to target specific llvmir.
deleted 2y ago
[deleted]
atoav 2y ago
Python is not a real programing language? That must come as a shocking revalation to the many thousand people running it successfully in production. /s As someone who programs C/C++/Python/Rust/JS you had me curious in the first half of the post. But that comment makes me wonder about the quality of the rest of what you're saying.
Y_Y 2y ago
I recognize the use of "not a real language" as traditional hyperbole[0]. I have my own gripes with python, even though it pays the bills, but this is just going to set off a load of people and is probably bad for discussion quality. Ironically it's very hard to write actual low-level parallel code (like CUDA) through python, there's really no choice but to call out to Fortran and C libraries for the likes of pytorch. [0] https://en.wikipedia.org/wiki/Real_Programmers_Don't_Use_Pascal https://en.wikipedia.org/wiki/Real_Programmers_Don't_Use_Pas...
xaellison 2y ago
as a major julia enthusiast I gotta say this is not how you get people to check it out buddy

xiaodai 2y ago

I kinda gave up on Julia for deep learning since it’s so buggy. I am using PyTorch now. Not great but at least it works!

enkursigilo 2y ago
Can you elaborate a bit more?
moelf 2y ago
what is so buggy, Julia the language or the deep learning libraries in Julia? in either case it would be good to have some examples.
- xiaodai 2y ago
  the deep learning libraries. can't figure out why one of my gradient didn't work so i switch implementation to pytorch and it worked perfectly fine.
- catgary 2y ago
  This is an old-ish article about Julia, but from what I can tell the core issues with autograd were never fixed: https://kidger.site/thoughts/jax-vs-julia/ https://kidger.site/thoughts/jax-vs-julia/
- currymj 2y ago
  julia the language is really good. but a lot of core infrastructure julia libraries are maintained by some overworked grad student. sometimes that grad student is a brilliantly productive programmer + the libraries reach escape velocity and build a community, and then you get areas where Julia is state of the art like in differential equation solving, or generally other areas of "classical" scientific computing. in other cases the grad student is merely a very good programmer, and they just sort of float along being "almost but not quite there" for a long time, maybe abandoned depending on the maintainer's career path. the latter case is pretty common in the machine learning ecosystem. a lot of people get excited about using a fast language for ML, see that Julia can do what they want in a really cool way, and then run into some breaking problem or missing feature ("will be fixed eventually") after investing some time in a project.

xyproto 2y ago

Why did Julia select a package naming convention that makes every project name look like a filename?

eigenspace 2y ago
It makes a julia package name very recognizable and easily searchable. It's actually something I really miss when I'm trying to look up packages in other languages.
NeuroCoder 2y ago
I thought that was weird too but then I realized it was on of the most useful tools for searching stuff online and getting exactly what I wanted.
infogulch 2y ago
Imagine searching for Plots.jl, Symbolics.jl, CUDA.jl if they didn't have the ".jl". I wish more package ecosystems used a convention like this.
stackghost 2y ago
Presumably for similar reasons to JavaScript, with names like Next.js, etc.

anon389r58r58 2y ago

Almost feels like a fallacy of Julia at this point, on the one hand Julia really needs a stable, high-performance AD-engine, but on the other hand it seems to be fairly easy to get a minimal AD-package off the ground. And so the perennial cycle continues and another Julia AD-package emerges, and ignores all/most previous work in order to claim novelty. Without a claim for a complete list: ReverseDiff.jl, ForwardDiff.jl, Zygote.jl, Enzyme.jl, Tangent.jl, Diffractor.jl, and many more whose name has disappeared in the short history of Julia...

0cf8612b2e1e 2y ago
I do not think this is meant to be a “real” library, but a didactic exercise inspired by Andrej Kaparthy‘s Python implementation.
- nextos 2y ago
  This is a didactic exercise. Julia is fantastic, but lacks funding to develop a differentiable programming ecosystem that can compete with Torch or Jax. These two have corporate juggernauts backing them. Still, it is quite remarkable how far Julia has got with few resources. Having an alternative to Python would benefit the ML ecosystem, which is too much of a monoculture right now. Julia has some really interesting statistics, probabilistic programming and physics-informed ML packages.
  - anon389r58r58 2y ago
    I think you are asking an ill-posed question in parts. Julia has a lot of great things, and needs to continue evolving to find an even better fit amongst the many programming languages available today and sustain itself long-term. Emulating Python's ML ecosystem is not going to be a viable strategy. The investment into the Python-based standard is just too large. What I could see happening though is that the continuous evolution of the ML ecosystem will further abstract components of the software stack down to an MLIR/LLVM abstraction level at which point something like Julia could also use this componentry. Sort of a continuum of components, where the "frontend" language and associated programming style is the remaining choice for the user to make.

thetwentyone 2y ago

Odd that the author excluded ForwardDiff.jl and Zygote.jl, both of which get a lot of mileage in the Julia AD world. Nonetheless, awesome tutorial and great to see more Julia content like this!

fithisux 2y ago

Another testament to the awesomeness of Julia