10 ms·

State of Machine Learning in Julia

256 points by agnosticmantis 5y ago

baggiponte 5y ago
That’s terribly interesting, given that this popped up too very recently: https://www.stochasticlifestyle.com/engineering-trade-offs-in-automatic-differentiation-from-tensorflow-and-pytorch-to-jax-and-julia/ https://www.stochasticlifestyle.com/engineering-trade-offs-i... However, while they speak of ML they refer only to deep learning, and I was wondering where Julia stands in terms of scikit-learn equivalents and if they ever took off.
- sanxiyn 5y ago
  There is ScikitLearn.jl if you want a literal port, and MLJ.jl if you want a spiritual one.
  - sundarurfriend 5y ago
    Do you know how much MLJ.jl is used out in the wild and how well it is maintained? I don't see it coming up much in discussions in, for eg., the Julia Zulip or Discourse forums, not nearly as much as Flux for example. It may just be that it's outside my visibility bubble, but the only context I see it mentioned is similar to this where someone mentions it as an available option.
    - thetwentyone 5y ago
      I've used MLJ for some small projects at work without issue.
    - adgjlsfhk1 5y ago
      I think a lot of the answer is that MLJ is pretty stable, and works so there's a lot less discussion of it. Discussion indicates a combination of interest and complexity.
      - sundarurfriend 5y ago
        Thanks, that's really good to know.
dash2 5y ago
I get the sense that the Julia guys are doing very well on solving interesting, hard technical problems like AD. But maybe less well at “dull” problems like good documentation and API design. That might bias them towards being a language for research rather than for production. Still very interested to see how things pan out.
- mountainriver 5y ago
  Yes and yes. Julia has a pretty bad developer experience. My biggest concern is that when you bring this up to the Julia crowd they just argue that it’s actually great and you just don’t get it. The Julia community feels a lot like the Scala community to me. Lots of proving why the theories are right, but in production it’s a mess
  - pjmlp 5y ago
    Mostly because in regards to where they are coming from the pain points lie elsewhere.
- isoprophlex 5y ago
  Exactly. To me they have a rather big blind spot when it comes to reaching out to users with docs, and understanding their user needs, exactly because they focus so much on these hard challenges. You can see it in pt 2: "where is the Julia ML ecosystem currently inferior?". They identify some gap with using cuda optimized kernels, where Julia doesn't make use of built in cuda magic; a solution is proposed where users might manually override compiler behavior (if I understand correctly). However, no mention at all about the gap in dev experience. Some examples: - TERRIFIC efforts of various groups in porting ml models from research to easy-to-use torch code - the super ergonomic data wrangling tools in torch. Practically speaking, ml ecosystems need quality interop: do i get my data in and out of the ml framework in a sensible way - the ease of getting started building your first toy ML model in torch because the docs and the API are so great Getting the ball rolling, adoption-wise, means overcoming the cost of switching to something new. And I don't see that happening soon because python+torch is just so comfy.
- oxff 5y ago
faisal_ksa 5y ago
I'm really rooting for Julia to improve its ML ecosystem. Julia is a JIT-compiled language that has the potential to become the all-encompassing solution for AI/ML/Data Science. I think their motto was "Walk like Python, Run like C." This could help to implement all their libraries in Julia instead of what is happening in Python (e.g., PyTorch is written in C++). I want to be clear; Python will be the go-to language for AI/ML for the foreseeable future. And It's a great language, but Julia is a breath of fresh air.
- enriquto 5y ago
  > I think their motto was "Walk like Python, Run like C." I was also seduced by the concept of "octave, but with fast loops". This is really all that was needed. But I'm a bit disenchanted by the "advanced" additional features of Julia that add (unnecessary, in my view) complexity to the language. All that multiple-dispatch stuff, all those arbitrary types. I would love a compilation option that forced everything to be of the same type (a multi-dimensional array of numbers), and then it could really run like C, including compilation time.
  - jstx1 5y ago
    Multiple dispatch is the main feature that makes everything else possible, not some additional feature that spoils things.
    - enriquto 5y ago
      I disagree with the word "everything" in your sentence. If all you want to do is numeric computations, you just want a sort of "modern fortran" that compiles and runs fast. No need for multiple dispatch here. Julia almost is this language, if it wasn't for the multiple dispatch thing whose only effect is to slow the compilation (when all you have is the same numeric type).
      - deleted 5y ago
        [deleted]
viciousplant 5y ago
That "Unreliable gradients" part is what I worry about most, why would I use a product when it just "returning incorrect gradients"?
- sethaxen 5y ago
  I'd be interested to know whether that occurred pre- or post- Zygote.jl starting to use ChainRules.jl internally. Zygote's own rules used to be tested with a half-baked finite differences scheme, and it was easy to miss bugs. All rules in ChainRules are tested using the testing helpers in ChainRulesTestUtils.jl, which are very thorough. Then, some bugs are not actually bugs. For example, the gradient of some function wrt to elements of a symmetric matrix is and should be different depending on whether the matrix is structurally symmetric or not. Or, when a function is not differentiable in the calculus sense, different ADs often have different conventions for what they will return. These cases come up all the time.
  - wnoise 5y ago
    > For example, the gradient of some function wrt to elements of a symmetric matrix is and should be different depending on whether the matrix is structurally symmetric Ugh. Yes and no, but mostly no. A gradient is a collection of partial derivatives. The normal meaning of partial derivative with respect to symmetric matrix elements does not actually exist, because you can't vary them independently. This is actually a generic problem for any overparameterized constrained system. People still want to be able to use the standard tools of calculus though. You can reasonably do things like use Lagrange multipliers to do constrained optimizations. Or you can express a symmetric matrix in terms of only the n(n+1)/2 variables, treat them as a vector and get the gradient over that. (This is often what is done without actually describing the underlying method. This is the so-called "symmetric gradient"). But it's generally not what you actually want, even though it can lead to the same stationary points. The best you can do is the unconstrained gradient restricted to the manifold of symmetric matrices (or whatever constrained object you are interested in). But even this is a (subtly) different thing than the gradient in the ambient space, even though it works in as large a set of cases as possible. Happily for symmetric differentials applied to symmetric matrices, the naive formulas of symmetrizing the gradient just work. See, e.g. https://arxiv.org/abs/1911.06491 https://arxiv.org/abs/1911.06491 for some tracking down of the history and cogent analysis of what went wrong and https://saturdaygenfo.github.io/posts/symmetric-gradients/ https://saturdaygenfo.github.io/posts/symmetric-gradients/ for why it matters in practice (e.g. wrong and changing direction of descent for gradient descent, though still "downwards").
usgroup 5y ago
The responses confuse "machine learning" with neural networks so far as I can see. I do lots of statsy things: good luck with that outside of R.
- jiggunjer 5y ago
  Even professors doing trendy "AI" research conflate the two terms.
  - dagw 5y ago
    Even professors doing trendy "AI" research conflate the two terms. Only because doing "AI" gets you bigger grants.
    - kgwgk 5y ago
      Tibshirani's Glossary Machine learning Statistics network, graphs model weights parameters learning fitting generalization test set performance supervised learning regression/classiﬁcation unsupervised learning density estimation, clustering large grant = $1,000,000 large grant = $50,000 nice place to have a meeting: nice place to have a meeting: Snowbird, Utah, French Alps Las Vegas in August
- deleted 5y ago
  [deleted]
- fault1 5y ago
  turing, mlj, etc are quite good. https://turing.ml/stable/ https://turing.ml/stable/ https://alan-turing-institute.github.io/MLJ.jl/dev/about_mlj/ https://alan-turing-institute.github.io/MLJ.jl/dev/about_mlj...
  - usgroup 5y ago
    Until you've used: https://r-nimble.org/ https://r-nimble.org/
savant_penguin 5y ago
Having played around with Julia recently there are two main reasons I would not recommend using it for ML: 1-Plotting takes too long. Forget fast iterations. You'll be left waiting for non trivial amounts of time to your first plot. The latency between tests is simply not good enough. 2-Libraries are just not mature. Plotting anything not too standard yields broken graphs and you have to look for a billion of backends to find one that does not break (hello twinx). Matplotlib is slow for some graphs but more often than not you get the right visualization (except for that suptitle being cut out of the graph bug that will never be fixed). The requests library is not robust enough. This one is quite possibility my fault. I have received "bad headers" errors for some websites, but doing the exact same "get"with python works just fine. The impression that Libraries die all the time. Could be just an impression, but it is quite common to find top stackoverflow answers that point you to dead libraries (with the common "use this other library instead" hint) I like Julia speed and well done memory management (especially from what I saw from Dataframe.jl high memory benchmarks). But right now I'd not use it in a critical environment. (And I also suck at reading and understanding the error messages thus far, but it's likely a lack of experience from my side)
- whlr 5y ago
  It’s worth noting that PyPlot.jl is a nearly seamless way to use matplotlib from Julia. JIT compilation of the middle layer does mean that it suffers from time to first plot problems though.
- cs702 5y ago
  As a longtime user of Python for ML/AI, I love Julia... but agree that most people will find Julia challenging to use for everyday data munging, or for developing production code, say, in a business environment. The Python ecosystem (people, resources, libraries, frameworks, etc.) is superior for most common, plain-vanilla use cases. However, if you are developing compute-intensive scientific software, I would recommend switching to Julia as soon as possible. IMHO, Julia is superior to the likes of Python, Fortran, C++, and C for those use cases. -- PS. By the way, you can fix matplotlib's suptitle issue by calling this Figure method after the fact: fig.tight_layout(rect=(0, 0, 1, 0.95)) # leaves 5% of top space for suptitle
melony 5y ago
When will Enzyme and Diffractor hit prod? The default right now for Flux is still Zygote.
muktuks 5y ago
Julia is great for NEW ML research. But it has serious downsides for mundane or even above average production work... Flux has historically been more of a research project then a production library. That's ok, it holds a lot of promise. But, time to first gradient can be excessive. Minutes to discover a small bug excessive. Last I checked all of the adjacent libraries around it were broken due to incredibly frequent changes to the API. Mlj doesn't impress me. There was a serious opportunity to change the way people do routine modelling using the strengths of julia. They basically made a half baked sklearn instead. Missed opportunity in my mind. In general the components for statistical libraries are exceptional. However the ecosystem falls short to do things with them in a stable way. The best libraries imo for data science work stem from the excellent db adapters, optimization library's, data frames jl, Turing jl, etc. Anything beyond that and it's usually better off just rolling your own or using another language all together... Current state of production, ie not code running a notebook is also ok at best...
- amkkma 5y ago
  What are your concerns with MLJ?
  - muktuks 5y ago
    MLJ doesn't concern me in a bad way. At it's heart it's an abstraction for ML components across many packages with some machinery on top for parameter tuning, etc. That's fine. What it's missing is ... impactful improvements on the failures of libraries like this. Docs need more examples and less pointers to packages that wrap other packages(albeit they are written well). Despite being a multi-year long effort by very big names it still feels incomplete, and if someone wasn't sold on Julia there is technically no reason not to use R or Python or Matlab. Missed opportunity is all
- UncleOxidant 5y ago
  > Julia is great for NEW ML research. But it has serious downsides for mundane or even above average production work... This was how PyTorch started out as well.
drannex 5y ago
Absolutely adore prototyping in Julia, absolutely heinous to work in production. Their memory usage is insanely high and their calc speeds are slow when you don't have the insanely high processing equipment needed. If they could just singularly focus on reducing memory consumption than the language will be likely _the defacto_ language for ML and analysis, but until then it really gets bogged down by just how much memory is needed for the most simple of functions.
rundigen12 5y ago
Paraphrase: ML in Julia is for Serious Scientists Doing Very Sciencey Things. Things you wouldn’t understand. But your desire to train deep learning models on GPUs and deploy them as apps elsewhere? ...Aw, c’mon, that’s boring. You can just use PyTorch for that. Now, who's up for a new round of Neural ODE benchmarks?!

That’s terribly interesting, given that this popped up too very recently: https://www.stochasticlifestyle.com/engineering-trade-offs-in-automatic-differentiation-from-tensorflow-and-pytorch-to-jax-and-julia/ https://www.stochasticlifestyle.com/engineering-trade-offs-i... However, while they speak of ML they refer only to deep learning, and I was wondering where Julia stands in terms of scikit-learn equivalents and if they ever took off.

sanxiyn 5y ago
There is ScikitLearn.jl if you want a literal port, and MLJ.jl if you want a spiritual one.
- sundarurfriend 5y ago
  Do you know how much MLJ.jl is used out in the wild and how well it is maintained? I don't see it coming up much in discussions in, for eg., the Julia Zulip or Discourse forums, not nearly as much as Flux for example. It may just be that it's outside my visibility bubble, but the only context I see it mentioned is similar to this where someone mentions it as an available option.
  - thetwentyone 5y ago
    I've used MLJ for some small projects at work without issue.
  - adgjlsfhk1 5y ago
    I think a lot of the answer is that MLJ is pretty stable, and works so there's a lot less discussion of it. Discussion indicates a combination of interest and complexity.
    - sundarurfriend 5y ago
      Thanks, that's really good to know.

dash2 5y ago

I get the sense that the Julia guys are doing very well on solving interesting, hard technical problems like AD. But maybe less well at “dull” problems like good documentation and API design. That might bias them towards being a language for research rather than for production. Still very interested to see how things pan out.

mountainriver 5y ago
Yes and yes. Julia has a pretty bad developer experience. My biggest concern is that when you bring this up to the Julia crowd they just argue that it’s actually great and you just don’t get it. The Julia community feels a lot like the Scala community to me. Lots of proving why the theories are right, but in production it’s a mess
- pjmlp 5y ago
  Mostly because in regards to where they are coming from the pain points lie elsewhere.
isoprophlex 5y ago
Exactly. To me they have a rather big blind spot when it comes to reaching out to users with docs, and understanding their user needs, exactly because they focus so much on these hard challenges. You can see it in pt 2: "where is the Julia ML ecosystem currently inferior?". They identify some gap with using cuda optimized kernels, where Julia doesn't make use of built in cuda magic; a solution is proposed where users might manually override compiler behavior (if I understand correctly). However, no mention at all about the gap in dev experience. Some examples: - TERRIFIC efforts of various groups in porting ml models from research to easy-to-use torch code - the super ergonomic data wrangling tools in torch. Practically speaking, ml ecosystems need quality interop: do i get my data in and out of the ml framework in a sensible way - the ease of getting started building your first toy ML model in torch because the docs and the API are so great Getting the ball rolling, adoption-wise, means overcoming the cost of switching to something new. And I don't see that happening soon because python+torch is just so comfy.
oxff 5y ago

faisal_ksa 5y ago

I'm really rooting for Julia to improve its ML ecosystem. Julia is a JIT-compiled language that has the potential to become the all-encompassing solution for AI/ML/Data Science. I think their motto was "Walk like Python, Run like C." This could help to implement all their libraries in Julia instead of what is happening in Python (e.g., PyTorch is written in C++). I want to be clear; Python will be the go-to language for AI/ML for the foreseeable future. And It's a great language, but Julia is a breath of fresh air.

enriquto 5y ago
> I think their motto was "Walk like Python, Run like C." I was also seduced by the concept of "octave, but with fast loops". This is really all that was needed. But I'm a bit disenchanted by the "advanced" additional features of Julia that add (unnecessary, in my view) complexity to the language. All that multiple-dispatch stuff, all those arbitrary types. I would love a compilation option that forced everything to be of the same type (a multi-dimensional array of numbers), and then it could really run like C, including compilation time.
- jstx1 5y ago
  Multiple dispatch is the main feature that makes everything else possible, not some additional feature that spoils things.
  - enriquto 5y ago
    I disagree with the word "everything" in your sentence. If all you want to do is numeric computations, you just want a sort of "modern fortran" that compiles and runs fast. No need for multiple dispatch here. Julia almost is this language, if it wasn't for the multiple dispatch thing whose only effect is to slow the compilation (when all you have is the same numeric type).
    - deleted 5y ago
      [deleted]

viciousplant 5y ago

That "Unreliable gradients" part is what I worry about most, why would I use a product when it just "returning incorrect gradients"?

sethaxen 5y ago
I'd be interested to know whether that occurred pre- or post- Zygote.jl starting to use ChainRules.jl internally. Zygote's own rules used to be tested with a half-baked finite differences scheme, and it was easy to miss bugs. All rules in ChainRules are tested using the testing helpers in ChainRulesTestUtils.jl, which are very thorough. Then, some bugs are not actually bugs. For example, the gradient of some function wrt to elements of a symmetric matrix is and should be different depending on whether the matrix is structurally symmetric or not. Or, when a function is not differentiable in the calculus sense, different ADs often have different conventions for what they will return. These cases come up all the time.
- wnoise 5y ago
  > For example, the gradient of some function wrt to elements of a symmetric matrix is and should be different depending on whether the matrix is structurally symmetric Ugh. Yes and no, but mostly no. A gradient is a collection of partial derivatives. The normal meaning of partial derivative with respect to symmetric matrix elements does not actually exist, because you can't vary them independently. This is actually a generic problem for any overparameterized constrained system. People still want to be able to use the standard tools of calculus though. You can reasonably do things like use Lagrange multipliers to do constrained optimizations. Or you can express a symmetric matrix in terms of only the n(n+1)/2 variables, treat them as a vector and get the gradient over that. (This is often what is done without actually describing the underlying method. This is the so-called "symmetric gradient"). But it's generally not what you actually want, even though it can lead to the same stationary points. The best you can do is the unconstrained gradient restricted to the manifold of symmetric matrices (or whatever constrained object you are interested in). But even this is a (subtly) different thing than the gradient in the ambient space, even though it works in as large a set of cases as possible. Happily for symmetric differentials applied to symmetric matrices, the naive formulas of symmetrizing the gradient just work. See, e.g. https://arxiv.org/abs/1911.06491 https://arxiv.org/abs/1911.06491 for some tracking down of the history and cogent analysis of what went wrong and https://saturdaygenfo.github.io/posts/symmetric-gradients/ https://saturdaygenfo.github.io/posts/symmetric-gradients/ for why it matters in practice (e.g. wrong and changing direction of descent for gradient descent, though still "downwards").

usgroup 5y ago

The responses confuse "machine learning" with neural networks so far as I can see. I do lots of statsy things: good luck with that outside of R.

jiggunjer 5y ago
Even professors doing trendy "AI" research conflate the two terms.
- dagw 5y ago
  Even professors doing trendy "AI" research conflate the two terms. Only because doing "AI" gets you bigger grants.
  - kgwgk 5y ago
    Tibshirani's Glossary Machine learning Statistics network, graphs model weights parameters learning fitting generalization test set performance supervised learning regression/classiﬁcation unsupervised learning density estimation, clustering large grant = $1,000,000 large grant = $50,000 nice place to have a meeting: nice place to have a meeting: Snowbird, Utah, French Alps Las Vegas in August
deleted 5y ago
[deleted]
fault1 5y ago
turing, mlj, etc are quite good. https://turing.ml/stable/ https://turing.ml/stable/ https://alan-turing-institute.github.io/MLJ.jl/dev/about_mlj/ https://alan-turing-institute.github.io/MLJ.jl/dev/about_mlj...
- usgroup 5y ago
  Until you've used: https://r-nimble.org/ https://r-nimble.org/

savant_penguin 5y ago

Having played around with Julia recently there are two main reasons I would not recommend using it for ML: 1-Plotting takes too long. Forget fast iterations. You'll be left waiting for non trivial amounts of time to your first plot. The latency between tests is simply not good enough. 2-Libraries are just not mature. Plotting anything not too standard yields broken graphs and you have to look for a billion of backends to find one that does not break (hello twinx). Matplotlib is slow for some graphs but more often than not you get the right visualization (except for that suptitle being cut out of the graph bug that will never be fixed). The requests library is not robust enough. This one is quite possibility my fault. I have received "bad headers" errors for some websites, but doing the exact same "get"with python works just fine. The impression that Libraries die all the time. Could be just an impression, but it is quite common to find top stackoverflow answers that point you to dead libraries (with the common "use this other library instead" hint) I like Julia speed and well done memory management (especially from what I saw from Dataframe.jl high memory benchmarks). But right now I'd not use it in a critical environment. (And I also suck at reading and understanding the error messages thus far, but it's likely a lack of experience from my side)

whlr 5y ago
It’s worth noting that PyPlot.jl is a nearly seamless way to use matplotlib from Julia. JIT compilation of the middle layer does mean that it suffers from time to first plot problems though.
cs702 5y ago
As a longtime user of Python for ML/AI, I love Julia... but agree that most people will find Julia challenging to use for everyday data munging, or for developing production code, say, in a business environment. The Python ecosystem (people, resources, libraries, frameworks, etc.) is superior for most common, plain-vanilla use cases. However, if you are developing compute-intensive scientific software, I would recommend switching to Julia as soon as possible. IMHO, Julia is superior to the likes of Python, Fortran, C++, and C for those use cases. -- PS. By the way, you can fix matplotlib's suptitle issue by calling this Figure method after the fact: fig.tight_layout(rect=(0, 0, 1, 0.95)) # leaves 5% of top space for suptitle

melony 5y ago

When will Enzyme and Diffractor hit prod? The default right now for Flux is still Zygote.

muktuks 5y ago

Julia is great for NEW ML research. But it has serious downsides for mundane or even above average production work... Flux has historically been more of a research project then a production library. That's ok, it holds a lot of promise. But, time to first gradient can be excessive. Minutes to discover a small bug excessive. Last I checked all of the adjacent libraries around it were broken due to incredibly frequent changes to the API. Mlj doesn't impress me. There was a serious opportunity to change the way people do routine modelling using the strengths of julia. They basically made a half baked sklearn instead. Missed opportunity in my mind. In general the components for statistical libraries are exceptional. However the ecosystem falls short to do things with them in a stable way. The best libraries imo for data science work stem from the excellent db adapters, optimization library's, data frames jl, Turing jl, etc. Anything beyond that and it's usually better off just rolling your own or using another language all together... Current state of production, ie not code running a notebook is also ok at best...

amkkma 5y ago
What are your concerns with MLJ?
- muktuks 5y ago
  MLJ doesn't concern me in a bad way. At it's heart it's an abstraction for ML components across many packages with some machinery on top for parameter tuning, etc. That's fine. What it's missing is ... impactful improvements on the failures of libraries like this. Docs need more examples and less pointers to packages that wrap other packages(albeit they are written well). Despite being a multi-year long effort by very big names it still feels incomplete, and if someone wasn't sold on Julia there is technically no reason not to use R or Python or Matlab. Missed opportunity is all
UncleOxidant 5y ago
> Julia is great for NEW ML research. But it has serious downsides for mundane or even above average production work... This was how PyTorch started out as well.

drannex 5y ago

Absolutely adore prototyping in Julia, absolutely heinous to work in production. Their memory usage is insanely high and their calc speeds are slow when you don't have the insanely high processing equipment needed. If they could just singularly focus on reducing memory consumption than the language will be likely _the defacto_ language for ML and analysis, but until then it really gets bogged down by just how much memory is needed for the most simple of functions.

rundigen12 5y ago

Paraphrase: ML in Julia is for Serious Scientists Doing Very Sciencey Things. Things you wouldn’t understand. But your desire to train deep learning models on GPUs and deploy them as apps elsewhere? ...Aw, c’mon, that’s boring. You can just use PyTorch for that. Now, who's up for a new round of Neural ODE benchmarks?!