6 ms·

Building a Language and Compiler for Machine Learning

deleted 8y ago
[deleted]
ChrisRackauckas 8y ago
As someone who works in merging differential equations and machine learning, I have found this kind of work essential for what I do. Pervasive AD that allows merging neural networks and diffeq solvers is allowing us to explore all of kinds of new models and new problems. Sure it doesn't impact vanilla machine learning all that much (though Zygote.jl does allow for a lot of optimizations that wouldn't be possible with tracing-based AD), but it definitely opens up a new wave of AI possibilities.
- mlevental 8y ago
  how do you use AD for diffeqs? do you really have systems of diffeqs large enough that you need AD for evaluating a solution at a particular point? or do you need backbrop for something that i can't imagine?
  - ChrisRackauckas 8y ago
    For parameter estimation. In parameter estimation, you have to evaluate the derivative of a cost function based on an ODE solution, which is usually something like the L2 norm between your numerical solution points and your data. The gradient of this cost function requires calculating the gradient of the solution with respect to the parameters, which can be done quite well via AD. We are finding that AD methods work better than the traditional sensitivity analysis methods in many cases.
    - mlevental 8y ago
      >parameter estimation you mean given some data that's modeled by an ODE you want to fit the ODE to the data (and therefore discover parameters of the ODE that would have produced that data) ?
      - ChrisRackauckas 8y ago
        Yes exactly.
- EE84M3i 8y ago
  As someone who works in merging differential equations and machine learning, I have found this kind of work essential for what I do. Pervasive AD that allows merging neural networks and diffeq solvers is allowing us to explore all of kinds of new models and new problems. Sure it doesn't impact vanilla machine learning all that much (though Zygote.jl does allow for a lot of optimizations that wouldn't be possible with tracing-based AD), but it definitely opens up a new wave of AI possibilities. I thought I was having some kind of stroke or terrible deja-vu (didn't I read this comment earlier this morning?) until I realized you copy and pasted your comment from 20 hours ago on https://news.ycombinator.com/item?id=18594103 https://news.ycombinator.com/item?id=18594103
  - ChrisRackauckas 8y ago
    Yeah, the other thread caught on so I copy/pasted this post to the thread people were using. But now this thread caught on too... something weird happened but :shrug:.
- dnautics 8y ago
  there are a few things about Flux that bugs me. It automatically assumes that I want to optimize matrix multiplications by parallelizing those against cores, which has Amdahl scaling, instead of parallelizing across samples in the batch, which has Gustafson scaling. It would probably help if batches and minibatches (or something like that) were datatypes, which they are not. Doing something like this would probably also help with distributing computation, down the line. I'm also not entirely sure what is going on under the hood with Tracker types, and the documentation is not that great, which became a problem when I was trying to chase down errors in something really custom I was doing. I much prefer Knet's way of autodifferentiating, which is more intuitive to me, but Knet's layering doesn't feel as nice as Flux's. I really wish GPU computation in Julia had a different semantic - by making it a 'virtual computational node', accessible using Distributed module with the same semantics as a totally separate node. That would really make async distributed batch processing a thing, the system could profile all the nodes in use and if we really want to get fancy be able to use something like JUMP to make best use of the processing power available to it.
  - taliesinb 8y ago
    Can you elaborate on what you mean by "parallelizing those against cores" vs "parallelizing across samples in a batch"?
    - dnautics 8y ago
      so for example, if you want to do M = [1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 1] v1 = [1,2,3,4] (column vector) v2 = [4,5,6,7] (column vector) you can either do (M * v1, M * v2) on two cores as r1a = [1 0 0 0 0 1 0 0] * v1 (core 1) r1b = [0 0 1 0 0 0 0 1] * v1 (core 2) r2a = [1 0 0 0 0 1 0 0] * v2 (core 1) r2b = [0 0 1 0 0 0 0 1] * v2 (core 2) then r1 = flatten([r1a, r1b]) r2 = flatten([r2a, r2b]) OR, you could just have the whole model in each core: r1 = M * v1 (core 1) r2 = M * v2 (core 2)
      - taliesinb 8y ago
manojlds 8y ago
https://news.ycombinator.com/item?id=18593453 https://news.ycombinator.com/item?id=18593453
pbalau 8y ago
> [...] bake for fifteen minutes and out pops a fully-featured ML stack where is logging, where is model storage and versioning, where is input data processing and normalizing, where is results processing?
- mlevental 8y ago
  lowbrow comment. the hard part of ML stacks is AD and GPU not all of those other things (i'm sure there has been zero cutting edge research done on better ways to log).
  - one-more-minute 8y ago
    Yes, and unlike AD and GPU support, things like logging have nothing (special) to do with ML. Julia has both very nice logging and plenty of good serialisation options, all of which works nicely with the ML stack. It's entirely unnecessary to duplicate these tools just so they can be baked in to a huge framework.
    - ChrisRackauckas 8y ago
      Well, I think there is something to be said there though. The reason why the Julia stack is nice is because Julia's standard logging tools can be used for logging in ML codes. Even other things like Julia's standard progress monitoring toolbars just work on ML codes. That's quite a surprising result. Tools which build a sub-language for graph building like TensorFlow have to build and document such tooling. So for newcomers to Julia, they will search the package documentation and package codes and find nothing. It is a confusing problem because the functionality exists but no one thought to document its usage for this context since it is just the standard Julia usage!
- xiphias2 8y ago
  I agree that it shouldn't be called a fully featured ML stack. Still, the most important feature (optimized compilation of models) is handled quite well. Whenever I have to look at Tensorflow source code to understand how it works, I see an over-complicated system that is too far from the research papers (which makes it hard to work with it).
yahyaheee 8y ago
I’m finding the ML work being done in Julia very refreshing. It feels like they are building things right from the ground up and the community is great to work with.
xvilka 8y ago
Hopefully one day Julia won't need patched LLVM. Will improve packaging in various distributions too.

deleted 8y ago

[deleted]

As someone who works in merging differential equations and machine learning, I have found this kind of work essential for what I do. Pervasive AD that allows merging neural networks and diffeq solvers is allowing us to explore all of kinds of new models and new problems. Sure it doesn't impact vanilla machine learning all that much (though Zygote.jl does allow for a lot of optimizations that wouldn't be possible with tracing-based AD), but it definitely opens up a new wave of AI possibilities.

mlevental 8y ago
how do you use AD for diffeqs? do you really have systems of diffeqs large enough that you need AD for evaluating a solution at a particular point? or do you need backbrop for something that i can't imagine?
- ChrisRackauckas 8y ago
  For parameter estimation. In parameter estimation, you have to evaluate the derivative of a cost function based on an ODE solution, which is usually something like the L2 norm between your numerical solution points and your data. The gradient of this cost function requires calculating the gradient of the solution with respect to the parameters, which can be done quite well via AD. We are finding that AD methods work better than the traditional sensitivity analysis methods in many cases.
  - mlevental 8y ago
    >parameter estimation you mean given some data that's modeled by an ODE you want to fit the ODE to the data (and therefore discover parameters of the ODE that would have produced that data) ?
    - ChrisRackauckas 8y ago
      Yes exactly.
EE84M3i 8y ago
As someone who works in merging differential equations and machine learning, I have found this kind of work essential for what I do. Pervasive AD that allows merging neural networks and diffeq solvers is allowing us to explore all of kinds of new models and new problems. Sure it doesn't impact vanilla machine learning all that much (though Zygote.jl does allow for a lot of optimizations that wouldn't be possible with tracing-based AD), but it definitely opens up a new wave of AI possibilities. I thought I was having some kind of stroke or terrible deja-vu (didn't I read this comment earlier this morning?) until I realized you copy and pasted your comment from 20 hours ago on https://news.ycombinator.com/item?id=18594103 https://news.ycombinator.com/item?id=18594103
- ChrisRackauckas 8y ago
  Yeah, the other thread caught on so I copy/pasted this post to the thread people were using. But now this thread caught on too... something weird happened but :shrug:.
dnautics 8y ago
there are a few things about Flux that bugs me. It automatically assumes that I want to optimize matrix multiplications by parallelizing those against cores, which has Amdahl scaling, instead of parallelizing across samples in the batch, which has Gustafson scaling. It would probably help if batches and minibatches (or something like that) were datatypes, which they are not. Doing something like this would probably also help with distributing computation, down the line. I'm also not entirely sure what is going on under the hood with Tracker types, and the documentation is not that great, which became a problem when I was trying to chase down errors in something really custom I was doing. I much prefer Knet's way of autodifferentiating, which is more intuitive to me, but Knet's layering doesn't feel as nice as Flux's. I really wish GPU computation in Julia had a different semantic - by making it a 'virtual computational node', accessible using Distributed module with the same semantics as a totally separate node. That would really make async distributed batch processing a thing, the system could profile all the nodes in use and if we really want to get fancy be able to use something like JUMP to make best use of the processing power available to it.
- taliesinb 8y ago
  Can you elaborate on what you mean by "parallelizing those against cores" vs "parallelizing across samples in a batch"?
  - dnautics 8y ago
    so for example, if you want to do M = [1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 1] v1 = [1,2,3,4] (column vector) v2 = [4,5,6,7] (column vector) you can either do (M * v1, M * v2) on two cores as r1a = [1 0 0 0 0 1 0 0] * v1 (core 1) r1b = [0 0 1 0 0 0 0 1] * v1 (core 2) r2a = [1 0 0 0 0 1 0 0] * v2 (core 1) r2b = [0 0 1 0 0 0 0 1] * v2 (core 2) then r1 = flatten([r1a, r1b]) r2 = flatten([r2a, r2b]) OR, you could just have the whole model in each core: r1 = M * v1 (core 1) r2 = M * v2 (core 2)
    - taliesinb 8y ago

manojlds 8y ago

https://news.ycombinator.com/item?id=18593453 https://news.ycombinator.com/item?id=18593453

pbalau 8y ago

> [...] bake for fifteen minutes and out pops a fully-featured ML stack where is logging, where is model storage and versioning, where is input data processing and normalizing, where is results processing?

mlevental 8y ago
lowbrow comment. the hard part of ML stacks is AD and GPU not all of those other things (i'm sure there has been zero cutting edge research done on better ways to log).
- one-more-minute 8y ago
  Yes, and unlike AD and GPU support, things like logging have nothing (special) to do with ML. Julia has both very nice logging and plenty of good serialisation options, all of which works nicely with the ML stack. It's entirely unnecessary to duplicate these tools just so they can be baked in to a huge framework.
  - ChrisRackauckas 8y ago
    Well, I think there is something to be said there though. The reason why the Julia stack is nice is because Julia's standard logging tools can be used for logging in ML codes. Even other things like Julia's standard progress monitoring toolbars just work on ML codes. That's quite a surprising result. Tools which build a sub-language for graph building like TensorFlow have to build and document such tooling. So for newcomers to Julia, they will search the package documentation and package codes and find nothing. It is a confusing problem because the functionality exists but no one thought to document its usage for this context since it is just the standard Julia usage!
xiphias2 8y ago
I agree that it shouldn't be called a fully featured ML stack. Still, the most important feature (optimized compilation of models) is handled quite well. Whenever I have to look at Tensorflow source code to understand how it works, I see an over-complicated system that is too far from the research papers (which makes it hard to work with it).

yahyaheee 8y ago

I’m finding the ML work being done in Julia very refreshing. It feels like they are building things right from the ground up and the community is great to work with.

xvilka 8y ago

Hopefully one day Julia won't need patched LLVM. Will improve packaging in various distributions too.