7 ms·

wat

KenoFischer 8y ago
This is a talk we gave at the "Compilers for Machine Learning" workshop at CGO this past Sunday. Obviously the slides are missing some of the context of the presentation, but the core argument is that one shouldn't build compilers for machine learning, but rather build general purpose compilers that are flexible and extensible. From there, you can easily get all the same benefits as dedicated machine learning compilers without building a huge monolith. The rest of the slides highlight some of the work done in the Julia community to work in this direction.
- ViralBShah 8y ago
  Adding to the context, we wrote up a little post about the C4ML workshop: https://juliacomputing.com/blog/2019/02/19/growing-a-compiler.html https://juliacomputing.com/blog/2019/02/19/growing-a-compile... Reproducing it here: The Compilers for Machine Learning workshop was recently held at CGO 2019. Since compiler techniques affect a large part of the machine learning stack, this workshop aimed to highlight research that incorporates compiler techniques and algorithms in optimizing machine learning workloads. The workshop included talks from various projects - Julia (Julia Computing), TVM (UW), Glow (Facebook), XLA (Google), nGraph (Intel), TensorRT (Nvidia), and the soon to release MLIR (Google). Our talk introduced the abstractions in the Julia language and the kind of compiler transforms involved in implementing them. We then had a deep dive into dynamic semantics + static analysis - our JAOT (Just-Ahead-Of-Time) analysis. Building on these capabilities, the Zygote system implements automatic differentiation, effectively treating it as a compiler problem, giving us differentiable programming for free. Finally, compiler backends for GPUs and TPUs give us high performance execution. All this comes together beautifully in Neural ODEs, which we had to show off as our first slide!
  - ViralBShah 8y ago
    Also, since Yann LeCun said yesterday that Deep Learning needs a new language, this has become a major topic of interest. https://venturebeat.com/2019/02/18/facebooks-chief-ai-scientist-deep-learning-may-need-a-new-programming-language/ https://venturebeat.com/2019/02/18/facebooks-chief-ai-scient... Of course, our view is that Julia is one such language that people should consider seriously. The talk linked here is a peek under the hood and shows that differentiable programming in Julia is not a special add-on, but something that fits naturally within the language.
    - ced 8y ago
      A long time ago, Yann LeCun wrote Lush, which was a numerically-focused Lisp dialect, with a focus on C interop. It might be one of Julia's closest sibling. > Lush is an object-oriented programming language designed for researchers, experimenters, and engineers interested in large-scale numerical and graphic applications. Lush is designed to be used in situations where one would want to combine the flexibility of a high-level, weakly-typed interpreted language, with the efficiency of a strongly-typed, natively-compiled language, and with the easy integration of code written in C, C++, or other languages. http://lush.sourceforge.net/index.html http://lush.sourceforge.net/index.html Is there more detail about his proposal? It must be well thought-out.
      - sitkack 8y ago
        A Lisp with expressive macro support is at least what is needed. Nothing I know of nothing else that has the affordances to support internal DSLs and creative control flow.
        yaantc 8y ago
        Julia as hygienic macros. From their documentation [1]: "The strongest legacy of Lisp in the Julia language is its metaprogramming support. Like Lisp, Julia represents its own code as a data structure of the language itself." Parts of Julia are even implemented in Lisp [2], although they tend to be ported to Julia now IIUC. But it's clear the Julia developers are well aware of Lisp and its strong points. [1] https://docs.julialang.org/en/v1/manual/metaprogramming/ https://docs.julialang.org/en/v1/manual/metaprogramming/ [2] https://discourse.julialang.org/t/the-role-of-femtolisp-in-julia/1902 https://discourse.julialang.org/t/the-role-of-femtolisp-in-j...
        pjmlp 8y ago
        Julia also feels a bit like Dylan, another Lisp offspring with Algol-like syntax.
        sitkack 8y ago
- SilasX 8y ago
  How ... whatever was the reason for thinking that you needed a ML-specific compiler? Do people also think you need a timecard-tracking-specific compiler?
  - KenoFischer 8y ago
    Well, I do think there are some very real shortcomings that current general purpose compilers have when applied to machine learning (aggrevated by the fact that lots of machine learning code is written in languages with poor compiler support), so lots people looked at that, wrote small optimizers and got good performance. But then it turned out that researchers wanted to do more and more things in their ML models and these small optimizers are turning into full-blown compilers, with not a lot of thought about whether that is truly the correct thing to do.
  - mlevental 8y ago
    do you know how modern ml works? do you know that every functional unit in a net needs to be differentiable and so needs to carry around either dual numbers (forward mode) or adjoints (reverse mode)? it's not as simple as just writing a math library.
    - ViralBShah 8y ago
      There's an extensive discussion of AD in the slides (the topic of this HN post), and how it is done in Julia. Precisely because it is not as simple as writing a math library is why you need language and compiler support.
      - mlevental 8y ago
        yes that's what i was pointing out the poster i responded to
    - sitkack 8y ago
      Please give the parent a benefit of the doubt. Do you even comments are alienating. It could have been written as Every functional unit in a net needs to be differentiable and so needs to carry around either dual numbers (forward mode) or adjoints (reverse mode). These requirements necessitate new languages and compilers that current imperative representations don’t currently support.
- jng 8y ago
  Is there a video of the talk? Would love to watch it. Congrats on the work, it looks great! I'm just missing more detail, since having just the slides is a bit on the dry side.
deleted 8y ago
[deleted]

wat

This is a talk we gave at the "Compilers for Machine Learning" workshop at CGO this past Sunday. Obviously the slides are missing some of the context of the presentation, but the core argument is that one shouldn't build compilers for machine learning, but rather build general purpose compilers that are flexible and extensible. From there, you can easily get all the same benefits as dedicated machine learning compilers without building a huge monolith. The rest of the slides highlight some of the work done in the Julia community to work in this direction.

ViralBShah 8y ago
Adding to the context, we wrote up a little post about the C4ML workshop: https://juliacomputing.com/blog/2019/02/19/growing-a-compiler.html https://juliacomputing.com/blog/2019/02/19/growing-a-compile... Reproducing it here: The Compilers for Machine Learning workshop was recently held at CGO 2019. Since compiler techniques affect a large part of the machine learning stack, this workshop aimed to highlight research that incorporates compiler techniques and algorithms in optimizing machine learning workloads. The workshop included talks from various projects - Julia (Julia Computing), TVM (UW), Glow (Facebook), XLA (Google), nGraph (Intel), TensorRT (Nvidia), and the soon to release MLIR (Google). Our talk introduced the abstractions in the Julia language and the kind of compiler transforms involved in implementing them. We then had a deep dive into dynamic semantics + static analysis - our JAOT (Just-Ahead-Of-Time) analysis. Building on these capabilities, the Zygote system implements automatic differentiation, effectively treating it as a compiler problem, giving us differentiable programming for free. Finally, compiler backends for GPUs and TPUs give us high performance execution. All this comes together beautifully in Neural ODEs, which we had to show off as our first slide!
- ViralBShah 8y ago
  Also, since Yann LeCun said yesterday that Deep Learning needs a new language, this has become a major topic of interest. https://venturebeat.com/2019/02/18/facebooks-chief-ai-scientist-deep-learning-may-need-a-new-programming-language/ https://venturebeat.com/2019/02/18/facebooks-chief-ai-scient... Of course, our view is that Julia is one such language that people should consider seriously. The talk linked here is a peek under the hood and shows that differentiable programming in Julia is not a special add-on, but something that fits naturally within the language.
  - ced 8y ago
    A long time ago, Yann LeCun wrote Lush, which was a numerically-focused Lisp dialect, with a focus on C interop. It might be one of Julia's closest sibling. > Lush is an object-oriented programming language designed for researchers, experimenters, and engineers interested in large-scale numerical and graphic applications. Lush is designed to be used in situations where one would want to combine the flexibility of a high-level, weakly-typed interpreted language, with the efficiency of a strongly-typed, natively-compiled language, and with the easy integration of code written in C, C++, or other languages. http://lush.sourceforge.net/index.html http://lush.sourceforge.net/index.html Is there more detail about his proposal? It must be well thought-out.
    - sitkack 8y ago
      A Lisp with expressive macro support is at least what is needed. Nothing I know of nothing else that has the affordances to support internal DSLs and creative control flow.
      - yaantc 8y ago
        Julia as hygienic macros. From their documentation [1]: "The strongest legacy of Lisp in the Julia language is its metaprogramming support. Like Lisp, Julia represents its own code as a data structure of the language itself." Parts of Julia are even implemented in Lisp [2], although they tend to be ported to Julia now IIUC. But it's clear the Julia developers are well aware of Lisp and its strong points. [1] https://docs.julialang.org/en/v1/manual/metaprogramming/ https://docs.julialang.org/en/v1/manual/metaprogramming/ [2] https://discourse.julialang.org/t/the-role-of-femtolisp-in-julia/1902 https://discourse.julialang.org/t/the-role-of-femtolisp-in-j...
        pjmlp 8y ago
        Julia also feels a bit like Dylan, another Lisp offspring with Algol-like syntax.
        sitkack 8y ago
SilasX 8y ago
How ... whatever was the reason for thinking that you needed a ML-specific compiler? Do people also think you need a timecard-tracking-specific compiler?
- KenoFischer 8y ago
  Well, I do think there are some very real shortcomings that current general purpose compilers have when applied to machine learning (aggrevated by the fact that lots of machine learning code is written in languages with poor compiler support), so lots people looked at that, wrote small optimizers and got good performance. But then it turned out that researchers wanted to do more and more things in their ML models and these small optimizers are turning into full-blown compilers, with not a lot of thought about whether that is truly the correct thing to do.
- mlevental 8y ago
  do you know how modern ml works? do you know that every functional unit in a net needs to be differentiable and so needs to carry around either dual numbers (forward mode) or adjoints (reverse mode)? it's not as simple as just writing a math library.
  - ViralBShah 8y ago
    There's an extensive discussion of AD in the slides (the topic of this HN post), and how it is done in Julia. Precisely because it is not as simple as writing a math library is why you need language and compiler support.
    - mlevental 8y ago
      yes that's what i was pointing out the poster i responded to
  - sitkack 8y ago
    Please give the parent a benefit of the doubt. Do you even comments are alienating. It could have been written as Every functional unit in a net needs to be differentiable and so needs to carry around either dual numbers (forward mode) or adjoints (reverse mode). These requirements necessitate new languages and compilers that current imperative representations don’t currently support.
jng 8y ago
Is there a video of the talk? Would love to watch it. Congrats on the work, it looks great! I'm just missing more detail, since having just the slides is a bit on the dry side.

deleted 8y ago

[deleted]