4 ms·

Halide: a language for fast, portable computation on images and tensors

146 points by espeed 8y ago

espeed 8y ago
Among other optimizations and advancements, Halide provides compiler support for advanced latency-hiding scheduling/optimizations beyond the capabilities of generic compilers, which essentially enables you to generate highly-parallel optimized CPU/GPU compute kernels to make memory accesses free.
all2 8y ago
https://www.youtube.com/watch?v=3uiEyEKji0M https://www.youtube.com/watch?v=3uiEyEKji0M Here is a relevant video from the creator. It has some fun illustrations to help with what its going on. /edit - letters
- espeed 8y ago
  In addition to this video overview by Andrew Adams from the Stanford course [1], there's also an MSR talk given by Jonathan Kelley [2], and there may also be a Google TechTalk somewhere on YouTube (don't remember for sure so just search for "Halide" or by one of the authors' names). And of course there's also the videos of lectures of the MIT 6.815/6.865 course on computational photography given by Frédo Durand (those videos are linked to from the Halide home page under the Getting Started section [3]). [1] Stanford CVPR2015 http://halide-lang.org/cvpr2015.html http://halide-lang.org/cvpr2015.html [2] Decoupling Algorithms from the Organization of Computation for High-Performance Graphics & Imaging (MSR talk given by Jonathan Kelley) [video] https://www.youtube.com/watch?v=dnFccCGvT90 https://www.youtube.com/watch?v=dnFccCGvT90 [3] MIT 6.815/6.865 Digital & Computational Photography lecture videos by Frédo Durand http://halide-lang.org/#gettingstarted http://halide-lang.org/#gettingstarted
- puzzle 8y ago
  The funny thing is that YT itself has been using Halide to process all videos, perhaps as early as when that one was uploaded. And anyone using Google Camera has been running Halide code, from Pixel phones all the way back to Glass, which took decent photos despite having hardware that, due to the form factor, was less powerful than the Galaxy Nexus that came out 1.5 years before.
  - all2 8y ago
    Do you have a source for this? Maybe examples of code in-situ?
    - puzzle 8y ago
      The use by YT and GCam has been mentioned, among others, at https://suif.stanford.edu/~courses/cs243/lectures/l14-halide.pdf https://suif.stanford.edu/~courses/cs243/lectures/l14-halide... (last slide) More mentions by Google: https://www.blog.google/products/pixel/pixel-visual-core-image-processing-and-machine-learning-pixel-2/ https://www.blog.google/products/pixel/pixel-visual-core-ima... https://productforums.google.com/forum/#!topic/phone-by-google/JuosYoHtgEE https://productforums.google.com/forum/#!topic/phone-by-goog...
    - __abadams__ 8y ago
      The HDR+ siggraph paper (hdrplusdata.org) states that they use Halide, as does the newer siggraph paper on the Pixel's portrait mode (http://graphics.stanford.edu/papers/portrait/wadhwa-portrait-sig18.pdf http://graphics.stanford.edu/papers/portrait/wadhwa-portrait...), and the paper on the Pixel's video denoising algorithm (http://www.chiakailiang.org/papers/ehmann_2018_rtv.pdf http://www.chiakailiang.org/papers/ehmann_2018_rtv.pdf). The Pixel camera is riddled with Halide code, but of course it's all closed-source. I can also vouch for the youtube use, having been tangentially involved in it and given permission to state that they use it.
stochastic_monk 8y ago
Previous discussions: https://news.ycombinator.com/from?site=halide-lang.org https://news.ycombinator.com/from?site=halide-lang.org
- espeed 8y ago
  I'm interested to see how well Halide's approach would pair with the GPU optimizations being put into GraphBLAS [1]. [1] https://hn.algolia.com/?query=GraphBLAS&sort=byPopularity&prefix&page=0&dateRange=all&type=all https://hn.algolia.com/?query=GraphBLAS&sort=byPopularity&pr...
  - __abadams__ 8y ago
    Halide doesn't currently have good ways to handle sparse matrices, unless they have some sort of special structure that lets you pack them into rectangular shapes (e.g. a constant number of non-zeros entries per row). It would be an interesting language extension. The big thing to add would be a way to reduce over the non-zero entries more efficiently than checking them all.
- dang 8y ago
  Mainly https://news.ycombinator.com/item?id=7544855 https://news.ycombinator.com/item?id=7544855 from 2014. Also https://news.ycombinator.com/item?id=10410832 https://news.ycombinator.com/item?id=10410832 from 2015.
bogdanoff_2 8y ago
If the creator of that page is reading this, please fix the color of the code example text. Right now it's a dark shade of gray that is hard to distinguish from the background until syntax highlighting kicks in. While you're at it, maybe figure out why it takes almost ten seconds for the syntax highlighting to kick in.
- kstrauser 8y ago
  I agree, although it was more like 2 seconds on my laptop. It was almost unreadable to me until the colors leapt in.
- jrk 8y ago
  Sorry, poor choice in the default Bootstrap style we adopted years ago. Under normal conditions the styles kick in in a second or less, but I've fixed it nonetheless.
ipsum2 8y ago
There was a paper released recently to do deep learning on Halide[1] that I'm excited to see merge[2] with the main Halide codebase. Seems like the creators of Halide are in this comment thread, does anyone know when this is happening? [1]: https://people.csail.mit.edu/tzumao/gradient_halide/ https://people.csail.mit.edu/tzumao/gradient_halide/ [2]: https://github.com/jrk/gradient-halide https://github.com/jrk/gradient-halide
- __abadams__ 8y ago
  It works but still needs a bit of cleanup. I was planning to get back to it after an upcoming conference deadline. Here's the PR: https://github.com/halide/Halide/pull/3220 https://github.com/halide/Halide/pull/3220
- gammaradiation 8y ago
  For deep learning specifically, the TVM project (http://tvm.ai http://tvm.ai) is a substantially more advanced project than Halide, with a much larger community. I’d highly recommend checking that out.
  - __abadams__ 8y ago
    You again? The only other time your account has posted was to plug TVM in a different post on Halide. If you're not a Tianqi Chen sockpuppet, then you should apologize to him for making him look like an asshole. For the benefit of others not aware: TVM took Halide, deleted the half of it they didn't understand, then reimplemented it (copy-pasting large chunks without credit, hacks and all) and claimed it was a revolutionary new deep learning IR. People who already knew Halide were left scratching their heads unable to tell how this wasn't just Halide re-marketed in a different community.
  - jrk 8y ago
    Good to see the TVM troll back. As ever, it's unclear what "substantially more advanced" means, and to cherry-pick just some of the biggest users, there's a large and growing amount of Halide code in Android, Photoshop, and numerous Google services. I'll let billions of devices and tens of thousands of servers speak for themselves.
keldaris 8y ago
Halide seems like it might have immense potential for fairly general applications in scientific computing, but all of the documentation, tutorials and examples seem dominated by image processing. Could someone elaborate on the limitations of Halide with respect to applications like finite difference and finite element methods, lattice Boltzmann simulations, etc.? These are all ultimately reducible to basic linear algebra and stencil kernels, which seem well suited to the design of Halide. Should computational scientists give it a serious look, or am I missing something?
- __abadams__ 8y ago
  Halide is generally good for pipelines that do math on multidimensional arrays, so quite possibly. It has a property that I think would make it hard to use for some applications in scientific computing though - while you can mutate in-place, you can't mutate an array ("Func" in Halide) after it has been used by another pipeline stage. This means that the dependency graph between your pipeline stages has no cycles except for self-cycles, which lets us do lots of clever stuff like bounds inference to guarantee the program is correct regardless of the schedule. I can imagine things like flow simulation having a hard time with that constraint. In the deep learning context this means that CNNs are ideal, but RNNs can be awkward or impossible.
  - boulos 8y ago
    Andrew: have you considered adding something akin to Rust's lifetimes here? While you'd have restrictions on the schedule, it seems like it's still let you express who in the "graph" messes with the data as tightly as possible.

146 points by espeed 8y ago

Among other optimizations and advancements, Halide provides compiler support for advanced latency-hiding scheduling/optimizations beyond the capabilities of generic compilers, which essentially enables you to generate highly-parallel optimized CPU/GPU compute kernels to make memory accesses free.

all2 8y ago

https://www.youtube.com/watch?v=3uiEyEKji0M https://www.youtube.com/watch?v=3uiEyEKji0M Here is a relevant video from the creator. It has some fun illustrations to help with what its going on. /edit - letters

espeed 8y ago
In addition to this video overview by Andrew Adams from the Stanford course [1], there's also an MSR talk given by Jonathan Kelley [2], and there may also be a Google TechTalk somewhere on YouTube (don't remember for sure so just search for "Halide" or by one of the authors' names). And of course there's also the videos of lectures of the MIT 6.815/6.865 course on computational photography given by Frédo Durand (those videos are linked to from the Halide home page under the Getting Started section [3]). [1] Stanford CVPR2015 http://halide-lang.org/cvpr2015.html http://halide-lang.org/cvpr2015.html [2] Decoupling Algorithms from the Organization of Computation for High-Performance Graphics & Imaging (MSR talk given by Jonathan Kelley) [video] https://www.youtube.com/watch?v=dnFccCGvT90 https://www.youtube.com/watch?v=dnFccCGvT90 [3] MIT 6.815/6.865 Digital & Computational Photography lecture videos by Frédo Durand http://halide-lang.org/#gettingstarted http://halide-lang.org/#gettingstarted
puzzle 8y ago
The funny thing is that YT itself has been using Halide to process all videos, perhaps as early as when that one was uploaded. And anyone using Google Camera has been running Halide code, from Pixel phones all the way back to Glass, which took decent photos despite having hardware that, due to the form factor, was less powerful than the Galaxy Nexus that came out 1.5 years before.
- all2 8y ago
  Do you have a source for this? Maybe examples of code in-situ?
  - puzzle 8y ago
    The use by YT and GCam has been mentioned, among others, at https://suif.stanford.edu/~courses/cs243/lectures/l14-halide.pdf https://suif.stanford.edu/~courses/cs243/lectures/l14-halide... (last slide) More mentions by Google: https://www.blog.google/products/pixel/pixel-visual-core-image-processing-and-machine-learning-pixel-2/ https://www.blog.google/products/pixel/pixel-visual-core-ima... https://productforums.google.com/forum/#!topic/phone-by-google/JuosYoHtgEE https://productforums.google.com/forum/#!topic/phone-by-goog...
  - __abadams__ 8y ago
    The HDR+ siggraph paper (hdrplusdata.org) states that they use Halide, as does the newer siggraph paper on the Pixel's portrait mode (http://graphics.stanford.edu/papers/portrait/wadhwa-portrait-sig18.pdf http://graphics.stanford.edu/papers/portrait/wadhwa-portrait...), and the paper on the Pixel's video denoising algorithm (http://www.chiakailiang.org/papers/ehmann_2018_rtv.pdf http://www.chiakailiang.org/papers/ehmann_2018_rtv.pdf). The Pixel camera is riddled with Halide code, but of course it's all closed-source. I can also vouch for the youtube use, having been tangentially involved in it and given permission to state that they use it.

stochastic_monk 8y ago

Previous discussions: https://news.ycombinator.com/from?site=halide-lang.org https://news.ycombinator.com/from?site=halide-lang.org

espeed 8y ago
I'm interested to see how well Halide's approach would pair with the GPU optimizations being put into GraphBLAS [1]. [1] https://hn.algolia.com/?query=GraphBLAS&sort=byPopularity&prefix&page=0&dateRange=all&type=all https://hn.algolia.com/?query=GraphBLAS&sort=byPopularity&pr...
- __abadams__ 8y ago
  Halide doesn't currently have good ways to handle sparse matrices, unless they have some sort of special structure that lets you pack them into rectangular shapes (e.g. a constant number of non-zeros entries per row). It would be an interesting language extension. The big thing to add would be a way to reduce over the non-zero entries more efficiently than checking them all.
dang 8y ago
Mainly https://news.ycombinator.com/item?id=7544855 https://news.ycombinator.com/item?id=7544855 from 2014. Also https://news.ycombinator.com/item?id=10410832 https://news.ycombinator.com/item?id=10410832 from 2015.

bogdanoff_2 8y ago

If the creator of that page is reading this, please fix the color of the code example text. Right now it's a dark shade of gray that is hard to distinguish from the background until syntax highlighting kicks in. While you're at it, maybe figure out why it takes almost ten seconds for the syntax highlighting to kick in.

kstrauser 8y ago
I agree, although it was more like 2 seconds on my laptop. It was almost unreadable to me until the colors leapt in.
jrk 8y ago
Sorry, poor choice in the default Bootstrap style we adopted years ago. Under normal conditions the styles kick in in a second or less, but I've fixed it nonetheless.

ipsum2 8y ago

There was a paper released recently to do deep learning on Halide[1] that I'm excited to see merge[2] with the main Halide codebase. Seems like the creators of Halide are in this comment thread, does anyone know when this is happening? [1]: https://people.csail.mit.edu/tzumao/gradient_halide/ https://people.csail.mit.edu/tzumao/gradient_halide/ [2]: https://github.com/jrk/gradient-halide https://github.com/jrk/gradient-halide

__abadams__ 8y ago
It works but still needs a bit of cleanup. I was planning to get back to it after an upcoming conference deadline. Here's the PR: https://github.com/halide/Halide/pull/3220 https://github.com/halide/Halide/pull/3220
gammaradiation 8y ago
For deep learning specifically, the TVM project (http://tvm.ai http://tvm.ai) is a substantially more advanced project than Halide, with a much larger community. I’d highly recommend checking that out.
- __abadams__ 8y ago
  You again? The only other time your account has posted was to plug TVM in a different post on Halide. If you're not a Tianqi Chen sockpuppet, then you should apologize to him for making him look like an asshole. For the benefit of others not aware: TVM took Halide, deleted the half of it they didn't understand, then reimplemented it (copy-pasting large chunks without credit, hacks and all) and claimed it was a revolutionary new deep learning IR. People who already knew Halide were left scratching their heads unable to tell how this wasn't just Halide re-marketed in a different community.
- jrk 8y ago
  Good to see the TVM troll back. As ever, it's unclear what "substantially more advanced" means, and to cherry-pick just some of the biggest users, there's a large and growing amount of Halide code in Android, Photoshop, and numerous Google services. I'll let billions of devices and tens of thousands of servers speak for themselves.

keldaris 8y ago

Halide seems like it might have immense potential for fairly general applications in scientific computing, but all of the documentation, tutorials and examples seem dominated by image processing. Could someone elaborate on the limitations of Halide with respect to applications like finite difference and finite element methods, lattice Boltzmann simulations, etc.? These are all ultimately reducible to basic linear algebra and stencil kernels, which seem well suited to the design of Halide. Should computational scientists give it a serious look, or am I missing something?

__abadams__ 8y ago
Halide is generally good for pipelines that do math on multidimensional arrays, so quite possibly. It has a property that I think would make it hard to use for some applications in scientific computing though - while you can mutate in-place, you can't mutate an array ("Func" in Halide) after it has been used by another pipeline stage. This means that the dependency graph between your pipeline stages has no cycles except for self-cycles, which lets us do lots of clever stuff like bounds inference to guarantee the program is correct regardless of the schedule. I can imagine things like flow simulation having a hard time with that constraint. In the deep learning context this means that CNNs are ideal, but RNNs can be awkward or impossible.
- boulos 8y ago
  Andrew: have you considered adding something akin to Rust's lifetimes here? While you'd have restrictions on the schedule, it seems like it's still let you express who in the "graph" messes with the data as tightly as possible.