7 ms·

Isn't this the kind of application where Julia would shine? If you're trying to develop something new or cutting edge it makes sense that frameworks don't nece

by Tarq0n 7y ago

Isn't this the kind of application where Julia would shine? If you're trying to develop something new or cutting edge it makes sense that frameworks don't necessarily suit you. Frameworks try to make common actions easy (provide a 'happy path'), very much the opposite of developing something new. Something like Julia which offers useful primitives, but can also let you compile general purpose code for CUDA seems like a better fit, rather than expecting frameworks to cater to you.

cs702 7y ago
The authors of the paper discuss how Julia fares with capsule networks in section 4.1:[a] > There are frameworks, such as Julia, which nominally use the same language to represent both the graph of operators and their implementations, but back-end designs can diminish the effectiveness of such a front end. In Julia, while 2D convolution is provided as a native Julia library, there is an overloaded conv2d function for GPU inputs which calls NVidia’s cuDNN kernel. Bypassing this custom implementation in favor of the generic code essentially hits a “not implemented” case and falls back to a path that is many orders of magnitude slower. [a] The paper uses capsule networks as its main example. See my earlier comment for context and a link to the paper: https://news.ycombinator.com/item?id=20304576 https://news.ycombinator.com/item?id=20304576
- ChrisRackauckas 7y ago
  Julia makes this substantially easier, but yes in many cases you might need to come up with a cache-optimized tensor operation kernel. The Julia Lab is building some tooling for the automatic construction of this stuff which is compatible with the AD frameworks, kind of like Halide, but in the Julia language and compatible with all of its generics. The real key though is that packages play nicely in the Julia sphere, so if someone writes a good Julia code for the operation and puts a package up, then you can take that package and utilize it in Flux even if the person did not write for ML purposes. There's a lot of quantum particle folks writing esoteric tensor operations in Julia that I personally have been pulling stuff from.
  - cs702 7y ago
    > ...in many cases you might need to come up with a cache-optimized tensor operation kernel... Yes, exactly. That's one of the issues raised by the authors of the paper. They note that in practice, most AI researchers will not do that. They iterate and test code very quickly and cannot invest the time/effort necessary (a) to figure out how to write a cache-optimized kernel every time they might need one, or even (b) wait for an automated kernel-writing tool to finish searching for and compiling an optimized kernel (which in practice often ends up being slower than manipulating the data in inefficient ways in order to use existing, inflexible, prebuilt kernels).
    - ChrisRackauckas 7y ago
      >They note that in practice, most AI researchers will not do that. I think the main issue there is that these kernel compilers are not well-integrated into the ML libraries. If it's a standard part of it, well-documented, easy to build (usually the difficult part is compiling someone's custom compiler...), and if the compiled results can be "stored" for future ML models to just use, I think it would see more adoption. As of now, tensor compilers for ML frameworks are more of a (good but shaky) research tool. I think a Julia approach of "do it on Julia code" (even the compiler, so it's easy to just ]add the package) could definitely break down some of these barriers to getting it done in a way that garners widespread adoption, but a lot of work will need to be done in order to get a good enough tensor compiler for people to care. If anything, this is a fun space with many opportunities.
      - cs702 7y ago
        Clearly, that would help. And clearly, there are opportunities for improvement -- as much as two orders of magnitude (!) in the case of capsule networks. That said, I think the authors are right about the challenge here: "...current frameworks excel at workloads where it makes sense to manually tune the small set of computations used by a particular model or family of models. Unfortunately, frameworks become poorly suited to research, because there is a performance cliff when experimenting with computations that haven’t previously been identified as important. While a few hours of search may be acceptable before production deployment, it is unrealistic to expect researchers to put up with such compilation times (recall this is just one kernel in what may be a large overall computation); and even if optimized kernels were routinely cached locally, it would be a major barrier to disseminating research if anyone who downloaded a model’s source code had to spend hours or days compiling it for their hardware before being able to experiment with it." EDIT: removed third paragraph.
- UncleOxidant 7y ago
  Maybe I'm not understanding the problem they hit, but couldn't they just modify conv2d that calls cuDNN to do what they want it to do? In Julia even CUDA kernals could be written in Julia. I realize that CUDA & CuDNN programming is not easy, but perhaps given Julia's powerful macro system and compiler tools this could be automated to some extent?
  - cameldrv 7y ago
    Part of the problem is that CuDNN is closed source. If CuDNN almost does what you want, you have to reimplement it, and your implementation will not be as fast as NVIDIAs because they have put a lot of work into it and also understand the subtleties of their GPU's architecture better. A lot of performance details of NVIDIAs GPUs are not documented.
    - ChrisRackauckas 7y ago
      And the other problem is that tensor kernels are just a bear to optimize. Even on the CPU, if you write a 3 loop implementation of matrix multiplication you're leaving behind a ton of performance. You want to use a blocked algorithm which specifically knows your cache sizes, and it comes out to like a 12 nested loop implementation which can be architecture-dependent and maybe have a tiny bit of assembly magic in the middle. That's just a CPU, and that's just a dense matmul. This is why things like Halide exist for trying to generate optimized kernels to things like stencil operations, since "just write the loop" really isn't a good strategy. Couple that with the fact that you need to specialize it to the architecture, and indeed a lot of it isn't documented, and you have a hard time getting something as fast as CuDNN. If you haven't seen this stuff, take a look at: https://www.cs.cmu.edu/afs/cs/project/pscico-guyb/realworld/www/slidesF10/FLP99.pdf https://www.cs.cmu.edu/afs/cs/project/pscico-guyb/realworld/... . That's cache-oblivious, so it still leaves some performance on the table (which is why BLAS doesn't use a cache-oblivious form), but gets the point across that you can do a ton of algorithmic work here. What Julia gives you is something where you can directly write the loops and compile to CUDA kernels, and it will run fast. The issue there isn't making the CUDA kernels, but coming up with the algorithm. CuDNN is proprietary so you can't just copy their tensor kernels into Julia. The Julia Lab has some individuals interested in building tensor compilers for this case though, where you can abstractly define such an operation and it will try to come up with a fast version. And then hopefully build binaries from it so it's "compile once"? But this stuff is just hard and there's no easy answer, though Julia is probably the best approach for building an answer IMO :).

by Tarq0n 7y ago

cs702 7y ago

The authors of the paper discuss how Julia fares with capsule networks in section 4.1:[a] > There are frameworks, such as Julia, which nominally use the same language to represent both the graph of operators and their implementations, but back-end designs can diminish the effectiveness of such a front end. In Julia, while 2D convolution is provided as a native Julia library, there is an overloaded conv2d function for GPU inputs which calls NVidia’s cuDNN kernel. Bypassing this custom implementation in favor of the generic code essentially hits a “not implemented” case and falls back to a path that is many orders of magnitude slower. [a] The paper uses capsule networks as its main example. See my earlier comment for context and a link to the paper: https://news.ycombinator.com/item?id=20304576 https://news.ycombinator.com/item?id=20304576

ChrisRackauckas 7y ago
Julia makes this substantially easier, but yes in many cases you might need to come up with a cache-optimized tensor operation kernel. The Julia Lab is building some tooling for the automatic construction of this stuff which is compatible with the AD frameworks, kind of like Halide, but in the Julia language and compatible with all of its generics. The real key though is that packages play nicely in the Julia sphere, so if someone writes a good Julia code for the operation and puts a package up, then you can take that package and utilize it in Flux even if the person did not write for ML purposes. There's a lot of quantum particle folks writing esoteric tensor operations in Julia that I personally have been pulling stuff from.
- cs702 7y ago
  > ...in many cases you might need to come up with a cache-optimized tensor operation kernel... Yes, exactly. That's one of the issues raised by the authors of the paper. They note that in practice, most AI researchers will not do that. They iterate and test code very quickly and cannot invest the time/effort necessary (a) to figure out how to write a cache-optimized kernel every time they might need one, or even (b) wait for an automated kernel-writing tool to finish searching for and compiling an optimized kernel (which in practice often ends up being slower than manipulating the data in inefficient ways in order to use existing, inflexible, prebuilt kernels).
  - ChrisRackauckas 7y ago
    >They note that in practice, most AI researchers will not do that. I think the main issue there is that these kernel compilers are not well-integrated into the ML libraries. If it's a standard part of it, well-documented, easy to build (usually the difficult part is compiling someone's custom compiler...), and if the compiled results can be "stored" for future ML models to just use, I think it would see more adoption. As of now, tensor compilers for ML frameworks are more of a (good but shaky) research tool. I think a Julia approach of "do it on Julia code" (even the compiler, so it's easy to just ]add the package) could definitely break down some of these barriers to getting it done in a way that garners widespread adoption, but a lot of work will need to be done in order to get a good enough tensor compiler for people to care. If anything, this is a fun space with many opportunities.
    - cs702 7y ago
      Clearly, that would help. And clearly, there are opportunities for improvement -- as much as two orders of magnitude (!) in the case of capsule networks. That said, I think the authors are right about the challenge here: "...current frameworks excel at workloads where it makes sense to manually tune the small set of computations used by a particular model or family of models. Unfortunately, frameworks become poorly suited to research, because there is a performance cliff when experimenting with computations that haven’t previously been identified as important. While a few hours of search may be acceptable before production deployment, it is unrealistic to expect researchers to put up with such compilation times (recall this is just one kernel in what may be a large overall computation); and even if optimized kernels were routinely cached locally, it would be a major barrier to disseminating research if anyone who downloaded a model’s source code had to spend hours or days compiling it for their hardware before being able to experiment with it." EDIT: removed third paragraph.
UncleOxidant 7y ago
Maybe I'm not understanding the problem they hit, but couldn't they just modify conv2d that calls cuDNN to do what they want it to do? In Julia even CUDA kernals could be written in Julia. I realize that CUDA & CuDNN programming is not easy, but perhaps given Julia's powerful macro system and compiler tools this could be automated to some extent?
- cameldrv 7y ago
  Part of the problem is that CuDNN is closed source. If CuDNN almost does what you want, you have to reimplement it, and your implementation will not be as fast as NVIDIAs because they have put a lot of work into it and also understand the subtleties of their GPU's architecture better. A lot of performance details of NVIDIAs GPUs are not documented.
  - ChrisRackauckas 7y ago
    And the other problem is that tensor kernels are just a bear to optimize. Even on the CPU, if you write a 3 loop implementation of matrix multiplication you're leaving behind a ton of performance. You want to use a blocked algorithm which specifically knows your cache sizes, and it comes out to like a 12 nested loop implementation which can be architecture-dependent and maybe have a tiny bit of assembly magic in the middle. That's just a CPU, and that's just a dense matmul. This is why things like Halide exist for trying to generate optimized kernels to things like stencil operations, since "just write the loop" really isn't a good strategy. Couple that with the fact that you need to specialize it to the architecture, and indeed a lot of it isn't documented, and you have a hard time getting something as fast as CuDNN. If you haven't seen this stuff, take a look at: https://www.cs.cmu.edu/afs/cs/project/pscico-guyb/realworld/www/slidesF10/FLP99.pdf https://www.cs.cmu.edu/afs/cs/project/pscico-guyb/realworld/... . That's cache-oblivious, so it still leaves some performance on the table (which is why BLAS doesn't use a cache-oblivious form), but gets the point across that you can do a ton of algorithmic work here. What Julia gives you is something where you can directly write the loops and compile to CUDA kernels, and it will run fast. The issue there isn't making the CUDA kernels, but coming up with the algorithm. CuDNN is proprietary so you can't just copy their tensor kernels into Julia. The Julia Lab has some individuals interested in building tensor compilers for this case though, where you can abstractly define such an operation and it will try to come up with a fast version. And then hopefully build binaries from it so it's "compile once"? But this stuff is just hard and there's no easy answer, though Julia is probably the best approach for building an answer IMO :).