3 ms·

Julia makes this substantially easier, but yes in many cases you might need to come up with a cache-optimized tensor operation kernel. The Julia Lab is building

Julia makes this substantially easier, but yes in many cases you might need to come up with a cache-optimized tensor operation kernel. The Julia Lab is building some tooling for the automatic construction of this stuff which is compatible with the AD frameworks, kind of like Halide, but in the Julia language and compatible with all of its generics. The real key though is that packages play nicely in the Julia sphere, so if someone writes a good Julia code for the operation and puts a package up, then you can take that package and utilize it in Flux even if the person did not write for ML purposes. There's a lot of quantum particle folks writing esoteric tensor operations in Julia that I personally have been pulling stuff from.

cs702 7y ago
> ...in many cases you might need to come up with a cache-optimized tensor operation kernel... Yes, exactly. That's one of the issues raised by the authors of the paper. They note that in practice, most AI researchers will not do that. They iterate and test code very quickly and cannot invest the time/effort necessary (a) to figure out how to write a cache-optimized kernel every time they might need one, or even (b) wait for an automated kernel-writing tool to finish searching for and compiling an optimized kernel (which in practice often ends up being slower than manipulating the data in inefficient ways in order to use existing, inflexible, prebuilt kernels).
- ChrisRackauckas 7y ago
  >They note that in practice, most AI researchers will not do that. I think the main issue there is that these kernel compilers are not well-integrated into the ML libraries. If it's a standard part of it, well-documented, easy to build (usually the difficult part is compiling someone's custom compiler...), and if the compiled results can be "stored" for future ML models to just use, I think it would see more adoption. As of now, tensor compilers for ML frameworks are more of a (good but shaky) research tool. I think a Julia approach of "do it on Julia code" (even the compiler, so it's easy to just ]add the package) could definitely break down some of these barriers to getting it done in a way that garners widespread adoption, but a lot of work will need to be done in order to get a good enough tensor compiler for people to care. If anything, this is a fun space with many opportunities.
  - cs702 7y ago
    Clearly, that would help. And clearly, there are opportunities for improvement -- as much as two orders of magnitude (!) in the case of capsule networks. That said, I think the authors are right about the challenge here: "...current frameworks excel at workloads where it makes sense to manually tune the small set of computations used by a particular model or family of models. Unfortunately, frameworks become poorly suited to research, because there is a performance cliff when experimenting with computations that haven’t previously been identified as important. While a few hours of search may be acceptable before production deployment, it is unrealistic to expect researchers to put up with such compilation times (recall this is just one kernel in what may be a large overall computation); and even if optimized kernels were routinely cached locally, it would be a major barrier to disseminating research if anyone who downloaded a model’s source code had to spend hours or days compiling it for their hardware before being able to experiment with it." EDIT: removed third paragraph.
- improbable22 7y ago
  I guess "easier" is a relative term; welding something together in your garage is never going to be as easy & efficient as buying the Toyota... but what the binding constraints on the field's progress are I don't claim to know. But since you seem to understand the paper a bit, may I ask: I thought convolution was done by FFT, shaving N^2 to NlogN and all that. Are these automated compilers they discuss smart enough to find that, or are they literally re-arranging (1)? Or is it all so memory-bound that this question makes no sense?
  - ChrisRackauckas 7y ago
    It's all so memory-bound that it really comes down to how you structure the computation.

cs702 7y ago

> ...in many cases you might need to come up with a cache-optimized tensor operation kernel... Yes, exactly. That's one of the issues raised by the authors of the paper. They note that in practice, most AI researchers will not do that. They iterate and test code very quickly and cannot invest the time/effort necessary (a) to figure out how to write a cache-optimized kernel every time they might need one, or even (b) wait for an automated kernel-writing tool to finish searching for and compiling an optimized kernel (which in practice often ends up being slower than manipulating the data in inefficient ways in order to use existing, inflexible, prebuilt kernels).

ChrisRackauckas 7y ago
>They note that in practice, most AI researchers will not do that. I think the main issue there is that these kernel compilers are not well-integrated into the ML libraries. If it's a standard part of it, well-documented, easy to build (usually the difficult part is compiling someone's custom compiler...), and if the compiled results can be "stored" for future ML models to just use, I think it would see more adoption. As of now, tensor compilers for ML frameworks are more of a (good but shaky) research tool. I think a Julia approach of "do it on Julia code" (even the compiler, so it's easy to just ]add the package) could definitely break down some of these barriers to getting it done in a way that garners widespread adoption, but a lot of work will need to be done in order to get a good enough tensor compiler for people to care. If anything, this is a fun space with many opportunities.
- cs702 7y ago
  Clearly, that would help. And clearly, there are opportunities for improvement -- as much as two orders of magnitude (!) in the case of capsule networks. That said, I think the authors are right about the challenge here: "...current frameworks excel at workloads where it makes sense to manually tune the small set of computations used by a particular model or family of models. Unfortunately, frameworks become poorly suited to research, because there is a performance cliff when experimenting with computations that haven’t previously been identified as important. While a few hours of search may be acceptable before production deployment, it is unrealistic to expect researchers to put up with such compilation times (recall this is just one kernel in what may be a large overall computation); and even if optimized kernels were routinely cached locally, it would be a major barrier to disseminating research if anyone who downloaded a model’s source code had to spend hours or days compiling it for their hardware before being able to experiment with it." EDIT: removed third paragraph.
improbable22 7y ago
I guess "easier" is a relative term; welding something together in your garage is never going to be as easy & efficient as buying the Toyota... but what the binding constraints on the field's progress are I don't claim to know. But since you seem to understand the paper a bit, may I ask: I thought convolution was done by FFT, shaving N^2 to NlogN and all that. Are these automated compilers they discuss smart enough to find that, or are they literally re-arranging (1)? Or is it all so memory-bound that this question makes no sense?
- ChrisRackauckas 7y ago
  It's all so memory-bound that it really comes down to how you structure the computation.