3 ms·
I would say that integrating AD at the level of LLVM IR is deep compiler integration. The AD has to consume IR and then emit more IR to actually take the deriva
by oivey 6y ago
I would say that integrating AD at the level of LLVM IR is deep compiler integration. The AD has to consume IR and then emit more IR to actually take the derivatives. If you’re using a Python wrapper for this you’ll need codegen for interacting with the generated IR, too. The upside with that approach is that it might more easily work across languages. The downside is that LLVM IR can’t rely on the rich invariants that even C exposes, which makes codegen and optimization harder.
An API is always going to be necessarily behind state-of-the-art because often research depends on inventing things that don’t fit within an existing API.
- sanxiyn 6y agoI think we are talking past each other. sillysaurusx is talking about OpenGL/Direct3D as a portability layer for hardwares. API vs AD is not a hardware portability layer concern. The proposal is to standardize OpenGL/Direct3D-like hardware portability layer defined by API. API vs AD you are talking about happens far above that and it is irrelevant.
- oivey 6y agoToday, things like PyTorch are AD systems, optimization systems, libraries of high performance interoperable common blocks, and more. This massive complexity is why there is no standard. If you have a compiler front end that can AD arbitrary code, then the framework no longer has to worry about that. If you have a compiler backend that can target multiple devices, then the framework doesn’t have to do that, either. The size and complexity of PyTorch/TensorFlow is exactly why there isn’t a standard hardware portable NN API. A sane API of hardware independent common blocks can’t happen without all the front and backend compiler stuff first. A PyTorch/TensorFlow-like API is always going to be massive, complicated, and hard to port to new hardware targets. Additionally, the complexity and reliance on C alone will make integrating exotic new concepts like say, differential equation solvers, extremely laborious.