3 ms·
> 1. we test many computation graphs (graph is changing constantly) 2. many-many scalar operations with roughly 10k—100k nodes in each graph 3. every graph sh
by 317070 3y ago
> 1. we test many computation graphs (graph is changing constantly)
2. many-many scalar operations with roughly 10k—100k nodes in each graph
3. every graph should be compiled and ran around 10k times both forward and backward
Now I'm curious to know where that is useful. Some kind of meta-learning approach?
- mratsim 3y agoMaybe instruction-level autograd. When you use a differentiable programming language.
- HarHarVeryFunny 3y agoBased on the "we TEST" wording, maybe evaluating graphs for fitness - some sort of genetic/evolutionary approach perhaps?
- dimatura 3y agoAutomatic differentiation isn't just for machine learning, it's useful in several other applications where you have an optimization problem built of many variables interacting in differentiable ways. One possible guess could be structure-from- motion or SLAM, where the graph is often dynamic and several thousands of nodes isn't uncommon (the nodes would be things such as camera poses and 3D landmarks for example). However, in that case there are other frameworks that are built with that scenario in mind (Ceres, GTSAM), and would probably be better baselines.
- charcircuit 3y agoOptimizing something with AD is machine learning.
- kragen 3y agoi want to point out that in many cases you can compute a differentiable relaxation of a nondifferentiable problem to get something you can optimize more easily (with, for example, automatic differentiation) and then use the relaxation to solve the original nondifferentiable problem, for example with branch-and-bound
- arogozhnikov 3y agoRobotic planning of liquid handling. There are many potential alternative ways to achieve the same result, each resulting in its own computation graph. I imagine any kind of trajectory planning for many agents faces similar challenges (e.g. robots in a factory).
- 317070 3y agoBut these computation graphs are similar every time. You would/should write these as a small graph with conditionals on large tensors, not as a fast changing graph of scalars. The latter will always be slower when you have an accelerator like a GPU.
- arogozhnikov 3y ago> But these computation graphs are similar every time. For now I'm testing one-after-another because failure of previous (overflow, too-small-transfer, etc) gives a hint about which part can be changed in the graph. I've been thinking about vectorizing over multiple guesses at once, but don't see a fast reliable way to merge several graphs (also that induces significant burden on other parts of the project - autograd is just one component).