4 ms·

Part of the motivation for SymForce was in fact serious attempts (multiple months) by us to do similar things with TensorFlow XLA and being stymied by some of t

Part of the motivation for SymForce was in fact serious attempts (multiple months) by us to do similar things with TensorFlow XLA and being stymied by some of these effects. Many of the problems we deal with, and many problems in robotics in general, deal with small "tensor sizes", where dimensions of individual variables are often <10, and batch sizes aren't huge either. SymForce-generated C++ code is typically one machine instruction per scalar per "op" - compiling TF / JAX etc using methods like you mention definitely helps a ton, but it doesn't reach that level of efficiency for small problems. And computing Hessians for second-order optimization tends to blow up op counts even more. Similarly, SymForce examines sparsity of individual matrices and jacobians at code generation time, so that operations with sparse matrices are similarly one instruction per op - at runtime there's no looping or indirection through sparse data structures or allocations of sparse matrices in the optimization loop. There's definitely a cross-over point where dimensions become large enough that TF / JAX / etc will be the tool to use - we'd love to provide some more benchmarks and comparisons here. For the first two experiments in the paper (see Tables II and III), we do compare to JAX and amortizing across a batch size of 1000, SymForce is still way ahead. We tried to be careful, but the code is available so please come beat up our numbers.

jxy 4y ago
> amortizing across a batch size of 1000 This put the problem outside of L2 and thus is not a fair comparison, especially if your problem is small enough to fit in L1. In addition, XLA does a poor job in optimizing for CPU cache reuse, so I'm not surprised that directly transcoded C++ code would be much faster than XLA. However, TensorFlow code can directly run on GPU. SymForce cannot.
- haykmartiros 4y ago
  To be clear, we tested a large set of batch sizes from 1 up to ~1e8. The smaller the batch size, the less competitive JAX was. The crossover point for JAX to be faster was amortizing across a batch size in the tens of thousands I believe. The ideal batch size for some of our applications is one to five. Please see symforce/benchmarks/notebooks for some of our timing. SymForce is not trying to compute on large data-parallelism, but it could generate an inner kernel for that. In that way it complements TensorFlow nicely.
- crubier 4y ago
  > TensorFlow code can directly run on GPU. SymForce cannot. I think this is just a matter of tweaking the code generator, and not a hard limitation of Symforce.

jxy 4y ago

> amortizing across a batch size of 1000 This put the problem outside of L2 and thus is not a fair comparison, especially if your problem is small enough to fit in L1. In addition, XLA does a poor job in optimizing for CPU cache reuse, so I'm not surprised that directly transcoded C++ code would be much faster than XLA. However, TensorFlow code can directly run on GPU. SymForce cannot.

haykmartiros 4y ago
To be clear, we tested a large set of batch sizes from 1 up to ~1e8. The smaller the batch size, the less competitive JAX was. The crossover point for JAX to be faster was amortizing across a batch size in the tens of thousands I believe. The ideal batch size for some of our applications is one to five. Please see symforce/benchmarks/notebooks for some of our timing. SymForce is not trying to compute on large data-parallelism, but it could generate an inner kernel for that. In that way it complements TensorFlow nicely.
crubier 4y ago
> TensorFlow code can directly run on GPU. SymForce cannot. I think this is just a matter of tweaking the code generator, and not a hard limitation of Symforce.