The next hoop to jump through would be to specify tape based autograd in a type system. The type system would not only have to specify the dimensions of the ma

The next hoop to jump through would be to specify tape based autograd in a type system. The type system would not only have to specify the dimensions of the matrices, but the sequence of operations as well (in order to reverse them and perform the correct backprop sequence without relying on the runtime).