3 ms·
That high-level idea is viable in many languages. I particularly like Zig's comptime as a clean way to implement it. You often don't want that sort of speciali
by hansvm 3y ago
That high-level idea is viable in many languages. I particularly like Zig's comptime as a clean way to implement it.
You often don't want that sort of specialization. Code like deep neural nets is a pain to handle (unless each layer is identical, which is admittedly moderately common) even with variadic generics and especially without, and it's hard for the compiler to generate the right specializations for (and the right number of specializations; anecdotally, you can expect enough code bloat that it causes problems).
Using branded types you can get a weak form of dependent typing and use that to get features like Tensor<f64, Dim<16>, Dim<MultipleOf<8>>, Dim<SensorChannel>>, where you're allowed to pass in concrete numbers, comptime-known constraints on runtime-known values, or even just weak information like that there's a particular brand (SensorChannel) on the type which allows you to match fully runtime-known values at compile-time and let the type-checker handle it (think of functions like matmul(Mat(A,B), Mat(B,C)), where the B has to be the same for both, but you very well might want to know it at runtime). It's still the library writer's job, unfortunately, to use `inline` and other such keywords appropriately to guide the compiler to the right number of specializations, but the API presented to people using that tensor library can be reasonably nice.
Incidentally, Zig _also_ makes that second thing (branded types) easy to write. I hacked a demo together [0] if you'd like to play around with the concept, but a dedicated tensor library should eventually offer something less verbose IMO.
If you're doing this with ML, my preferred strategy is to go the other way. Only write specializations that are easy for the chip to execute (e.g., appropriately aligned and length-aligned large powers of 2), and build code on top of that. If the computational kernel has to do padding or shifting or other garbage, even at the head/tail of a vector, then something in your model is horribly wrong anyway; just make that unrepresentable.
[0] https://github.com/hmusgrave/pdv https://github.com/hmusgrave/pdv