3 ms·
The dynamic budget negotiation idea is interesting. One thing I've found working with on-device AI models is that the "budget" isn't just about what the device
by ashishdhiman23 7mo ago
The dynamic budget negotiation idea is interesting. One thing I've found working with on-device AI models is that the "budget" isn't just about what the device can handle in theory — it shifts meaningfully between runs on the same hardware.
Ran MobileNetV2 100 times on a Snapdragon 8 Gen 3 and got 83% latency spread. Cold start was 2.689ms, steady state median was 0.369ms. The first couple of runs are basically a different device thermally.
Curious how Composite Budget Contracts handle that kind of within-session variance. Do you re-negotiate after warmup or is the contract set once at initialization?