3 ms·
It all depends how many models you are serving from that flash and whether they all fit in there together. If you need to evict and load models, the flash will
by rbanffy 2mo ago
It all depends how many models you are serving from that flash and whether they all fit in there together. If you need to evict and load models, the flash will die a horrible death. This is one resource you should NEVER underprovision.
- rando1234 2mo agoThis is a game changer for local inference IMO, where you will basically never need to do this. On the other hand in a serverless/cloud setting it may be more problematic if you don't want to strand GPUs with locally attached HBF.