3 ms·
Loading assets JIT for the next frame from NVMe hasn't been a thing for 20 years though. Different kinds of latency floors. What I was asking is whether LLM in
by the8472 3y ago
Loading assets JIT for the next frame from NVMe hasn't been a thing for 20 years though. Different kinds of latency floors.
What I was asking is whether LLM inference can be structured in such a way that only a fraction of the weight is needed at a time and then the next ones can be loaded JIT as the processing pipeline advances.