3 ms·
Disk where? LLM requests are routed dynamically. You might not even land in the same data center.
by slashdave 6mo ago
Disk where? LLM requests are routed dynamically. You might not even land in the same data center.
- FuckButtons 6mo agoBut if you have a tiered cache, then waiting several seconds / minutes is still preferable to getting a cache miss. I suspect the larger problem is the amount of tinkering they are doing with the model makes that not viable.