4 ms·
That has to be cold-start, and next N requests would surely be using the already started thing? It sounds bananas they'd even mention using something like that
by diggan 1y ago
That has to be cold-start, and next N requests would surely be using the already started thing? It sounds bananas they'd even mention using something like that with 19 seconds latency for all requests in any context.
- agcat 1y agoThat's true. Traditional single-tier storage can not meet the throughput and latency demand. My cofounder wrote this piece on a three-tiered storage architecture optimized for both performance and cost - https://nilesh-agarwal.com/three-tier-storage-architecture-for-fast-llm-inference-in-the-cloud/ https://nilesh-agarwal.com/three-tier-storage-architecture-f...