6 ms·
the only real benefit is privacy which 99.9% of people dont get about. Almost all serving metrics (cost, throughput, ttft) are better with large gpu clusters. L
by eldenring 10mo ago
the only real benefit is privacy which 99.9% of people dont get about. Almost all serving metrics (cost, throughput, ttft) are better with large gpu clusters. Latency is usually hidden by prefill cost.
- cowpig 10mo agoMore and more people I talk to care about privacy, but not in SF
- mistercheph 10mo agoand sovereignty. I can go into the woods with a fuzzy approximation of all internet text in my backpack