3 ms·
Whether something is "impractical" depends on your expectations. High-latency unattended inference is definitely viable, even though it doesn't align much with
by zozbot234 5mo ago
Whether something is "impractical" depends on your expectations. High-latency unattended inference is definitely viable, even though it doesn't align much with what's being run in hyperscale datacenters.
- dns_snek 5mo agoI'd like to meet the person who's been using a 1 token/second system as their primary LLM for at least a few weeks. Anyone? I think 1 token/second is optimistic here - and even then it's over 11 days per million tokens.