3 ms·
When people say that you "can't do" something what they actually mean is that it's completely impractical (if not impossible).
by dns_snek 5mo ago
When people say that you "can't do" something what they actually mean is that it's completely impractical (if not impossible).
- zozbot234 5mo agoWhether something is "impractical" depends on your expectations. High-latency unattended inference is definitely viable, even though it doesn't align much with what's being run in hyperscale datacenters.
- dns_snek 5mo agoI'd like to meet the person who's been using a 1 token/second system as their primary LLM for at least a few weeks. Anyone? I think 1 token/second is optimistic here - and even then it's over 11 days per million tokens.