4 ms·
You'd have to hold a ton of connections open for 24 hours+
by willcipriano 3y ago
You'd have to hold a ton of connections open for 24 hours+
- mike_hearn 3y agoThat isn't hard. They aren't going to serve a billion connections are they, and WhatsApp scaled a full NxN low latency messaging system up with a handful of people. At any rate, that's only the case if every request actually sits in the queue for 24 hours. The 24 hour period is clearly chosen to let them absorb free capacity overnight (all such services have diurnal cycles), but unless their implementation is really shoddy at any given moment their workers will be popping work off an internal queue anyway to take advantage of moment-sized lulls in traffic. Done properly you can saturate your hardware 100% all the time... there's certainly enough inferencing demand out there. So at that point the question is just one of external API.