4 ms·

30ms p99 for a cache read! Serverless might have been a problem, but I'm not sure it was the problem. In my experience a p99 of 2ms is more typical - 30ms is th

30ms p99 for a cache read! Serverless might have been a problem, but I'm not sure it was the problem. In my experience a p99 of 2ms is more typical - 30ms is the sort of time I'd expect for p99 on a database query in production serving. You don't need process-local caches to get the sort of performance they're looking for, and there are good reasons why most teams avoid stateful processing, it's much harder to get right and has bad failure modes.

hunterpayne 1y ago
Fun fact, the overhead for a DB query (not including the network latency) is about 250 microseconds (or 0.25ms). And a typical DB kernel can do at least 16MB/core second of throughput. So that 30ms query I would expect is scanning several GB of data to find its results. Try doing that yourself and your latency would probably grow into seconds. Systems programming is hard... PS moving from stateful processing to something with a distributed cache is moving from RAM to network. That's at least a 10x drop off in throughput.