I haven't used these Buckets nor have I scaled anything to that level with any need for uptime guarantees, so I'm certainly really naive. But at the same time,

I haven't used these Buckets nor have I scaled anything to that level with any need for uptime guarantees, so I'm certainly really naive. But at the same time, simplifying the interfaces seems to me to (a) provide less points for buggy failure (b) allow you to write lots of machine learning methods that all "just work" on buckets. It might all be custom, but if you're spending lots of time building, testing, analyzing, and deploying statistical models from scratch (the place I do have expertise) then a simple, consistent data interface is godsend. Scaling might still be custom, but if you can expose the Bucket interface then it "doesn't matter" what happens underneath. If there's some real bottlenecking then it's probably important to break through that interface or to expand it (like that flush-rate parameter the Op mentions). Until there is though, you're winning on experimental fluidity.