3 ms·
This is definitely one of those things which sounds like it should be so simple until you actually try to do it, and then run into the hundreds of corner cases
by zaroth 11y ago
This is definitely one of those things which sounds like it should be so simple until you actually try to do it, and then run into the hundreds of corner cases which break only in production during your 3am batch downloads.
In theory, it should be "easy" to scale since each account is fully discrete. More accounts, add a few more "workers" to pull jobs off the queue and everything is happy. Even on the front-end each company is hitting their own private data set, their own indices, etc. So unless you are discovering individual data sets / companies that themselves have a couple orders of magnitude more events than you are prepared to process... Even still, the total number of discrete metrics, and the cardinality of the metrics doesn't change all that much just because event counts themselves (total billings) increase.
So I can definitely see from both perspectives here... why is something like this so hard to scale up? Best answer is because the real world presents challenges well beyond the theoretical complexity of the problem at hand.