3 ms·

I agree with the parent poster. If you had a separate loosely coupled Stripe importer system you could process those Stripe event streams into the required aggr

I agree with the parent poster. If you had a separate loosely coupled Stripe importer system you could process those Stripe event streams into the required aggregates via a queue. That way you save your precious web server resources for processing real-time web requests without the performance hit. We churned off BareMetrics because of those server and performance issues a while back.

callmeed 11y ago
>> If you had a separate loosely coupled Stripe importer system you could process those Stripe event streams into the required aggregates via a queue. Sorry, but it's not quite that easy. Stripe has rate-limiting and paginates response data. Providing metrics like "churn growth rate" between 2 dates requires you have all the data every day. You can somewhat rely on webhooks but not for historical data prior to a customer's sign-up date. Oh, and Baremetrics has plenty of queues. (Source: former Baremetrics engineer)
- zaroth 11y ago
  This is definitely one of those things which sounds like it should be so simple until you actually try to do it, and then run into the hundreds of corner cases which break only in production during your 3am batch downloads. In theory, it should be "easy" to scale since each account is fully discrete. More accounts, add a few more "workers" to pull jobs off the queue and everything is happy. Even on the front-end each company is hitting their own private data set, their own indices, etc. So unless you are discovering individual data sets / companies that themselves have a couple orders of magnitude more events than you are prepared to process... Even still, the total number of discrete metrics, and the cardinality of the metrics doesn't change all that much just because event counts themselves (total billings) increase. So I can definitely see from both perspectives here... why is something like this so hard to scale up? Best answer is because the real world presents challenges well beyond the theoretical complexity of the problem at hand.

callmeed 11y ago

>> If you had a separate loosely coupled Stripe importer system you could process those Stripe event streams into the required aggregates via a queue. Sorry, but it's not quite that easy. Stripe has rate-limiting and paginates response data. Providing metrics like "churn growth rate" between 2 dates requires you have all the data every day. You can somewhat rely on webhooks but not for historical data prior to a customer's sign-up date. Oh, and Baremetrics has plenty of queues. (Source: former Baremetrics engineer)

zaroth 11y ago
This is definitely one of those things which sounds like it should be so simple until you actually try to do it, and then run into the hundreds of corner cases which break only in production during your 3am batch downloads. In theory, it should be "easy" to scale since each account is fully discrete. More accounts, add a few more "workers" to pull jobs off the queue and everything is happy. Even on the front-end each company is hitting their own private data set, their own indices, etc. So unless you are discovering individual data sets / companies that themselves have a couple orders of magnitude more events than you are prepared to process... Even still, the total number of discrete metrics, and the cardinality of the metrics doesn't change all that much just because event counts themselves (total billings) increase. So I can definitely see from both perspectives here... why is something like this so hard to scale up? Best answer is because the real world presents challenges well beyond the theoretical complexity of the problem at hand.