5 ms·
I can concur. Most of Uber's traffic for real-time analytics was served by Uber's Elasticsearch clusters with customized querying layer. My team was also a user
by hintymad 6y ago
I can concur. Most of Uber's traffic for real-time analytics was served by Uber's Elasticsearch clusters with customized querying layer. My team was also a user of the ES service. The service was sometimes unstable, but in general was fast enough for us, and scalability was never a problem. My team got paged only once from the ES team because we drove a huge spike to the team, and they ended up scanning more than 500 million records per second, for which they had to scramble to scale out. But even at that time, there was no visible degradation on our side, let alone outage. The owning team also presented their pain points, and none of the pains had to do with CPU not being fast enough.
By the way, Uber already had two real-time analytics systems before AresDB. One is the aforementioned ES-based service, and the other is Pinot, which was owned by a Pinot contributor. I was in one of those so-called alignment meetings about using AresDB. Engineers from both the ES service and Pinot were there. It was a disaster. The engineers simply asked what AresDB was supposed to solve, and presented charts over charts to show that computation or lack of join operator was never a problem (because for analytics, data streams can be pre-joined), while efficient IO was. The AresDB team simply repeated that join was important, and parallel computation was critical.
I left the company soon after, so I'm not sure how many critical use cases AresDB has been serving since then. Hopefully they do find some sweet spot to justify the cost of developing such system.
- alquemist 6y agoNot a big fan of AresDB because of the glaring IO issues. That being said, pushing 'lack of join operator was never a problem' is textbook Sapir–Whorf. 'We've trained our captive users to never ask a question that requires a join, or use convoluted workarounds if they absolutely must do so, and we are proud of it' just leaves a bad taste. I wouldn't be surprised if Pinot were to add join support in the near future...
- hintymad 6y agoTrue that join is nice. I was just not sure if real-time join is truly necessary for Uber’s use cases, or AresDB is so much better than a query engine like Presto to justify its effort. Or more fundamentally, is GPU the solution to expensive joins?
- deleted 6y ago[deleted]
- ambrood 6y agoLove the shoutout to Gairos. But man that on-call was a pain in the arse.