13 ms·
AresDB: Uber’s GPU-Powered Open-Source, Real-Time Analytics Engine
- njn 8y agoSometimes I feel like CTO's jobs are just to sound smart, come up with weird ideas, and make graphs like this that tickle the intellects of their company's shareholders.
- gristle 8y agoWhat? This is a really interesting technical design post outlining how they use technology to solve real problems at scale.
- deleted 8y ago[deleted]
- coderintherye 8y agoWell, it does depend company to company, but in a lot of places the role is or includes coming up with innovative ideas and to be smart in choosing which ones to implement.
- sfkeller 8y agoDid someone say innovative database? PG-Strom with PostgreSQL under the hood (what else?) exists since over 4 years: https://github.com/heterodb/pg-strom https://github.com/heterodb/pg-strom
- arnon 8y agoSQream DB - around since 2010 MapD (now OmniSci) - around since 2013 Kinetica - around since 2009 ...
- based2 8y agohttps://github.com/uber/aresdb https://github.com/uber/aresdb
- kerng 8y agoInteresting post with lots of architectural details. I like how they highlight the different existing things they tried and what the shortcomings were which led to building this, although not sure where I could use this to solve a problem.
- samaparicio 8y agoLiked this post. Would love more detail on GPU query offloading and on how different the engine looks compared to one that would run queries on the CPU, as that seems to be the primary innovation
- dragontamer 8y agoThis article has a great overview, but it is dense and uses a lot of unique terminlology: https://moderngpu.github.io/join.html https://moderngpu.github.io/join.html In many ways, this paper is easier to read instead: http://www.csee.usf.edu/~tuy/pub/SSDBM17.pdf http://www.csee.usf.edu/~tuy/pub/SSDBM17.pdf But I'm pretty sure the "moderngpu" version of merge-join is more optimal.
- jclay 8y agoInteresting. MapD and this project both use Thrust under the hood and from what I gather, both attempting to address the same issue. Can anyone speak to the differences? While I originally didn’t get the case for GPU accelerated databases, it makes more and more sense given that the bandwidth speeds between GPU and CPU are steadily increasing, making the GPU an increasingly attractive option since the latency for GPU<->CPU syncs is diminishing.
- polskibus 8y agoMapD is now called OmniSci and is mentioned in Uber article. I bet Uber had a good look at MapD internals.
- tmostak 8y agoThe MapD (now OmniSci) execution engine is actually built around a JIT compiler that translates SQL to LLVM IR for most operations, but it does use Thrust for a few things like sort. One performance advantage of using a JIT is that the system can fuse operators (like filter, group by and aggregate, and project) into one kernel without the overhead of calling multiple kernels (and particularly the overhead of materializing intermediate results, versus keeping these results in registers). Outputting LLVM IR also makes it easy for us run on CPU as well by for the most part bypassing the need to generate GPU specific code (for example, in CUDA). It does make coding on the system a bit more difficult, but we found that the performance gains were worth it. Also I would add that this system seems, at least for now, geared toward solving a specific set of problems Uber is grappling with around time-series data, whereas OmniSci is trying to tackle general purpose OLAP/analytics problems. Not that solving specific use cases is not great/valid, or that they don't plan to expand the functionality of AresDB over time to encompass broader use cases.
- yayr 8y agoUber's article mentions: "...but as we evaluated the product, we realized that it did not have critical features for Uber’s use case, such as deduplication." however, I also see, they used go quite intensively. Maybe, they also did it for the productivity gains while implementing their additional features
- twotwotwo 8y agoAny good resources on where GPU DBs offer significant wins? Especially, but not only, if it wins in cost efficiency for some workload. My naïve impression working at smaller scales is that in a reasonably balanced system, storage is often the bottleneck, and if it can deliver data fast enough the CPU probably won't break a sweat counting or summing or so on. The amount of interest in GPU databases, though, suggests that's not the case for some interesting workloads.
- dominotw 8y agoCMU has a lecture series on youtube on GPU databases. https://db.cs.cmu.edu/seminar2018/ https://db.cs.cmu.edu/seminar2018/
- scribu 8y agoThis page [1] lists query times on a 1.1 billion row dataset, performed using a variety of systems, from GPU DBs all the way to SQLite. [1] https://tech.marksblogg.com/benchmarks.html https://tech.marksblogg.com/benchmarks.html Edit: Unfortunately, each run is on different hardware, but it at least gives you an idea of what's possible.
- walterbell 8y agoAnyone used this GPU database that ranks at the top of that list, above the open-source MapD? https://www.brytlyt.com https://www.brytlyt.com
- bufferoverflow 8y agoNote that it's at the top of the list probably because it's running on a cluster. It would be awesome to see such a comparison on some standard hardware, like a large AWS GPU instance (eg1.2xlarge). Also note that the dataset is 600GB, so it won't fit a sinlge GPU, not even close.
- datumdadum 8y agoHaven't, but it's worth noting that hardware is probably attributable to them edging out Mapd since they're on a 5-node minsky cluster featuring nvlink, hence as arnon said, are benefiting from 9.5x faster transfer from disk than than PCIe 3.0. That blog has not yet tested Mapd's IBM Power version-- would be interesting to see how it compared on that cluster. CMU has an interesting lecture if you want to learn more about Brytlyt https://www.youtube.com/watch?v=oL0IIMQjFrs https://www.youtube.com/watch?v=oL0IIMQjFrs Couple interesting things to note: - They attain some of their speed by requiring that data be pre-sorted https://youtu.be/oL0IIMQjFrs?t=1092 https://youtu.be/oL0IIMQjFrs?t=1092 https://youtu.be/oL0IIMQjFrs?t=1127 https://youtu.be/oL0IIMQjFrs?t=1127 - They've built their database on Postgres for query planning, but for any query which does not match what they've accelerated on GPU, they do not have the ability to failover to utilizing postgres on the CPU. https://youtu.be/oL0IIMQjFrs?t=3260 https://youtu.be/oL0IIMQjFrs?t=3260 - Data is brought into GPU memory at table CREATE time, so the cost of transferring data from disk->host RAM->GPU RAM is not reflected. Probably wouldn't work if you want to shuffle data in/out of GPU RAM across changing query workloads. https://youtu.be/oL0IIMQjFrs?t=1310 https://youtu.be/oL0IIMQjFrs?t=1310 - The blog had to use data type DATE instead of DATETIME for Brytlyt since it doesn't support the latter. DATETIME was used for the other DBs, which is a heavier computation. https://tech.marksblogg.com/billion-nyc-taxi-rides-p2-16xlarge-brytlytdb-2.html https://tech.marksblogg.com/billion-nyc-taxi-rides-p2-16xlar... So all-in-all it seems like a more carefully constructed, hardware balanced comparison would be needed to see which the quickest would be.
- trollied 8y ago> In JSON-format, AQL provides better programmatic query experience than SQL for dashboard and decision system developers, because it allows them to easily compose and manipulate queries using code without worrying about issues like SQL injection Oh, right. Surely your "data scientists" and people that are familiar with a STANDARD Query Language would be better off using a SQL interface? Worrying about SQL injection? What sort of crappy codebase do you even have?
- dang 8y agoWould you please not post nasty comments to HN, regardless of how crappy someone's codebase may be? https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- tanilama 8y agoAny benchmarks against traditional data warehouse solutions?
- reilly3000 8y agohttps://tech.marksblogg.com/benchmarks.html https://tech.marksblogg.com/benchmarks.html It paints a clear picture of how performant GPU analytics can be.
- vxxzy 8y agoWhen will we finally see some OpenCL (AMD) GPU database software?
- polskibus 8y agoIs there a feature in OpenCL or AMD that makes database development easier in such hardware than on NVidia hardware?
- yayr 8y agohttps://en.wikipedia.org/wiki/OpenCL https://en.wikipedia.org/wiki/OpenCL "When releasing OpenCL version 2.2, the Khronos Group announced that OpenCL would be merging into Vulkan in the future." I guess CUDA is a more save and stable bet for the near term future if you need to choose between the alternatives...
- slavik81 8y agoComments on this matter from the Khronos president: > Khronos has talked about the convergence between OpenCL and Vulkan - a little clumsily as it turns out - as the message has been often misunderstood - our bad. To clarify: > a. OpenCL is not going away or being absorbed into Vulkan - OpenCL Next development is active > b. In parallel with a. - it is good for developer choice if Vulkan also grows its compute capabilities > c. OpenCL can help b. through projects like clspv that test and push the bounds of what OpenCL kernels can be run effectively over a Vulkan run-time - and maybe inspire Vulkan compute capabilities to be expanded. Related article: https://www.phoronix.com/scan.php?page=news_item&px=Vulkan-OpenCL-Interop-2019 https://www.phoronix.com/scan.php?page=news_item&px=Vulkan-O...
- dragontamer 8y agoI wouldn't call AMD a champion of OpenCL. Only OpenCL 2.0 is supported, and not completely on ROCm yet. OpenCL 2.0 isn't too hot on the AMDGPU-pro drivers either, and SPIR-V intermediate language doesn't really work well on AMD's GPUs (it either runs slowly or fails to run). Frankly, it seems like AMD is going to be more successful with HCC and its HIP (CUDA-compatibility layer). It plays a distant 2nd place vs NVidia, but the CUDA-like single source environment is a superior development platform. -------- OpenCL 2.2 seems to be only supported on Intel platforms. Either Intel GPUs, Intel CPUs (to AVX512), or Intel Altera FPGAs. If I were to write OpenCL for AMD GPUs, I'd stick with OpenCL1.2 unless there was an absolute need for OpenCL 2.0 features. The OpenCL 1.2 stuff just seems more mature and better tested on AMD GPU Platforms. Hopefully ROCm changes that, but that's my opinion for the current state of affairs.
- polskibus 8y agoI wonder if they considered adding GPU support to ClickHouse instead of building another database.
- arnon 8y agoAdding GPU support to an existing database is incredibly hard, and usually doesn't pay off very well. We were considering doing that some years ago, but opted for writing our own (GPU accelerated) database too. We talked about it recently at a CMU Database Group talk (relevant parts start at around 17:00 - https://youtu.be/bSYFwd50EcU?t=1028 https://youtu.be/bSYFwd50EcU?t=1028)
- mandeepj 8y agoThey could not find anything existing in 2019 that they have to build a new db?
- fargo 8y agoWhat happens when you have more VC money than sense
- subprotocol 8y agoVery cool! I love geeking out on analytics tech and look forward to studying its design further. My take as I see it so far (please correct me if I'm wrong)- * As a datapoint Pinot/Druid/Clickhouse can do 1B timeseries on one server. AresDB sounds like it's in the same ballpark here * Pinot/Druid don't do cross table joins where AresDB can. My understanding is these are at (or near?) sub-second which would be a very distinguishing feature. I'm not sure how this will translate to when distributed mode is built out, as shuffling would become the bottleneck. Maybe there would be some partitioning strategy that within a partition allows arbitrary joining or something? * Clickhouse can do cross table joins, but aren't going to be sub-second * AresDB supports event-deduping. I think this can easily be handled by the upstream systems (samza, spark, flink, ..) in lambda * Reliance on fact/dimension tables. - This design/encoding is probably to help overcome transfer from memory to GPU, which in my limited experience with Thrust was always the bottleneck. - High cardinality columns would make dimension tables grow very large and could become un-unmanageable (unless they are somehow trimmable?)
- peferron 8y agoRegarding your second point: your intuition seems good, as Alipay apparently extended Druid to performs joins this way, with good performance [1]. Unfortunately it looks like they won't finish open-sourcing it, but it at least validates the idea. [1] https://github.com/apache/incubator-druid/issues/6457 https://github.com/apache/incubator-druid/issues/6457
- lyxsus1 8y agoHow does it compare with pg-strom ( https://github.com/heterodb/pg-strom https://github.com/heterodb/pg-strom )?
- sagichmal 8y agoCould someone with more context compare and contrast this with typical OLAP stuff?
- northerdome 8y ago"However, these existing solutions did not provide an adequate way for an engineer to get promoted. Thus, we decided building our own was necessary."
- threeseed 8y agoWhat existing solutions solve their particular needs ? And don't forget that rolling out commercial products in an enterprise is often expensive and time consuming e.g. involving legal, procurement, architecture etc.
- deleted 8y ago[deleted]
- m0zg 8y agoYou aren't wrong, but that's how 99.9% of innovation happens. And every now and then something useful comes of it.
- dang 8y ago"Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- datumdadum 8y agoSeems like a fairly benign humorous comment to me.
- usgroup 8y agoThis is cool to see. check out pgstrom for Postgres I’d you haven’t heard about it. Especially if you’re curious about perf gains with a gpu in like for like comparisons. One peeve, not loving the idea of having to use AQL and not having a command line analytical interface. Does AQL support windowing functions, CTEs, sub queries ?
- alfalfasprout 8y agoI'm surprised GPUs were the right call for this use case. Not to say GPUs aren't useful in DBs, but I/O and keeping the GPUs working hard quickly becomes a bottleneck. I'd assume the workloads must be non-trivial in order for this to be superior to general purpose CPUs using SIMD + in-memory data.
- samstave 8y agoI have what may be a completely stupid question, but the imagination in me just posited it to me, so I thought I'd ask: Can you translate raytracnig to DB relations? You have a raytracing engine, and many proven approaches to be able to determine what is visible from any given point. What if the "point" was, in fact, "the query" and from that point - you apply the idea of raytracing to see what pieces of information are the relevant fields in a space of a DB that pulls in all those fields - and the shader translates to the "lens" or "answer" you want to have from that points perspective? Is this where a GPU would help? Why? Why not?
- alfalfasprout 8y agoIn theory, yes. Rays get bounced around from a light source until they hit a camera (or decay below a threshold). The net intensity is recorded for all the rays in the camera plane. Computing the net intensity is a reduction over all the rays hitting that pixel in the camera plane. Then you have NUM_BOUNCES many map steps to compute the position and intensity of the ray after each bounce. So in theory these map and reduce operations could be expressed in a database context. In practice, does it make sense? Not really since each ray is not the same amount of work. One ray can go straight to the camera. Another could bounce many, many times going through many objects before hitting the camera or dying out altogether. GPUs are terrible at handling workloads with high thread divergence (some threads run much slower than others).
- rhdev7150 8y agoDoes Uber use Salesforce? I’m asking because I’m saying no way with all the work their engineering team does, custom this and custom that. I know there is a random blog on the internet that says they did in 2016 but I think the source is wrong. They have smart people and a need for big data quickly, throwing money at enterprise Salesforce just seems backwards to how I see them work. I think it’s cool they push technology and would love to know what the politics of that is like internally and how the higher ups trust the engineers vision.
- arnon 8y agoIt's a bit of a bummer they didn't choose to use a common language like some flavour of SQL, but opted to create another proprietary API
- xiaodai 8y agoI wonder if someone should design a DPU for database workloads. I think the GPU is used because it's available not because it's optimize for the type of work people want to do with data.
- Vekz 8y agosounds like great FGPA prototype idea
- mbesto 8y ago> software powered by machine learning models leverages data to predict rider supply and driver demand; and data scientists use data to improve machine learning models for better forecasting. Why does any of that require real-time analysis? What's the business need for this being instantaneous? Feels very NIH...
- treypitt 8y agonational institute of health?
- IanCal 8y agoNot Invented Here - refers to people avoiding bringing in something already built from outside and building in-house instead. https://en.wikipedia.org/wiki/Not_invented_here https://en.wikipedia.org/wiki/Not_invented_here
- jon37 8y agoMost businesses don't require real time forecasting, but I would speculate that Uber likely uses it to decide when and where to activate surge pricing.
- nemothekid 8y agoA friend, who works for Uber, was once explaining the difference between AirBnb’s tech stack and Uber’s even though they are both “marketplaces” In short, Uber has a ton more infrastructure because they need to do more in real time. Calculating the price of a trip from point A to point B needs to be done in real time. Then there are other factors like Pool, surge, and various rider preferences that all need to be accounted for. Uber is sort of unique in how much of their stack nessicates real time analytics
- mbesto 8y agoI don't disagree but the use cases Uber is talking about is not app functionality but for their analytics team (from the article): - Build dashboards to monitor our business metrics - Make automated decisions (such as trip pricing and fraud detection) based on aggregated metrics that we collect - Make ad hoc queries to diagnose and troubleshoot business operations issues None of those require real-time information, because there isn't real-time decision-making required from the information presented.* With the exception of the "automated decisions", then I don't see the need.
- yingw787 8y agoOoh nice! I work at a GPU database company (kinetica.com) and we are targeting larger enterprises and trying to replace their legacy store and analytics stacks. We’re releasing our 7.0 database stack for general availability tomorrow.
- davidhyde 8y ago> GPU technology has advanced significantly over the years, making it a perfect fit for real-time computation and data processing in parallel. I wouldn’t call GPUs perfect for performing analytics functions. It can be painful to force your data processing algorithms through shaders. What GPUs are is cheap in terms of dollar cost per flop and power consumption.
- nostrademons 8y agoPresumably they write the algorithm-to-shading munging once, in the query engine, and then clients of the database don't have to worry about it at all. It looks like they didn't even write it, outsourcing it to Nvidia Thrust.
- subhajeet2107 8y agoHow does it compares to ClickHouse ? Isnt creating a proprietary Query Language going to be a problem for Adhoc quries? Why create yet another language when the industry is standardizing on SQL or a subset of SQL
- jamesblonde 8y agoClickHouse is interesting - it's also an apache project. I am suprised Uber didn't build AresDB on it. The stigma, unfortunately, of coming from Russia makes it hard for the project to gain mindshare in the Valley.
- deleted 8y ago[deleted]
- peferron 8y agoClickHouse is licensed under the Apache License 2.0, but isn't an "Apache project" as in an ASF project. Maybe that's what you meant—just want to make that clear for other readers. ClickHouse scan performance is through the roof, but it also seems fairly difficult to operate compared to the alternatives. For example, the concept of "deep storage" in Druid and Pinot makes rebalancing and replication trivial (at the expense of additional storage space). ClickHouse doesn't have that and requires more babysitting. And that's without even going into something like BigQuery, which is on a completely different level regarding operational simplicity if your use cases support it. Also, if queries are heavily filtered by arbitrary dimensions, then ClickHouse starts to lose its edge compared to fully-indexed systems. This makes ClickHouse fairly niche IMO, despite being exceptional at what it does.
- youngtaff 8y agoAFAIK CloudFlare run it for their analytics platform so it'd be interesting to see why it works for them
- jgrahamc 8y agoYes we do. We moved our entire analytics backend to ClickHouse some time ago. It works well for us: https://blog.cloudflare.com/http-analytics-for-6m-requests-per-second-using-clickhouse/ https://blog.cloudflare.com/http-analytics-for-6m-requests-p...
- low_tech_love 8y agoCool, another GPU application that has nothing to do with graphics; isn’t it time we start building new hardware for these things?
- johnhenry 8y agoIt turns out that "graphics processing units" are good for more applications than just processing "graphics"... perhaps it would make sense to rename them something more inclusive?
- elephantum 8y agoLike Tensor Processing Units?
- swittmer 8y agoA GPU accelerated database with custom API for analytics? Jedox does that since 2007 and some of the biggest global companies use it. Jedox is not only an in-memory DB with optional GPU acceleration, it is a complete software suite with powerful tooling for ETL, Excel integration, Web-based spreadsheets and dashboarding. see https://www.jedox.com https://www.jedox.com and https://www.jedox.com/en/resources/jedox-gpu-accelerator-brochure/ https://www.jedox.com/en/resources/jedox-gpu-accelerator-bro...
- vishivatnik 8y agoSLAVA UKRAINE!!! GEROYAM SLAVA!!!!