9 ms·
MapD Open Sources GPU-Powered Database
- drej 9y agoI ran cloc to get a better idea of the tech in this. Interesting. Edit: I noticed a lot of Go, but only one live script in the repo. So I excluded the 'ThirdParty' folder, where some vendored libraries reside. Edit2: That 90K SLOC single-file C goodness is a full SQLite release, which is not vendored in the ThirdParty folder, taken it out as well. -------------------------------------------------------------------------------- Language files blank comment code -------------------------------------------------------------------------------- C++ 127 4385 3960 54609 C/C++ Header 158 2960 4273 14301 Java 42 1599 1498 9539 CUDA 8 301 47 3125 CMake 36 377 967 2616 yacc 1 133 156 742 Maven 6 0 126 507 Go 1 72 12 491 Bourne Shell 5 72 43 369 Bourne Again Shell 3 38 0 208 lex 1 28 12 204 XML 1 12 12 182 Python 5 65 25 168 C# 1 12 11 112 JSON 1 0 0 73 SQL 3 0 0 25 -------------------------------------------------------------------------------- SUM: 399 10054 11142 87271 --------------------------------------------------------------------------------
- solidr53 9y agoWow thats interesting, most of the languages other than C,C++, CUDA, Go and Java must be build related things, right?
- mattnewton 9y agoI assume they are bindings? But I haven't found an API doc that confirms that. Edit: never mind, many of the 1-2 file languages are excluded from the latest list, so they were in the "third party" folder.
- ape4 9y agoIs processing speed the bottleneck for databases?
- remus 9y agoIt depends what you're doing. I haven't looked in a while, but last time I did the idea behind MapD was to produce interactive visualisations over lots of data. That can certainly get pretty computationally expensive.
- random3 9y agoThat's the big question I have too. I think typically it's I/O in one form or another. With GPUs sending data to the GPU seems to be another cost in the path. However if you consider the GPU part as a way to push down processing power close to data, then you can probably solve more complex problems like graph processing efficiently within the DB..
- jandrewrogers 9y agoBandwidth for moving data is typically the limiting factor if you are doing everything correctly, very rarely computation. Bottlenecks for workloads are commonly memory or network bandwidth these days.
- manigandham 9y agoIt can be. It's a complex balance between processing speed and IO (from disk, ram and network). GPUs are much more parallel and a great fit for most queries which can be broken up, so combine that with very fast GPU ram and local storage and you can eliminate network/distributed fetches to make scans even faster.
- detaro 9y agoGPUs can also have better memory bandwidth, if the relevant parts of the dataset fit in GPU memory and the access patterns work out. Which could happen for clever index structures and things like that. Or complex processing over limited results of (sub-)queries.
- tmostak 9y agoGPUs have phenomenal memory bandwidth (multiple terabytes per second in a multi-GPU server) in addition to the computational parallelism they are known for. MapD caches the most recently touched data in the VRAM across multiple GPUs, meaning it can execute scan queries over billions of records in milliseconds.
- Meai 9y agoThe demo https://www.mapd.com/demos/tweetmap/ https://www.mapd.com/demos/tweetmap/ is pretty laggy, I think you should probably make it more performant so people don't think it's the DB that is causing the problem.
- tigershark 9y agoAnd why finland is "scrambled" with that big rectangle of noisy tweets with uniform distribution even in the sea? O_o
- tmostak 9y agoIt's a bot.
- Bedon292 9y agoAvailable on the AWS marketplace already set up: https://aws.amazon.com/marketplace/pp/B071H71L2Y https://aws.amazon.com/marketplace/pp/B071H71L2Y Can test it for as little as $0.90 / hour on a p2.xlarge. I know I absolutely cannot wait to play with it.
- reacharavindh 9y agoThis blog has a few performance numbers based on a NYC taxi rides dataset, and some setup notes. Might be relevant for HNers here. http://tech.marksblogg.com/billion-nyc-taxi-rides-nvidia-pascal-titan-x-mapd.html http://tech.marksblogg.com/billion-nyc-taxi-rides-nvidia-pas... I have nothing to do with the blog. Just sharing.
- mmcclellan 9y agoThis is where I first got interested in MapD. The author has a new post detailing how to compile MapD: http://tech.marksblogg.com/compiling-mapd-ubuntu-16.html http://tech.marksblogg.com/compiling-mapd-ubuntu-16.html
- ccleve 9y agoI would very much like to see a blog post or some other explanation of how a GPU speeds up a database query. Most traditional SQL databases are I/O-bound, meaning that the cost of pulling index pages from disk into memory overwhelms everything else, and adding more CPU doesn't do much good. Even for in-memory databases the cost of pulling a page into GPU memory is likely to be high. The calculations you do are pretty simple -- you mostly traverse arrays of pointers and do intersections and accumulations. No floating point, just increment and compare integers. GPUs really shine when you can parallelize operations on blocks of data that are already in GPU memory. That's why they're great for graphics and games. I'm not seeing how they help with database queries over large volumes of data which cannot fit in GPU memory. I'd really love to read some kind of tutorial on the topic. There's likely some kind of magic here that I don't see.
- imaginenore 9y agoYour claim of I/O bound is probably only true for very large databases where indexes don't fit into tens/hundreds of GB of RAM typically available. Modern SSDs, even at the consumer level are already doing 2.5GB/s reads and 100K IOPs. I'd say most database setups are CPU limited.
- dtnewman 9y agoFrom their product page [1]: > MapD Core is an in-memory, column store, SQL relational database that was designed from the ground up to run on GPUs. However, memory is getting cheaper and cheaper these days. [1] https://www.mapd.com/products/core/ https://www.mapd.com/products/core/
- arnon 9y agoThe GPU may help in compressing and decompressing data, encrypting and decrypting as well as sorting it (block-wise) ahead of heavy operations. Compressing and decompressing are examples of reducing I/O, and shifting it over to compute instead - and the GPU is ideal for that. Most decent columnar databases are compute bound. I/O is less of a bottleneck compared to OLTP style databases.
- deleted 9y ago
- wiradikusuma 9y agoFor the laymen, is this something that I can use to replace MySQL/Postgre (with ACID), assuming I only use ANSI SQL and no stored-proc?
- remus 9y agoProbably not. MySQL and postgres are both very general purpose relational databases that have a broad range of features aimed at a wide range of use cases. MapD is designed for more specific use cases, specifically analytic querying and visualisation.
- wesm 9y agoI see they are supporting Apache Arrow (http://arrow.apache.org/ http://arrow.apache.org/) in their result set converter, which will be nice for interoperability with other data system that use Arrow: https://github.com/mapd/mapd-core/blob/21fc39fab9e1dc2c1682bcf3dcc2b49d5503aea6/QueryEngine/ResultSetConversion.cpp https://github.com/mapd/mapd-core/blob/21fc39fab9e1dc2c1682b... (I'm on the Arrow PMC)
- tmostak 9y agoYes its definitely alpha functionality now but we're very excited about this piece and potentially supporting more of the Arrow spec. In case you missed it check out the first project we are working on with Continuum and H2O with the "GPU Data Frame" here: https://github.com/gpuopenanalytics/pygdf https://github.com/gpuopenanalytics/pygdf.
- latenightcoding 9y agoAfter reading a book on RDBMS design I had a similar idea, but after thinking for a couple of minutes I just could't come up with a use case of a GPU-powered database, it would be significantly slower in most scenarios.
- sargun 9y agoI imagine that it's less interesting for general purpose OLTP, which is what a lot of RDBMSs target. GPUs become really interesting for OLAP use cases where you're trying to push through hypercubes as quickly as possible, and computation across them is embarrassingly parralizable.
- _wmd 9y agoJust curious, which book did you read?
- latenightcoding 9y agoI read a couple this is one of them: http://db.cs.berkeley.edu/papers/fntdb07-architecture.pdf http://db.cs.berkeley.edu/papers/fntdb07-architecture.pdf
- polskibus 9y agoThis is great stuff! I wonder how it makes IBM, Oracle and MS feel right now. I can even see INSERTs allowed, wonder how it works with JOINs and if it is ever going to support transactions.
- rsp1984 9y agoThis is quite cool. I'm curious though what the monetization strategy is now. After all they're a for-profit company that has taken a lot of VC. Sure, they could sell integration and maintenance services on top but services are typically much harder to scale than software -- not quite what gets investors excited.
- shusson 9y agoMy understanding is that the distributed[1] capability of MapD is not open sourced. And I'm guessing a lot of the companies (especially the ones with money to burn) that would use MapD require at least some horizontal scaling. [1]http://docs.mapd.com/latest/mapd-core-guide/distributed/ http://docs.mapd.com/latest/mapd-core-guide/distributed/
- red2awn 9y agoHow does this compares to other GPU database like BlazingDB?