3 ms·
This just seems like something riding GPU hype. We run a medium sized Memsql cluster, and get better performance on larger tables with more complex joins than t
by qeternity 10y ago
This just seems like something riding GPU hype. We run a medium sized Memsql cluster, and get better performance on larger tables with more complex joins than they do in their demo video...and we do this for much cheaper than the lowest g2 instance + AMI. Our data is quite narrow, so fits in RAM quite well, but I would bet that even Citus would fare quite well against this.
- obstinate 10y agoI suppose it depends on what operations you do on your table and what your query stream looks like. It's hard to imagine compute being the bottleneck for most SQL workloads . . .
- Wheaties466 10y agoOn a Postgresql query before version 9.6 is limited to a single CPU core. Depending on the amount of data it has to query it absolutely can take forever.
- qeternity 10y agoEven now, it's only multicore when doing full table scans (iirc...)
- jasonmp85 10y agoTrue (see here: https://www.postgresql.org/docs/9.6/static/parallel-plans.html https://www.postgresql.org/docs/9.6/static/parallel-plans.ht... ), but in PostgreSQL 10 there are Parallel Bitmap Heap Scans and Parallel Index Scans (see here: http://rhaas.blogspot.com/2017/03/parallel-query-v2.html http://rhaas.blogspot.com/2017/03/parallel-query-v2.html ).
- user5994461 10y agoConsider a paid database if that's what you need. It's not new that postgre has some limitations compared to the SQL servers/Oracle.
- qeternity 10y agoFor OLAP use cases, CPU is most often a bottleneck...
- user5994461 10y agoDoesn't OLAP implies branching/conditions? GPUs are terrible at evaluating if.
- arnon 10y agoOLAP is implied as a catch-all for analytics in this case
- threeseed 10y agoSQL is an overloaded term these days. There are plenty of SQL queries happening on non-database platforms e.g. Spark, Impala all of which are entirely CPU bound.
- felipe_aramburu 10y agoIf your problem can fit in RAM then this is probably not something that would help you. This solution is for people who are looking for massive scale. Also you are running a query from memory. We can run from memory as well and get much better throughput. The new version of BlazingDB also incorporates skip data which can improve join performance by several orders of magnitude for complex joins.
- qeternity 10y agoWhat are you talking about? I am talking about terabytes of RAM in the cluster. I would love to see some benchmarks of terabyte datasets. Everything I saw from your demo video was under 1b rows. Hardly qualifies as massive scale.
- felipe_aramburu 10y agoWhen we think about TB of data we do not want to store that in RAM. So if you are looking at a 10TB dataset. You may be willing to have 10TB of ram to fit it in but we aren't trying to solve that problem. We try to solve the problem of reducing the amount of information that has to be pulled from disk to be analyzed by having fine grain partitioning and using data skipping. We are optimizing the compression and decompression of the data to reduce file i/o bottleknecks. So your 10TB dataset can be analyzed with a 100GB of RAM. I agree that video is not very compelling. We are giving a talk at GTC that will show you something 3 orders of magnitude more interesting than that. https://gputechconf2017.smarteventscloud.com/connect/search.ww#loadSearch-searchPhrase=&searchType=session&tc=0&sortBy=dayTime&p=&i(40701)=109194 https://gputechconf2017.smarteventscloud.com/connect/search....
- ruw1090 10y agoSo blazing db mainly uses gpus to speed up compression/decompression? How do gpus help with partitioning and data skipping?
- felipe_aramburu 10y ago