3 ms·

If your problem can fit in RAM then this is probably not something that would help you. This solution is for people who are looking for massive scale. Also you

If your problem can fit in RAM then this is probably not something that would help you. This solution is for people who are looking for massive scale. Also you are running a query from memory. We can run from memory as well and get much better throughput. The new version of BlazingDB also incorporates skip data which can improve join performance by several orders of magnitude for complex joins.

qeternity 10y ago
What are you talking about? I am talking about terabytes of RAM in the cluster. I would love to see some benchmarks of terabyte datasets. Everything I saw from your demo video was under 1b rows. Hardly qualifies as massive scale.
- felipe_aramburu 10y ago
  When we think about TB of data we do not want to store that in RAM. So if you are looking at a 10TB dataset. You may be willing to have 10TB of ram to fit it in but we aren't trying to solve that problem. We try to solve the problem of reducing the amount of information that has to be pulled from disk to be analyzed by having fine grain partitioning and using data skipping. We are optimizing the compression and decompression of the data to reduce file i/o bottleknecks. So your 10TB dataset can be analyzed with a 100GB of RAM. I agree that video is not very compelling. We are giving a talk at GTC that will show you something 3 orders of magnitude more interesting than that. https://gputechconf2017.smarteventscloud.com/connect/search.ww#loadSearch-searchPhrase=&searchType=session&tc=0&sortBy=dayTime&p=&i(40701)=109194 https://gputechconf2017.smarteventscloud.com/connect/search....
  - ruw1090 10y ago
    So blazing db mainly uses gpus to speed up compression/decompression? How do gpus help with partitioning and data skipping?
    - felipe_aramburu 10y ago
      BlazingDB uses gpus to speed up transformations, predicates, running predicates on skip data metadata, gpu accelerated join algorithms. In terms of partitioning we use both to generate the data skipping meta data on the partitions and to reorder data around so that we can minimize the number of comparisons joins will likely require or reduce the number of partitions that need to be operated on. At large scale reorganizing information becomes very expensive the gpus allow us to crunch those numbers very quickly.

qeternity 10y ago

What are you talking about? I am talking about terabytes of RAM in the cluster. I would love to see some benchmarks of terabyte datasets. Everything I saw from your demo video was under 1b rows. Hardly qualifies as massive scale.

felipe_aramburu 10y ago
When we think about TB of data we do not want to store that in RAM. So if you are looking at a 10TB dataset. You may be willing to have 10TB of ram to fit it in but we aren't trying to solve that problem. We try to solve the problem of reducing the amount of information that has to be pulled from disk to be analyzed by having fine grain partitioning and using data skipping. We are optimizing the compression and decompression of the data to reduce file i/o bottleknecks. So your 10TB dataset can be analyzed with a 100GB of RAM. I agree that video is not very compelling. We are giving a talk at GTC that will show you something 3 orders of magnitude more interesting than that. https://gputechconf2017.smarteventscloud.com/connect/search.ww#loadSearch-searchPhrase=&searchType=session&tc=0&sortBy=dayTime&p=&i(40701)=109194 https://gputechconf2017.smarteventscloud.com/connect/search....
- ruw1090 10y ago
  So blazing db mainly uses gpus to speed up compression/decompression? How do gpus help with partitioning and data skipping?
  - felipe_aramburu 10y ago
    BlazingDB uses gpus to speed up transformations, predicates, running predicates on skip data metadata, gpu accelerated join algorithms. In terms of partitioning we use both to generate the data skipping meta data on the partitions and to reorder data around so that we can minimize the number of comparisons joins will likely require or reduce the number of partitions that need to be operated on. At large scale reorganizing information becomes very expensive the gpus allow us to crunch those numbers very quickly.