3 ms·
It is true though. A data lake is what companies end up with by dumping everything into HDFS, regardless of it's importance and relevance. So, it is just a hoar
by arnon 8y ago
It is true though. A data lake is what companies end up with by dumping everything into HDFS, regardless of it's importance and relevance. So, it is just a hoarding database.
I'll let the BlazingSQL guys answer the workloads question, but yeah, copying data to the GPU (host -> device) is an expensive operation, and it has to be justified.
With SQream DB (disclaimer: am a developer on SQream DB), we make decisions in the query compiler and optimizer if we want to copy data over to the GPU, or keep it in CPU for processing. The optimizer will know if it makes sense.
For example, when you're doing heavy transforms (eg. `SELECT (SQRT(POWER(x,2)+44)/w+RADIANS(COS(y)*z)) FROM t`), it may make sense to have it in the GPU anyway.
- yayr 8y agoWill you consider offering community editions for SQream in the future? It‘s a considerable differentiator for BlazingDB and PG Strom I guess
- arnon 8y agoThere is no such plan that I'm aware of
- twtw 8y ago> copying data to the GPU (host -> device) Copying data to the CPU (peripheral -> host) is expensive too, and it's perfectly possible to just replace peripheral -> host with peripheral -> device. I don't know if that is what blazingsql does, but it shouldn't be discounted.
- felipe_aramburu 8y agoIf you mean something like network card ==> device via RDMA we are not doing that yet but are currently working on this implementation. These past few months we have focused more on adding functionality to libgdf. We are now optimizing some of the i/os like this one you mention here. As of now if it goes over the wire it is being copied into pre allocated pinned memory buffers then sent to GPU. This will go away in about a month or so.