4 ms·
Super cool, and thanks for sharing! Are you familiar with APL/K/Q/J ? These implement column store databases which a programming language significantly more ex
by beagle3 15y ago
Super cool, and thanks for sharing!
Are you familiar with APL/K/Q/J ? These implement column store databases which a programming language significantly more expressive and usable. K and J are also significantly faster than any other SQL database -- it would be interesting to compare K's ultra-optimized in-memory processing to the GPU -- I'm not at all sure GPU is going to win here in practice.
See https://github.com/kevinlawler/kona https://github.com/kevinlawler/kona for an open source K implementation (the wiki has docs). Same language, newer version (with database integration) and english language instead of ascii chars is called Q see http://kx.com/q/d/ http://kx.com/q/d/ (kdb1, q1 if you like longer, kdb, q if you like terser), and http://code.kx.com/wiki/Main_Page http://code.kx.com/wiki/Main_Page
- silentbicycle 15y agoI'm not sure K's in-memory processing is incompatible with running on a GPU, either. I've been working on a fork of Kona that has a very different architecture (e.g. a bytecode VM, generational mark/sweep GC, lexical scope), and I plan on experimenting with CUDA/OpenCL once the fundamentals are set.
- beagle3 15y agoOh, K's in memory processing model is surely compatible with modern GPUs - at last for most primitives. The thing I'm not sure about is whether the transfer overhead (GPU, CPU, Memory) would still be worth it in the general case. There's a question of what you're actually comparing to - e.g. you can have a (relatively) naive implementation, like the first commit of Kona was. You can have Arthur's cache-optimized-but-still-pure-C-and-CPU implementation (I think the main kona branch is comparable these days, but I'm not sure), which is faster. And then, you can compare it to an SSE2/SSE4 version, still on the CPU; such a version does not exist afaik, but is easier to code than you would expect if you use ispc (https://github.com/ispc/ispc/tree/master/examples https://github.com/ispc/ispc/tree/master/examples) or cilk-plus -- simple things with no data dependencies, as many K primitives are, often get a 4x speedup with ISPC. Oh, and by the way, thanks for your work on kona and this message -- I've only been following kevinlawler's branch, but I'll start following you too.
- silentbicycle 15y agoThe transfer overhead would only need to happen once if there's a sequence of GPU-able primitives, though. The dataflow analysis for that isn't too hard. I'll check ispc out too, thanks. My fork* is using some SSE, via GCC-specific extensions (http://gcc.gnu.org/onlinedocs/gcc/Vector-Extensions.html http://gcc.gnu.org/onlinedocs/gcc/Vector-Extensions.html). It could be special-cased into Kona as-is, but I've found a better way to implement scalars than the elaborate preprocessor hack I did there. (Besides, tuning the reference counting would probably have a bigger impact, something like Deutsch-Bobrow at least.) * It's not on github, yet; I don't usually post things there until they're reasonably usable end-to-end. My kona fork is mostly for pull requests; I'm talking about a completely new implementation. Would you like to continue this discussion off HN? My contact info is in my profile.
- vessenes 15y agoMy first thought looking at this was "Ooooh, it would be so nice if my GPU could parse Q.." In fact, Q thoroughly sold me on tightly coupling a small expressive language with a parallel-friendly datastore. So many data problems require just this combination, and most general computing languages are surprisingly bad at it in comparison.