7 ms·
PGStrom: GPU-accelerated PostgreSQL
- spacemanmatt 11y agoThis is cool but is there actual news about the project other than existence?
- jpgvm 11y agoUnfortunately most of the discussion seems to be going down in Japanese. That said if you don't mind just reading diffs you can get an idea of the state of the project by taking a gander at their github: https://github.com/pg-strom/devel https://github.com/pg-strom/devel
- elchief 11y agoJust our friendly monthly reminder...
- mosselman 11y agoI imagine this isn't production ready. Or is it? It looks very cool and like a good direction to go in. You'd need a VPS with GPU access though.
- iopq 11y agoI got excited until I saw CUDA instead of OpenCL...
- ogrisel 11y agoAnd the use of GPL license which means that this code cannot be contributed upstream to PostgreSQL itself because it uses a more liberal license: http://opensource.org/licenses/postgresql http://opensource.org/licenses/postgresql.
- ksec 11y agoOk this is annoying, why do they choose GPL instead of something similar to PostGre?
- pjmlp 11y agoIt would help if OpenCL eco-system was half as good as CUDA, specially in Fortran, C++, alternative languages support and debuggers.
- iopq 11y agoDoesn't matter, I can't run it on my machine.
- tsuru 11y agoHere are a couple more resources where the lead gives a bit more description: http://on-demand.gputechconf.com/gtc/2015/presentation/S5276-Kohei-KaiGai.pdf http://on-demand.gputechconf.com/gtc/2015/presentation/S5276... http://www.slideshare.net/kaigai/gpgpu-accelerates-postgresql http://www.slideshare.net/kaigai/gpgpu-accelerates-postgresq... edit: His slideshare page http://www.slideshare.net/kaigai http://www.slideshare.net/kaigai has even more recent presentations. I've only had time to skim much of slides, but it seems he's flip-flopped a couple of times between OpenCL and CUDA
- ck2 11y agoWhat the what? 10 times faster for one join? https://wiki.postgresql.org/images/5/50/PGStrom_Fig_MicroBenchGpuJoin.png https://wiki.postgresql.org/images/5/50/PGStrom_Fig_MicroBen... Or am I reading that incorrectly? How is that possible?
- lazyjones 11y ago> How is that possible? I'll guess: It's all in-memory, so not I/O-bound. Rows are processed in parallel with PGStrom (in 15MB chunks), while Postgres does it sequentially in one thread. Apparently moving the data around is fast enough not to matter.
- mslot 11y agoJoining a large number of tables is a sweet spot for PGStrom where in which there are a lot of parallelizable computations. For a lot of queries it can actually be slower.
- calpaterson 11y agoHopefully the query planner would choose whether or not to use PGStrom for a specific query. Many interesting queries in normalised databases involve a lot of joins. In a BCNF database most interesting queries use 3 or more joins. Whether or not I believe the microbenchmarks is another thing :)
- dsugarman 11y agoThink of GPUs as excelling at MapReduce, either they do the same operation on many data points or they reduce the data by combining them in an aggregate. The query listed is taking an avg on a variable # of joins. Each join is doing a sequential search which can be a lot faster if you can process this in parallel and then when you find the right data you want to aggregate, you are then doing the aggregate. Both are incredibly faster in parallel. The gpu was designed so you can, lets say, move all vertices in a character's body 1 inch forward, you would need to process this one translation for potentially thousands of vertices. It was also designed to do bitmap operations very quickly where you need to get the same image, but smaller so the GPU would combine nearby pixels to provide a good heuristic of what that same image would look like further away.
- lugus35 11y agoWhat would be nice is a document which would explain how to convert queries to use the GPU as its best, a document which would not require its reader to be a GPU geek nor a Postgresql expert. A technology can only succeed when the average programmer can use it without doing much errors, or can find quickly a solution using google.
- logicallee 11y ago>how to convert queries to use the GPU as its best, a document which would not require its reader to be a GPU geek nor a Postgresql expert Uh, isn't this like asking "how do I do calculus without being a calculus geek"? I mean it's specialized - how do you get around being a "GPU geek" if you're coding for a GPU "at its best"?
- spacemanmatt 11y agoAgreed. This is squarely aimed at database geeks which is a fairly esoteric audience. It's gonna get geeky.
- jjaredsimpson 11y agoNot being a member of the target audience doesn't mean the speaker is doing something wrong.
- niutech 11y agoHere is a paper on how to accelerate SQL database operations on a GPU: http://www.cs.virginia.edu/~skadron/Papers/bakkum_sqlite_gpgpu10.pdf http://www.cs.virginia.edu/~skadron/Papers/bakkum_sqlite_gpg... (University of Virginia, 2010) and an ealier work: http://www.researchgate.net/profile/Qiong_Luo2/publication/237618362_Relational_Query_CoProcessing_on_Graphics_Processors1/links/0046353abc1a0c5dfd000000.pdf http://www.researchgate.net/profile/Qiong_Luo2/publication/2... (Hong Kong University of Science and Technology, 2009) And here are similar pojects: http://www.mapd.com/ http://www.mapd.com/ https://github.com/antonmks/Alenka https://github.com/antonmks/Alenka
- onion2k 11y agoA technology can only succeed when the average programmer can use it without doing much errors, or can find quickly a solution using google. I don't think that's true. A technology might only be able to become a mainstream product if the average programmer can use it, but there are a lot of niche products that most programmers will never use. That doesn't mean those technologies haven't succeeded. If you only ever use things that are similar to the things you already know (e.g. things you can use without making errors) then you will only make very slow progress as a developer. You'll always be avoiding the things that are different to your existing skill set. Sometimes, if you want to use something radically different to what you already use, you have to put in the effort and learn something that is hard. You can't expect the technology to come to you.
- michaelt 11y agoI was under the impression that to get the speed benefits of CUDA you had to coalesce memory access. Does anyone know how nested loop joins can use indexes without messing up the memory access coalescing?
- tempodox 11y agoIf I simplify this & other news to the statement, “our GPUs are Turing-complete”, then one question seems to follow logically: When will our CPUs become multi-core on the scale of a GPU? Or is that a stupid question that only looks logical?
- papercrane 11y agoGPU's were designed for a problem that is easy to make massively parallel. If we tried to make our CPUs like GPUs we'd end up with something not very useful for most of what we do.
- belorn 11y agoWhen/If HP finish the machine with 2500 cores, that will basically result in the CPU becoming as parallel as GPU's.
- bibabo 11y agoDoes the machine share the same memory layout/architecture as a GPU (I don't know each one). Thanks.
- eloff 11y agoThat's just a bunch of standard (ARM currently) boxes networked together with a special backplane that allows for DMA between different servers. Not really comparable to a GPU.
- Symmetry 11y agoWhat NVidia calls a core and what Intel calls a core really aren't comparable. If you go by the Intel definition which is more or less "Capable of independantly issuing an instruction" then you might find 15 cores in a Xeon versus 24 SMM "cores" in the top NVidia chip. Going by the NVidia definition which is more or less "Things that can execute instructions from a queue" NVidia would have 3072 cores in it's top chip and Intel would have 120 cores or "execution ports" as they call them.
- Symmetry 11y ago
- kyloon 11y agoThis reminds me of MapD (http://www.mapd.com http://www.mapd.com), though I am not sure which database MapD builds their stuffs on.
- headgasket 11y agoIt actually works with openCL. I had to do a few changes to make it work on MacOSX, but I beleive the language barrier and the fact that his project is in a flux and I came out of the blue caused these few lines to not merge. Please see my fork of his dev branch, that works with 9.5dev: https://github.com/pg-strom/devel/pull/144 https://github.com/pg-strom/devel/pull/144 correction: those changes to compile on macox did get pulled so https://github.com/pg-strom/devel https://github.com/pg-strom/devel should build with openCL on your mac to link with 9.5dev.
- javajosh 11y agoAnyone have a sense for the kinds of queries this would actually help with?
- pcwalton 11y agoProbably queries that have a lot of brute-force comparisons involving full table scans. "select foo from bar where baz > 30" with no index, for example. (Source: I've done a fair bit of experimentation in GPU queries in the domain of CSS selector matching.)
- radiowave 11y agoInteresting thought. I was assuming that the impressive performance figures based on joining 10 tables together would be making use of indexes, because with full table scans it might be difficult to keep the GPUs fed with data.
- jessedhillon 11y agoIn the article linked, they show a multi-way join on integer keys, which is able to leverage accelerated integer comparison (presumably.) I'd imagine a whole range of GIS queries also have the potential to be sped up by GPU offloading.
- irremediable 11y agoI like it that there's lots of robust criticism in the comments, but I think people are maybe being a little pessimistic. This is pretty exciting, isn't it? GPUs have sped things up in a lot of technologies, and I'm intrigued to see what they can do for databases.
- victorbojica 11y agodoes anyone know if this would work with postgis?