8 ms·

Alenka: GPU database engine

102 points by espeed 10y ago

m1sta_ 10y ago
Looks interesting. Need a lot more information.
gcp 10y ago
Should say CUDA instead of GPU.
- MayeulC 10y ago
  That's true. Anyone knows how good are the CUDA -> OpenCL translators? Or has an open CUDA compiler has been written (LLVM front end, likely)? So, in a nutshell, is it possible to run CUDA on a non-nVidia GPU nowadays (AMD or else)?
  - dagw 10y ago
    otoy.com claimed to have developed a CUDA -> OpenCL and CUDA -> CPU compiler, but they've not released anything and they've been very quite about it for the past 9 month or so. There's also this: https://github.com/vtsynergy/CU2CL https://github.com/vtsynergy/CU2CL but I know nothing about it.
  - floatboth 10y ago
    https://github.com/GPUOpen-ProfessionalCompute-Tools/HIP/blob/f052f43b3b3d48ac79f2111c7da74c24f1ad29b2/docs/markdown/hip_porting_guide.md#porting-a-new-cuda-project%22 https://github.com/GPUOpen-ProfessionalCompute-Tools/HIP/blo... CUDA → HIP translator. HIP is an abstraction over CUDA and AMD's HCC: https://github.com/RadeonOpenCompute/hcc/wiki https://github.com/RadeonOpenCompute/hcc/wiki
general_ai 10y ago
Calling this a "database" is a bit of an exaggeration.
- keknaut 10y ago
  why?
  - rotten 10y ago
    We don't really call pandas a database either. It looks like a data processing tool/library. "real" databases have integrated persistence models, as well as discussions and design tradeoffs regarding ACID transactions and scalability. There are query planners and indexes, constraints (type, value, foreign key) and other logic that can help enforce business rules around the data. There are triggers and embedded functions too. Still, it is pretty cool technology and I think something that will be integrated with a "real" database near you sometime soon.
    - paulmd 10y ago
      That's a matter of perspective. I don't think this is too different in concept from, say, Bigtable, which is billed by Google as a database.
      - general_ai 10y ago
        BT wasn't actually billed as "database" internally, until they had to sell it in Google Cloud where the definition of what constitutes a database is much looser. Inside Google it's known as a multidimensional hash table.
        paulmd 10y ago
        I would definitely agree that's a more precise definition. Constraining "database" to only ever refer to RDBMS is what I really have a problem with. But there really isn't a brightline between what's a table, and what's a filesystem, and what's a database. You can put a blob in a database or BigTable, and MongoDB is really close to being a flat file conceptually. You can have a filesystem or database that is content-accessible. You can have a filesystem that is atomic and supports rollbacks and can store relational data like symlinks. A virtual filesystem like LVM can support schema-like volumes on top of it. At the end of the day it's all just technology that lets me abstract my writes so I can deal with a more simplistic model backed by certain guarantees about behavior. I want to write a program that does XYZ, not write a filesystem/database driver. From there it's all just various tradeoffs.
chatman 10y ago
License information missing.
- MayeulC 10y ago
  Apache 2.0, going by the source files. Edit: though the "bison" files are GPLv3+, son It might be a safer bet.
  - nrjdhsbsid 10y ago
    Wot? Somebody threw up this "database" without even checking the licenses? I'm sure it works great
    - krona 10y ago
      fyi, the Bison GPL exception is in its documentation: https://www.gnu.org/software/bison/manual/html_node/Conditions.html https://www.gnu.org/software/bison/manual/html_node/Conditio... The exception is littered throughout bison generated code: "This special exception was added by the Free Software Foundation..."
      - nrjdhsbsid 10y ago
        Fair point, I'll leave my comment up so yours make sense. The downvotes should be fun :)
vegabook 10y ago
good to see open source activity in this space given the eye-watering prices that mapd is charging. https://aws.amazon.com/marketplace/pp/B01M0ZY2OV?qid=1484735254624&sr=0-1&ref_=srh_res_product_title https://aws.amazon.com/marketplace/pp/B01M0ZY2OV?qid=1484735...
nrjdhsbsid 10y ago
I keep hearing the promise of GPU databases but they don't seem to be terribly useful for most real world workloads. It reminds me of the big hoopla for GPU h264 encoders. When they came out everyone realized the quality was worse and not much faster. Some things don't lend themselves to parallel processing, notably anything linear like transactions. I mean yeah the GPU can sort a hundred billion items a second but how often do you really need to sort that many items using a database? In 99.9% of uses you have indexing or limits on the number of results. Just saying, this program looks more like a stream processing platform with a SQL-like frontend than a full database
- zer0t3ch 10y ago
  > everyone realized the quality was worse and not much faster I can't speak for the h264 situation, as I wasn't involved at the time, but with the newer h265 hardware, it is obnoxiously faster with minimal (in terms of end-user, not distributor) quality loss.
  - dijit 10y ago
    Are those the CPU instructions or the GPU ones? I remember when h264 CPU accelerators hit the scene. It was a game changer.
    - zer0t3ch 10y ago
      I don't know anything about h265 CPU accelerators, I'm comparing GPU HEVC (h265) encoding with a dedicated chip on the GPU to standard CPU encoding. (Using libx265 IIRC)
- floatboth 10y ago
  > GPU h264 encoders Are there any GPU (shader) encoders? There are dedicated fixed-function hardware encoders (VCE and NVENC) on graphics cards, and they're very popular, because they're the best way to record game footage if you don't have dedicated capture hardware. You don't want x264/5 hogging the CPU when you're playing games!
kakoni 10y ago
For postgres checkout https://github.com/pg-strom/devel https://github.com/pg-strom/devel
- arnon 10y ago
  I don't think the development of pg-strom is still active, as far as I can tell
marklit 10y ago
If anyone is interested in getting this setup I put together some install notes along with some speed bumps I encountered while benchmarking Alenka a few months back: http://tech.marksblogg.com/alenka-open-source-gpu-database.html http://tech.marksblogg.com/alenka-open-source-gpu-database.h...

102 points by espeed 10y ago

m1sta_ 10y ago

Looks interesting. Need a lot more information.

gcp 10y ago

Should say CUDA instead of GPU.

MayeulC 10y ago
That's true. Anyone knows how good are the CUDA -> OpenCL translators? Or has an open CUDA compiler has been written (LLVM front end, likely)? So, in a nutshell, is it possible to run CUDA on a non-nVidia GPU nowadays (AMD or else)?
- dagw 10y ago
  otoy.com claimed to have developed a CUDA -> OpenCL and CUDA -> CPU compiler, but they've not released anything and they've been very quite about it for the past 9 month or so. There's also this: https://github.com/vtsynergy/CU2CL https://github.com/vtsynergy/CU2CL but I know nothing about it.
- floatboth 10y ago
  https://github.com/GPUOpen-ProfessionalCompute-Tools/HIP/blob/f052f43b3b3d48ac79f2111c7da74c24f1ad29b2/docs/markdown/hip_porting_guide.md#porting-a-new-cuda-project%22 https://github.com/GPUOpen-ProfessionalCompute-Tools/HIP/blo... CUDA → HIP translator. HIP is an abstraction over CUDA and AMD's HCC: https://github.com/RadeonOpenCompute/hcc/wiki https://github.com/RadeonOpenCompute/hcc/wiki

general_ai 10y ago

Calling this a "database" is a bit of an exaggeration.

keknaut 10y ago
why?
- rotten 10y ago
  We don't really call pandas a database either. It looks like a data processing tool/library. "real" databases have integrated persistence models, as well as discussions and design tradeoffs regarding ACID transactions and scalability. There are query planners and indexes, constraints (type, value, foreign key) and other logic that can help enforce business rules around the data. There are triggers and embedded functions too. Still, it is pretty cool technology and I think something that will be integrated with a "real" database near you sometime soon.
  - paulmd 10y ago
    That's a matter of perspective. I don't think this is too different in concept from, say, Bigtable, which is billed by Google as a database.
    - general_ai 10y ago
      BT wasn't actually billed as "database" internally, until they had to sell it in Google Cloud where the definition of what constitutes a database is much looser. Inside Google it's known as a multidimensional hash table.
      - paulmd 10y ago
        I would definitely agree that's a more precise definition. Constraining "database" to only ever refer to RDBMS is what I really have a problem with. But there really isn't a brightline between what's a table, and what's a filesystem, and what's a database. You can put a blob in a database or BigTable, and MongoDB is really close to being a flat file conceptually. You can have a filesystem or database that is content-accessible. You can have a filesystem that is atomic and supports rollbacks and can store relational data like symlinks. A virtual filesystem like LVM can support schema-like volumes on top of it. At the end of the day it's all just technology that lets me abstract my writes so I can deal with a more simplistic model backed by certain guarantees about behavior. I want to write a program that does XYZ, not write a filesystem/database driver. From there it's all just various tradeoffs.

chatman 10y ago

License information missing.

MayeulC 10y ago
Apache 2.0, going by the source files. Edit: though the "bison" files are GPLv3+, son It might be a safer bet.
- nrjdhsbsid 10y ago
  Wot? Somebody threw up this "database" without even checking the licenses? I'm sure it works great
  - krona 10y ago
    fyi, the Bison GPL exception is in its documentation: https://www.gnu.org/software/bison/manual/html_node/Conditions.html https://www.gnu.org/software/bison/manual/html_node/Conditio... The exception is littered throughout bison generated code: "This special exception was added by the Free Software Foundation..."
    - nrjdhsbsid 10y ago
      Fair point, I'll leave my comment up so yours make sense. The downvotes should be fun :)

vegabook 10y ago

good to see open source activity in this space given the eye-watering prices that mapd is charging. https://aws.amazon.com/marketplace/pp/B01M0ZY2OV?qid=1484735254624&sr=0-1&ref_=srh_res_product_title https://aws.amazon.com/marketplace/pp/B01M0ZY2OV?qid=1484735...

nrjdhsbsid 10y ago

I keep hearing the promise of GPU databases but they don't seem to be terribly useful for most real world workloads. It reminds me of the big hoopla for GPU h264 encoders. When they came out everyone realized the quality was worse and not much faster. Some things don't lend themselves to parallel processing, notably anything linear like transactions. I mean yeah the GPU can sort a hundred billion items a second but how often do you really need to sort that many items using a database? In 99.9% of uses you have indexing or limits on the number of results. Just saying, this program looks more like a stream processing platform with a SQL-like frontend than a full database

zer0t3ch 10y ago
> everyone realized the quality was worse and not much faster I can't speak for the h264 situation, as I wasn't involved at the time, but with the newer h265 hardware, it is obnoxiously faster with minimal (in terms of end-user, not distributor) quality loss.
- dijit 10y ago
  Are those the CPU instructions or the GPU ones? I remember when h264 CPU accelerators hit the scene. It was a game changer.
  - zer0t3ch 10y ago
    I don't know anything about h265 CPU accelerators, I'm comparing GPU HEVC (h265) encoding with a dedicated chip on the GPU to standard CPU encoding. (Using libx265 IIRC)
floatboth 10y ago
> GPU h264 encoders Are there any GPU (shader) encoders? There are dedicated fixed-function hardware encoders (VCE and NVENC) on graphics cards, and they're very popular, because they're the best way to record game footage if you don't have dedicated capture hardware. You don't want x264/5 hogging the CPU when you're playing games!

kakoni 10y ago

For postgres checkout https://github.com/pg-strom/devel https://github.com/pg-strom/devel

arnon 10y ago
I don't think the development of pg-strom is still active, as far as I can tell

marklit 10y ago

If anyone is interested in getting this setup I put together some install notes along with some speed bumps I encountered while benchmarking Alenka a few months back: http://tech.marksblogg.com/alenka-open-source-gpu-database.html http://tech.marksblogg.com/alenka-open-source-gpu-database.h...