5 ms·

I used to think the same thing as much of my early career was spent in graphics. Then I got much more familiar with relational databases and realized I was an i

by rpedela 12y ago

I used to think the same thing as much of my early career was spent in graphics. Then I got much more familiar with relational databases and realized I was an idiot. For example I spent about 3 months hand tuning code that read a quad from a TB sized image stored on disk into a GPU so it could analyzed by computer vision algorithms. Then I realized that using a DB would have been about as fast and I would have saved 3 months of time. Unfortunately I didn't realize this until I left graduate school. In conclusion, yes you often don't need a database and sometimes it is wrong tool but usually it saves you a lot of time because it is fast enough and general purpose enough.

sklogic 12y ago
How exactly using an RDBMS would have helped accessing a plain two dimensional array?
- rpedela 12y ago
  Accessing is easy. Making it fast enough off disk to feed the beast (GPU) is not. The DB has already solved that problem and given a really simple interface: SQL. My hand tuned code was probably faster but not enough to warrant 3 months of effort. I should mention this was several years ago and on a laptop since that was what the researchers used who needed to analyze the data. So I couldn't really solve performance issues with more RAM, etc.
  - sklogic 12y ago
    Sorry, I cannot believe this. SQL faster than mmap?!? Something unheard of. And I really do not understand why it should take 3 months to write a single line with an `mmap` call.
    - SamReidHughes 12y ago
      Yep, "using mmap" is worse performance-wise than what database engines do. It's not a suitable interface for shoveling data onto disk in a manner such that you can say, "I am done writing this to disk," and "This file will never be in an inconsistent state in the event of kill -9 or powerloss," with maximum performance. Also, it's not enough to cry "use mmap," if you don't want a corrupted file. I think your antipathy towards relational databases has more to do with your unwillingness to learn how to use them than anything else. Otherwise you'd be capable of speaking with other people here on their level.
      - sklogic 12y ago
        I'm pretty well aware of how to use RDBMSes. I even implemented RDBMS engines before. In this case, nobody is writing into a file, so mmap should have been a perfect solution.
    - rpedela 12y ago
      There are a lot details I am leaving out because I don't have time or patience to explain them in a HN comment. That 3 months turned into an entire chapter (~15 pages) of my thesis. But I will tell you a little bit about it. The data we were analyzing was biological and it wasn't actually an image. However it could reformatted as an image so we could apply computer vision algorithms to it. The thesis was first about the computer vision stuff being applied to biology and second about moving the computation to the GPU to speed it up. It went from ~1 hour to 15 secs when the computation was moved to the GPU. The problem with the GPU though is it only has so much RAM, and the PCIe bus to transfer data to it is rather slow. To make matters worse, getting data off disk is even slower and the GPU expects a certain data format for optimal processing. The majority of the 3 months was not writing code. The majority of the 3 months was developing our own custom file format. The GPU wanted a particular format for optimal data transfer and the computer vision algorithms needed a particular format to be able to work. So it was about creating a file format that was fast off disk and through the PCIe bus and still had the correct formatting necessary for the computer vision algorithms. The exact specifics of the data format and why it had to be a certain way are what would take many pages of text to explain so I will stop. If I used a DB instead, I could have simply put the data in a table and used rather simple SQL commands to get the data off the disk in the format the GPU and computer vision algorithms wanted. And it would have been about as fast and only taken maybe a week at most of effort (remember most of the time was spent on the data format). In other words, I could have graduated 3 months sooner which seems worth it to me. Just to be clear, I never said SQL is faster than mmap although there are cases where it can be. What I said was it was usually not worth the effort to make it as fast or faster. It is kinda like C/C++ vs Python/Ruby/Javascript/etc. Yes your program will be a lot more efficient if you use C/C++, but is it really worth the time if you can do the same thing in a higher level language with little effort? There are cases where it is appropriate to use C/C++, but I would argue it is more the exception than the rule. Databases are kinda like that for me now after learning how to use them properly. Can I beat them for my specific use case? Probably yes, but is it really worth the effort? Most of the time no, but there are exceptions.
      - sklogic 12y ago
        Very interesting indeed. Is your thesis published? Btw., have you considered using a uniform memory architecture (e.g., Intel HD, or an embedded GPU)? And I still cannot understand why do you need the "R" in "RDBMS" in your case. Sound like you could have been benefited from a simple tuple storage (see a bit of discussion in this thread). I doubt you need indexes and relations.

by rpedela 12y ago

sklogic 12y ago

How exactly using an RDBMS would have helped accessing a plain two dimensional array?

rpedela 12y ago
Accessing is easy. Making it fast enough off disk to feed the beast (GPU) is not. The DB has already solved that problem and given a really simple interface: SQL. My hand tuned code was probably faster but not enough to warrant 3 months of effort. I should mention this was several years ago and on a laptop since that was what the researchers used who needed to analyze the data. So I couldn't really solve performance issues with more RAM, etc.
- sklogic 12y ago
  Sorry, I cannot believe this. SQL faster than mmap?!? Something unheard of. And I really do not understand why it should take 3 months to write a single line with an `mmap` call.
  - SamReidHughes 12y ago
    Yep, "using mmap" is worse performance-wise than what database engines do. It's not a suitable interface for shoveling data onto disk in a manner such that you can say, "I am done writing this to disk," and "This file will never be in an inconsistent state in the event of kill -9 or powerloss," with maximum performance. Also, it's not enough to cry "use mmap," if you don't want a corrupted file. I think your antipathy towards relational databases has more to do with your unwillingness to learn how to use them than anything else. Otherwise you'd be capable of speaking with other people here on their level.
    - sklogic 12y ago
      I'm pretty well aware of how to use RDBMSes. I even implemented RDBMS engines before. In this case, nobody is writing into a file, so mmap should have been a perfect solution.
  - rpedela 12y ago
    There are a lot details I am leaving out because I don't have time or patience to explain them in a HN comment. That 3 months turned into an entire chapter (~15 pages) of my thesis. But I will tell you a little bit about it. The data we were analyzing was biological and it wasn't actually an image. However it could reformatted as an image so we could apply computer vision algorithms to it. The thesis was first about the computer vision stuff being applied to biology and second about moving the computation to the GPU to speed it up. It went from ~1 hour to 15 secs when the computation was moved to the GPU. The problem with the GPU though is it only has so much RAM, and the PCIe bus to transfer data to it is rather slow. To make matters worse, getting data off disk is even slower and the GPU expects a certain data format for optimal processing. The majority of the 3 months was not writing code. The majority of the 3 months was developing our own custom file format. The GPU wanted a particular format for optimal data transfer and the computer vision algorithms needed a particular format to be able to work. So it was about creating a file format that was fast off disk and through the PCIe bus and still had the correct formatting necessary for the computer vision algorithms. The exact specifics of the data format and why it had to be a certain way are what would take many pages of text to explain so I will stop. If I used a DB instead, I could have simply put the data in a table and used rather simple SQL commands to get the data off the disk in the format the GPU and computer vision algorithms wanted. And it would have been about as fast and only taken maybe a week at most of effort (remember most of the time was spent on the data format). In other words, I could have graduated 3 months sooner which seems worth it to me. Just to be clear, I never said SQL is faster than mmap although there are cases where it can be. What I said was it was usually not worth the effort to make it as fast or faster. It is kinda like C/C++ vs Python/Ruby/Javascript/etc. Yes your program will be a lot more efficient if you use C/C++, but is it really worth the time if you can do the same thing in a higher level language with little effort? There are cases where it is appropriate to use C/C++, but I would argue it is more the exception than the rule. Databases are kinda like that for me now after learning how to use them properly. Can I beat them for my specific use case? Probably yes, but is it really worth the effort? Most of the time no, but there are exceptions.
    - sklogic 12y ago
      Very interesting indeed. Is your thesis published? Btw., have you considered using a uniform memory architecture (e.g., Intel HD, or an embedded GPU)? And I still cannot understand why do you need the "R" in "RDBMS" in your case. Sound like you could have been benefited from a simple tuple storage (see a bit of discussion in this thread). I doubt you need indexes and relations.