5 ms·
There are a lot details I am leaving out because I don't have time or patience to explain them in a HN comment. That 3 months turned into an entire chapter (~15
by rpedela 12y ago
There are a lot details I am leaving out because I don't have time or patience to explain them in a HN comment. That 3 months turned into an entire chapter (~15 pages) of my thesis. But I will tell you a little bit about it.
The data we were analyzing was biological and it wasn't actually an image. However it could reformatted as an image so we could apply computer vision algorithms to it. The thesis was first about the computer vision stuff being applied to biology and second about moving the computation to the GPU to speed it up. It went from ~1 hour to 15 secs when the computation was moved to the GPU. The problem with the GPU though is it only has so much RAM, and the PCIe bus to transfer data to it is rather slow. To make matters worse, getting data off disk is even slower and the GPU expects a certain data format for optimal processing.
The majority of the 3 months was not writing code. The majority of the 3 months was developing our own custom file format. The GPU wanted a particular format for optimal data transfer and the computer vision algorithms needed a particular format to be able to work. So it was about creating a file format that was fast off disk and through the PCIe bus and still had the correct formatting necessary for the computer vision algorithms. The exact specifics of the data format and why it had to be a certain way are what would take many pages of text to explain so I will stop.
If I used a DB instead, I could have simply put the data in a table and used rather simple SQL commands to get the data off the disk in the format the GPU and computer vision algorithms wanted. And it would have been about as fast and only taken maybe a week at most of effort (remember most of the time was spent on the data format). In other words, I could have graduated 3 months sooner which seems worth it to me.
Just to be clear, I never said SQL is faster than mmap although there are cases where it can be. What I said was it was usually not worth the effort to make it as fast or faster. It is kinda like C/C++ vs Python/Ruby/Javascript/etc. Yes your program will be a lot more efficient if you use C/C++, but is it really worth the time if you can do the same thing in a higher level language with little effort? There are cases where it is appropriate to use C/C++, but I would argue it is more the exception than the rule.
Databases are kinda like that for me now after learning how to use them properly. Can I beat them for my specific use case? Probably yes, but is it really worth the effort? Most of the time no, but there are exceptions.
- sklogic 12y agoVery interesting indeed. Is your thesis published? Btw., have you considered using a uniform memory architecture (e.g., Intel HD, or an embedded GPU)? And I still cannot understand why do you need the "R" in "RDBMS" in your case. Sound like you could have been benefited from a simple tuple storage (see a bit of discussion in this thread). I doubt you need indexes and relations.
- rpedela 12y agoIt is published and you can find it somewhere in Purdue's library. It is not online though. It is a regret of mine that we didn't publish at a conference or journal, but at the time I was too concerned with getting out of there. I like school and research, but I didn't like the poverty. At the time (2006), embedded GPUs were not available or just becoming available. In this case, I didn't need relations but given the data size, indexes would have been very helpful since I had to retrieve quads so not to overflow GPU RAM. Any DB (in the most general sense) with indexing would have been fine. Since I can setup Postgres in about two minutes, that is easiest for me.
- sklogic 12y agoSo, back to my point - you did not need specifically RDBMS. I have nothing against tuple storages, stream DBMSes, berkeley-db style caches, etc. All my hate is reserved to the relational crap.
- rpedela 12y agoOh I see. Specifically for RDBMS and graphics: video game engines. Game engines are a bit weird because sometimes you need hierarchical data structures, stream processing, multi-core processing (e.g. physics), etc. I worked on a game engine for a few years, but again I did not know RDBMSes would be helpful. As far as I am aware, most game engines still do not use RDBMSes, but I think there are cases where they would make life a lot easier because parts of the game engine are relational. For example, a game object often has many properties and those properties relate to other game objects or pieces of the game engine itself. A game object usually has a graphical representation, a physical representation, a representation related to animation, etc. And you need to keep these different representations in sync. If the physical representation moves, then the other representations have to be moved in sync with the physical one. If one of the representations gets deleted, then they all need to be deleted. So tables and foreign keys would make this a lot easier. It is necessary? No because you can do it with OO, inheritance, trees, but it is actually a lot of effort to do it that way. That is one example, but I think there are several areas where RDBMSes can be helpful inside a game engine. Just to be clear, there are several areas in a game engine where RDBMSes make no sense too.