3 ms·
I do a lot of high performance scientific computing and find the most flexible and high performance way to access data- especially in parallel- is often a read
by _yb2s 2y ago
I do a lot of high performance scientific computing and find the most flexible and high performance way to access data- especially in parallel- is often a read only sqlite database on a ramdisk. It feels super hacky, but it is easier to setup and faster than anything else I’ve found.
- edweis 2y agoWe would love a write up about your findings.
- _yb2s 2y agoProbably not, I’m not a CS person and just found a hacky thing that was surprisingly fast enough for my use. I didn’t do a systematic comparison with other methods.
- jfkfif 2y agoMore performant than using HDF5?
- _yb2s 2y agoI doubt it- I didn’t do a systematic comparison, but stuck with this because it was fast enough that it wasn’t the bottleneck
- antognini 2y agoHonestly it doesn't really feel like a hack to me. Fast efficient parallel reads are something that SQLite was specifically designed to do. A friend of mine in astronomy once observed that a lot of people in the sciences should get comfortable working with databases because otherwise they just end up inadvertently spending a huge amount of effort rolling their own really crappy database.