3 ms·
I think that depends on what you're doing with the data. If you're just grabbing one thing and working with it, or looping through and processing everything, m
by Falkon1313 6y ago
I think that depends on what you're doing with the data. If you're just grabbing one thing and working with it, or looping through and processing everything, maybe not.
But if you're doing more complicated query-like stuff, especially if you want to allow for queries you haven't thought of yet, then the DB might be useful.
Sometimes a hybrid of query-able metadata in a DB along with plain old data files is good.
That depends very much on your data, how much things key to each other, and what you're doing with it.
- xxs 6y ago>doing more complicated query-like stuff That's some kind of fallacy - standard datastructures would totally destroy any-sql-alike thing, if it comes to performance (and memory footprint). I guess it does depend on where the background comes when it comes to convenience - or how people tend to see their data. However like I said - for close to 3 decades I have not seen a single reason to do so. On the contrary I've had cases where optimization of 3 orders of magnitude was possible.
- throwaway08320 6y ago> for close to 3 decades I have not seen a single reason to do so Evidently you don't have dataset far exceeding the amount of RAM you can afford. For a good example look at LMDB.
- jawns 6y agoIt's easier to find devs who know basic SQL than it is to find devs who know pandas or whatever your language specific SQL-like library is. And the more complicated the queries, the more the gulf widens.
- recursive 6y agoI don't think pandas was the proposal here. I think "standard data structures" refers to arrays, hash tables, trees, and the like.
- busterarm 6y agoPerformance is not god. It is not the altar at which we sacrifice all other considerations.