4 ms·
Ask HN: Fast Read-Only Relational Data Store
I don't mind if building the database takes on the order of days, but I'd like queries to be as quick as possible. If I get new data, I am fine rebuilding the database from scratch rather than inserting. (New data comes in on the order of months to years).
I have several related datasets that I'd like to query. The datasets are currently stored in flat files. The largest data set is 4 million rows x 20 columns, the next largest is 50000x1300 (or 650 million x 3 if "melted"). There are 5 bigger data tables, and several small metadata tables.
What sort of database would be best for this size and these goals? What additional information would you need before recommending a solution?
- tmaly 9y agoI needed a readonly key value system. I ended up building out my own key value store for several million entries using a memory mapped file in Go. The total file size is about 17GB. I originally wanted to use Boltdb for this, but it was taking too long to load, and the author mentioned in a github issue that just using a memory mapped file would be faster. I use a dictionary of keys to offsets in the memory mapped file to perform my lookup. My total loading time only takes a few minutes. If you only need to query on a simple key value, this approach may work for you.
- cpburns2009 9y agoHave you considered using SQLite? [1] [1]: https://www.sqlite.org/about.html https://www.sqlite.org/about.html