3 ms·
It's a data acquisition system which records various data collected out in the field by operators on standard laptops. They're frequently operating in places wi
by mastax 5y ago
It's a data acquisition system which records various data collected out in the field by operators on standard laptops. They're frequently operating in places with no cell coverage so we can't "just" stream to the cloud, we store the data locally and then bulk upload later. Also want to be able to visualize the data on the laptops which may be larger than ram.
It's not a particularly demanding use-case, especially with respect to querying. Could just use some serialization format, parquet maybe, but I know SQLite works reliably and has good libs. It's nice to have standard tools available, and I know no matter what changes I'll be able to read these databases in 20 years. Didn't have a lot of time to work on this project so I didn't spend much time evaluating alternatives.
- marcosdumay 5y agoOh, that's interesting. I didn't think about batch importing local data. Yes, I imagine you want to minimize the time importing data, and so halving the size of your main data would speed things up by a sizeable amount.