3 ms·

It's can be a real problem when you need to store 13,000,000 floats per minute. Is it hard to work around? No. But every time I use SQLite it requires half a d

by mastax 5y ago

It's can be a real problem when you need to store 13,000,000 floats per minute. Is it hard to work around? No. But every time I use SQLite it requires half a dozen little hacks to work around it's strangeness and I would prefer to not have to do that.

marcosdumay 5y ago
Are you talking about the increased IO and disk usage due to doubles being twice the size? Because the conversion itself is usually done in hardware and is extremely fast. Anyway, I've never touched a serialization library that doesn't mess with the data types it gets. There is always some conversion needed, and this one is on the easy end of the scale. The fact that composed types (records and arrays) aren't on the list is a much larger troublemaker. (Regardless, I'm curious about the applications of 13M inserts per minute on a synchronous single-computer database. Pushing a machine to that extreme looks very interesting. Is that application something you can share?)
- mastax 5y ago
  It's a data acquisition system which records various data collected out in the field by operators on standard laptops. They're frequently operating in places with no cell coverage so we can't "just" stream to the cloud, we store the data locally and then bulk upload later. Also want to be able to visualize the data on the laptops which may be larger than ram. It's not a particularly demanding use-case, especially with respect to querying. Could just use some serialization format, parquet maybe, but I know SQLite works reliably and has good libs. It's nice to have standard tools available, and I know no matter what changes I'll be able to read these databases in 20 years. Didn't have a lot of time to work on this project so I didn't spend much time evaluating alternatives.
  - marcosdumay 5y ago
    Oh, that's interesting. I didn't think about batch importing local data. Yes, I imagine you want to minimize the time importing data, and so halving the size of your main data would speed things up by a sizeable amount.

by mastax 5y ago

marcosdumay 5y ago

Are you talking about the increased IO and disk usage due to doubles being twice the size? Because the conversion itself is usually done in hardware and is extremely fast. Anyway, I've never touched a serialization library that doesn't mess with the data types it gets. There is always some conversion needed, and this one is on the easy end of the scale. The fact that composed types (records and arrays) aren't on the list is a much larger troublemaker. (Regardless, I'm curious about the applications of 13M inserts per minute on a synchronous single-computer database. Pushing a machine to that extreme looks very interesting. Is that application something you can share?)

mastax 5y ago
It's a data acquisition system which records various data collected out in the field by operators on standard laptops. They're frequently operating in places with no cell coverage so we can't "just" stream to the cloud, we store the data locally and then bulk upload later. Also want to be able to visualize the data on the laptops which may be larger than ram. It's not a particularly demanding use-case, especially with respect to querying. Could just use some serialization format, parquet maybe, but I know SQLite works reliably and has good libs. It's nice to have standard tools available, and I know no matter what changes I'll be able to read these databases in 20 years. Didn't have a lot of time to work on this project so I didn't spend much time evaluating alternatives.
- marcosdumay 5y ago
  Oh, that's interesting. I didn't think about batch importing local data. Yes, I imagine you want to minimize the time importing data, and so halving the size of your main data would speed things up by a sizeable amount.