3 ms·

Reading data via index would lead to scattered memory reads on actual data, which is a dead-end from an optimisation standpoint unfortunately. Using index woul

Reading data via index would lead to scattered memory reads on actual data, which is a dead-end from an optimisation standpoint unfortunately. Using index would have been much simpler but also slower.

IgorPartola 5y ago
That’s assuming you’ll constantly be doing reads of only the last 10 seconds of data. If that’s the case, this isn’t the database you should use (or even the data model). If you are tracking vehicles or IoT devices or whatnot the reason you’d do those kinds of reads is because you want to know where things are right now, so using a data store optimized for knowing where they used to be at some arbitrary point in time is not the right tool. If you want to keep track of the latest state all the time store it in a another data structure instead. Once you flush your input buffer, everything becomes sequential so after your 10 second window you no longer have this problem. And even for the latest 10 seconds reading an index won’t be terrible as far as performance because your input buffer is relatively small (compared to your huge dataset).
- jstrong 5y ago
  the design you propose is to write 10sec chunks of data to an append-only log, which will contain chunks from multiple tables/series? if so, then reading larger ranges than 10sec requires scattered reads of a bunch of small chunks interspersed in the log file. that doesn't sound like it will be fast? the other big tradeoff is compression ratio, which writing data in 10sec chunks (as its final form) would be terrible for.
  - IgorPartola 5y ago
    How you actually write data to the files is mostly irrelevant. You could write each column of each table into a separate file or a file per table. I would probably further partition each file by date with some relatively large range (day or week or month). I don’t see why you’d ever want to write multiple tables into one file. Not sure why you’d think of this as chunks of 10 seconds, that’s just the size of the input buffer. The records aren’t related to each other beyond that or grouped in any way. As far as compaction, I suspect for the kinds of workloads you are using this kind of storage for you won’t get much in terms of deduplication. IoT devices sending sensor readings are unlikely to produce loads of stable readings and the time stamps will be ever increasing.

IgorPartola 5y ago

That’s assuming you’ll constantly be doing reads of only the last 10 seconds of data. If that’s the case, this isn’t the database you should use (or even the data model). If you are tracking vehicles or IoT devices or whatnot the reason you’d do those kinds of reads is because you want to know where things are right now, so using a data store optimized for knowing where they used to be at some arbitrary point in time is not the right tool. If you want to keep track of the latest state all the time store it in a another data structure instead. Once you flush your input buffer, everything becomes sequential so after your 10 second window you no longer have this problem. And even for the latest 10 seconds reading an index won’t be terrible as far as performance because your input buffer is relatively small (compared to your huge dataset).

jstrong 5y ago
the design you propose is to write 10sec chunks of data to an append-only log, which will contain chunks from multiple tables/series? if so, then reading larger ranges than 10sec requires scattered reads of a bunch of small chunks interspersed in the log file. that doesn't sound like it will be fast? the other big tradeoff is compression ratio, which writing data in 10sec chunks (as its final form) would be terrible for.
- IgorPartola 5y ago
  How you actually write data to the files is mostly irrelevant. You could write each column of each table into a separate file or a file per table. I would probably further partition each file by date with some relatively large range (day or week or month). I don’t see why you’d ever want to write multiple tables into one file. Not sure why you’d think of this as chunks of 10 seconds, that’s just the size of the input buffer. The records aren’t related to each other beyond that or grouped in any way. As far as compaction, I suspect for the kinds of workloads you are using this kind of storage for you won’t get much in terms of deduplication. IoT devices sending sensor readings are unlikely to produce loads of stable readings and the time stamps will be ever increasing.