3 ms·
> Aggregation and expiry start to look a lot like dimensionality: they can be implemented asynchronously in a separate policy layer.It doesn't seem important t
by rdtsc 11y ago
> Aggregation and expiry start to look a lot like dimensionality: they can be implemented asynchronously in a separate policy layer.It doesn't seem important that the actual storage engine is mindful of these; in fact, it's probably better ignored for efficiency's sake.
I wrote my own time series database and actually having well working expiry was harder than expected. I sharded each one of the files into 2GB or 24h (whatever came first) chunks. Then had to be careful how they got deleted. There were various rules there such as ("make sure to not fill partition more than 80%", "stop when partition is full" or "keep data no more than 90 days").
My number of different time series was actually pretty low but each value was a record with complex data inside it i.e. not just simple ints or floats. For example, one series was capturing network packets so it looked like [ (timestamp,packet), (timestamp, packet), ... ] . But then an indexing service was running separately in a separate OS process to generate additional complex indexing from the primary data.
- MichaelGG 11y agoWhat industry or app was this for? I wrote something that sounds like this for VoIP packet capture. Though the rates were around 5TB/day. I'd keep the latest indexes in RAM before delta-encoding and writing out. Periodically, merge multiple indexes into one. Worked surprisingly well, and the indexing overhead was only a few bits per packet (given enough similarity in packets over a short chunk of time (say, 1 minute), then just indexing unique values per chunk -- it was all fairly efficient.)
- rdtsc 11y agoVoIP capture was one of the functions in the system too. But it had others. It looks like your implementation is more sophisticated, I didn't do any delta-encoding. Just periodic fsyncs. The maximum we tested was up to 1TB/day. Main index ( [(timestamp,offset_in_data)] was simply synced every second. It just looked like a [uint64|uint32|...]. Then there is an advanced service with different plugins, runs in a separate process, trails the main data stream. Each plugin can then write out its own custom index data to file and is also saving its internal state (so it can recover during a hard crash).
- MichaelGG 11y agoEh well mine was mostly a hack to index SIP packets. Other existing products just shoved every packet into a MySQL table. One column per field, with indexes. You can imagine that does not scale very well. So I ended up doing similar to what you describe. Fun how designs converge. You'd think there'd be some generic libraries for this kind of stuff. Now there's LevelDB and its SSTable is sort of close but uses Snappy so it can't operate on compressed data. What's the common library that implements fast, compressed, immutable key values without lots of per record overhead?