3 ms·
What industry or app was this for? I wrote something that sounds like this for VoIP packet capture. Though the rates were around 5TB/day. I'd keep the latest in
by MichaelGG 11y ago
What industry or app was this for? I wrote something that sounds like this for VoIP packet capture. Though the rates were around 5TB/day. I'd keep the latest indexes in RAM before delta-encoding and writing out. Periodically, merge multiple indexes into one.
Worked surprisingly well, and the indexing overhead was only a few bits per packet (given enough similarity in packets over a short chunk of time (say, 1 minute), then just indexing unique values per chunk -- it was all fairly efficient.)
- rdtsc 11y agoVoIP capture was one of the functions in the system too. But it had others. It looks like your implementation is more sophisticated, I didn't do any delta-encoding. Just periodic fsyncs. The maximum we tested was up to 1TB/day. Main index ( [(timestamp,offset_in_data)] was simply synced every second. It just looked like a [uint64|uint32|...]. Then there is an advanced service with different plugins, runs in a separate process, trails the main data stream. Each plugin can then write out its own custom index data to file and is also saving its internal state (so it can recover during a hard crash).
- MichaelGG 11y agoEh well mine was mostly a hack to index SIP packets. Other existing products just shoved every packet into a MySQL table. One column per field, with indexes. You can imagine that does not scale very well. So I ended up doing similar to what you describe. Fun how designs converge. You'd think there'd be some generic libraries for this kind of stuff. Now there's LevelDB and its SSTable is sort of close but uses Snappy so it can't operate on compressed data. What's the common library that implements fast, compressed, immutable key values without lots of per record overhead?