2 ms·
VoIP capture was one of the functions in the system too. But it had others. It looks like your implementation is more sophisticated, I didn't do any delta-encod
by rdtsc 11y ago
VoIP capture was one of the functions in the system too. But it had others. It looks like your implementation is more sophisticated, I didn't do any delta-encoding. Just periodic fsyncs. The maximum we tested was up to 1TB/day.
Main index ( [(timestamp,offset_in_data)] was simply synced every second. It just looked like a [uint64|uint32|...]. Then there is an advanced service with different plugins, runs in a separate process, trails the main data stream. Each plugin can then write out its own custom index data to file and is also saving its internal state (so it can recover during a hard crash).
- MichaelGG 11y agoEh well mine was mostly a hack to index SIP packets. Other existing products just shoved every packet into a MySQL table. One column per field, with indexes. You can imagine that does not scale very well. So I ended up doing similar to what you describe. Fun how designs converge. You'd think there'd be some generic libraries for this kind of stuff. Now there's LevelDB and its SSTable is sort of close but uses Snappy so it can't operate on compressed data. What's the common library that implements fast, compressed, immutable key values without lots of per record overhead?