2 ms·
Honestly, I would go append-only blocks that contain binary/compressed format with synchronizing marks so worst case you get some nulls but every reader can syn
by p_l 2mo ago
Honestly, I would go append-only blocks that contain binary/compressed format with synchronizing marks so worst case you get some nulls but every reader can synchronize where they are in the stream without blocking anyone.
Might take more space on disk than theoretical best of journald storage format with its absurd hashtables, but it fulfills the job of system log better and more complex format should be done in log aggregation layer.
- pineapplepizza6 2mo agoYou could even generate a bloom filter for every block, and when you have a full block of bloom filters, write them out to an index file.
- p_l 2mo agoMy personal idle walking-with-dog kind of design was a linear binary record file with regular marks letting you resynchronize where you are (and stamp cryptographically) with minimal seeks, and separate indexing files with bloom filters and the like. If the indexes are corrupted or deleted, they can be reconstructed from the main log, main log is single-writer/multiple-readers with no locking in any form necessary, and easier to survive kernel/hw failure