Or serialize to one line per message (custom format, or JSONLINEs work too), and write them to text files, one file per topic per day (or per hour if your traf

Or serialize to one line per message (custom format, or JSONLINEs work too), and write them to text files, one file per topic per day (or per hour if your traffic is high). It's a bit more code, but look at the advantages: - Easy to examine data in any text editor - Reading is easy in every major programming language. - Easy to search for data given channel and time ranges (grep) - If you want to reprocess every bit of data in a single channel, it's very fast. No network involved at all, and you are getting a whole bunch of lines in a single read() syscall. - Simple and very fast to truncate old data, even if you want different truncation schedule per channel -- even if there is a lot of it. - Simple to backup: existing file-based backup solutions just work.