3 ms·

One problem I can think of with binary logging would be catastrophe. I have had to pull HDs, go through the logs, line by line and piece together what happened

One problem I can think of with binary logging would be catastrophe. I have had to pull HDs, go through the logs, line by line and piece together what happened in those final moments. Text logs are easy to get at, I don’t need a special program to read, I just need something to display a raw byte on the screen. Also, I wonder what the actual performance benefits are. I remember back when most protocols were text and/or binary. Usually binary was slower when messing about with stringy stuff. For example, using memcached to cache strings. Why encode/decode strings in a binary protocol when you can just use strings all the way down? I don’t know if that premise is still true, but for us older people, it makes sense to keep inherently stringy stuff stringy to avoid overhead.

lavabiopsy 5y ago
>Text logs are easy to get at, I don’t need a special program to read I've seen this type of comment a lot on reddit and I don't understand why people say this. You do not really need a special program to read the systemd logs either, in the event that journalctl doesn't work you can easily use `strings` to read the logs. Feel free to try it right now just for fun, the binary format of journald is actually extremely simple and well-documented and just stores the logs sequentially, it's not "encoding" the strings at all. Why would it need to? It's all fulltext data.
kaba0 5y ago
Re the failure part: you can just plug in any USB stick with a normal linux distro and it will read it out just fine. Re performance: Strings are most defintely worse than some fixed size format. It’s not an accident that FPGAs were used to serialize/deserialize XMLs, why Google made a whole new binary protocol, etc. Just think about it, you have to parse a string character by character (yeah some clever vector magic can make some string processing go a few chars at a time), while a binary format with fixed layout will be practically random access. You can query the nth entry, etc.
- withinboredom 5y ago
  You also have to parse a binary protocol byte by byte, or at least the bytes you care about. But the question is, why are we parsing logs at all?
  - lavabiopsy 5y ago
    Journald doesn't actually parse the log strings. Any parseable entities are stored in separate fields (see `man systemd.journal-fields`)
olau 5y ago
I sort of agree, but I think in the end, your recovery thing will have to include the decoder and that's it. As I recall, there were two reasons for the binary format - a signing scheme to make it tamper proof and more metadata on each log entry.

lavabiopsy 5y ago

>Text logs are easy to get at, I don’t need a special program to read I've seen this type of comment a lot on reddit and I don't understand why people say this. You do not really need a special program to read the systemd logs either, in the event that journalctl doesn't work you can easily use `strings` to read the logs. Feel free to try it right now just for fun, the binary format of journald is actually extremely simple and well-documented and just stores the logs sequentially, it's not "encoding" the strings at all. Why would it need to? It's all fulltext data.

kaba0 5y ago

Re the failure part: you can just plug in any USB stick with a normal linux distro and it will read it out just fine. Re performance: Strings are most defintely worse than some fixed size format. It’s not an accident that FPGAs were used to serialize/deserialize XMLs, why Google made a whole new binary protocol, etc. Just think about it, you have to parse a string character by character (yeah some clever vector magic can make some string processing go a few chars at a time), while a binary format with fixed layout will be practically random access. You can query the nth entry, etc.

withinboredom 5y ago
You also have to parse a binary protocol byte by byte, or at least the bytes you care about. But the question is, why are we parsing logs at all?
- lavabiopsy 5y ago
  Journald doesn't actually parse the log strings. Any parseable entities are stored in separate fields (see `man systemd.journal-fields`)

olau 5y ago

I sort of agree, but I think in the end, your recovery thing will have to include the decoder and that's it. As I recall, there were two reasons for the binary format - a signing scheme to make it tamper proof and more metadata on each log entry.