4 ms·
Plaintext is not a great format for (system) logs
- 7thaccount 2y agoYou can make it all some kind of json object and use plenty of tools to parse. I think text is really awesome though. You can read it with any editor and there are a zillion easy to learn Linux tools like grep, cut, awk...etc for interactively creating a oneliner in a short amount of time that you can use to get what you want out of the text. I used to have a bunch of those when I was in charge of some applications on a Linux server. Text is very universal and I'm not just talking about log files.
- imglorp 2y agoMissed opportunity: it would really be a different conversation if, around 1980 or so, syslog(3) became a standard and was adopted at that time by everyone. Logs really would be in much the same format and you could use simple text tools on them. After you agreed on the standard parts timestamp, facility, and level, you'd still need some standard structure for the variant message parts though, which is where json is good: it's arbitrarily deep. https://www.man7.org/linux/man-pages/man3/syslog.3.html https://www.man7.org/linux/man-pages/man3/syslog.3.html
- fareesh 2y agotext is great for awk, grep, etc perhaps some llm-powered awk command generator would be useful?
- crest 2y agoif only syslog() supported something like libxo...
- deleted 2y ago[deleted]
- exabrial 2y agoplaintext and grep were find for your grandpappy and they're fine for you.
- likeabatterycar 2y agoAbsolutely nothing wrong with binary logs as long as the system provides a utility to get the data out of that format. For that matter, journalctl is terrible, cryptic, and difficult to use.
- crest 2y agoWhen your "compressed database" is both larger and slower to query than the plaintext, has an unstable on disk format, and isn't even crash safe like journald's database I have to assume it's a sick joke and not a serious upgrade from plaintext logs on a (compressing) file system.
- lucianbr 2y agoOn the other hand, multiple competing binary log formats make it harder to process the metadata than plaintext.
- JackSlateur 2y agotldr: the author confuses plain text (aka, storing in ascii files) with storing in some binary format (whatever it is). Said confusion makes the whole post worthless, in my opinion.
- tomsmeding 2y agoWhere is this confusion made? The author compares syslog-style logging (plain text, with some tools grafting on ad-hoc metadata embedding formats) with systemd-style journal logging (binary with special tools to read them).
- JackSlateur 2y agoA log entry is composed of many things: some event nature, a user, who sent the entry, when it happened etc The core problem is that log messages themselves almost invariably come with additional metadata, often fairly rich metadata, but if you store things in plain text it's difficult to handle that metadata So this is wrong: there are few "metadata" with logs : the "metadata" are part of the log, are stored with the log, they are the actual data for which we generated the log. It has nothing to do with "plain text": structured-logging is a thing, and it is plain text. And writing random english sentences to a mysql database is also a thing, and it is not plain text. And you can move from one to another, just as you can compress a logfile (which is then no longer plain text) etc. So the plain text vs binary is a format issue, and the logs' content is another, unrelated topic So I do not know what's the point here, except saying "journald is awesome", which is just an opinion
- tomsmeding 2y agoThis is fair. However, I think the situation the OP is complaining about is not one application that does structured logging (where I agree with you that even if the data is "plain text" in some sense, it's not terribly productive to think of it as plain text), but instead about a system log that many unrelated applications write to. Each of those applications has their own way of structuring their logs: there is no standard. Hence you end up with something that is technically plain text, but also really nothing more because there is no consistent structure. So it's really just plain text.
- PreInternet01 2y agoAu contraire, plain text is just about the only format for logs that works. Having to use some obscure tool to query logs just isn't worth it, especially not in 20 years time, when you still need to analyze said logs but the tool doesn't even run on any contemporary systems anymore and the documentation about the binary format has been irretrievably lost. And yeah, sure, you want to use structured logging. So, in addition to the greppable log message, include a Magic Separator Character (say, \t) that you treat as 'end-of-line' in human-oriented processing, and have your key-value-pair structured data following that for automated tooling to have its way with. Or, be really creative with [key=value] blocks that are both human- and machine-readable. Some of my most painful logging experiences were having to extract application logs from binary blobs created by a proprietary Windows tool (no, not the main Windows event log: that's bad, but at least documented). Even the most recent versions of the official viewer just crashed, the vendor was not interested in fixing that (since we were not a customer, just a third party in need of log data, but even an offer of a modest payment was met with indifference...), and the actual format turned out to be byzantine beyond words. So, yeah, give me plain text all day, every day...
- jmclnx 2y agoThe title should be "Plaintext is not a great format for (system) logs for people have no idea about regexp" :) In my opinion, plain text wins for everything, very portable and people can read with very little knowledge.