4 ms·

If you are archiving logs by first compressing them using something like gzip, you no longer have plain text log files. Well yeah, but that's because I'm alrea

If you are archiving logs by first compressing them using something like gzip, you no longer have plain text log files. Well yeah, but that's because I'm already done parsing the log and/or using it for debugging. If I wasn't done with it, I wouldn't have archived it. But I know that when I pull a 3 year old log out of the archive, I can gunzip it and will still be able to read it, no matter how much the file format has changed since then, and I don't have to dig up a 3 year old magic decoder program (that may only run on one operating system) to be able to see what's in the file.

kiallmacinnes 10y ago
This kinda feels like a really really minor point to the article, the rest of the article - revolving around parseability of logs is a REALLY important part of running any production infrastructure. Anyone who's ran production infra for long enough has written all the regexes, mostly from scratch, mostly badly. Myself included. None of my parsers would have caught the ssh 'root from 8.8.8.8'@server log. Applications that offer structured logging are superior (in terms of logging), and we need to ensure we build applications that support this going forward. For those in the python world, https://pypi.python.org/pypi/python-logstash https://pypi.python.org/pypi/python-logstash is great. It's somewhat undocumented, but it can be used without logstash, writing python logs with their original arguments to JSON on disk (which we then had an agent ship to logstash, because, disk turns out to be a decent buffer for handling network glitches). e.g. LOG.error("Invalid user %(user)s", {"user": "Kiall"}) can be logged as "ERROR: Invalid user Kiall", or: {"message": "Invalid user Kiall", "level": "ERROR", "extras": {"user": "Kiall"}}. I'd love to get the raw uninterpolated message in there too, so that I could match on that, then show the formatted message, while allowing filtering on the extras.
wtbob 10y ago
> But I know that when I pull a 3 year old log out of the archive, I can gunzip it and will still be able to read it, no matter how much the file format has changed since then, and I don't have to dig up a 3 year old magic decoder program (that may only run on one operating system) to be able to see what's in the file. Oh, but you can rest assured that systemd will never, ever have a breaking change. Lennart Poettering can be trsuted to never do something that would cause anybody any trouble! /s And that's the problem: systemd is a great idea, and it does a lot of really great things, but the man in charge simply can't be trusted not to break things. It was one thing when he was breaking desktops with PulseAudio; it's an entirely different thing now that his decisions can break servers (how many systems have broken due to nohup no longer doing its job?), and break the ability to retrieve logs over time. I honestly think that systemd is going to arrive at a good place eventually, and a great deal of credit will belong to Poettering. But it will also cause a great deal of harm before it gets there, and he deserves a great deal of the blame.
- brainfire 10y ago
  Can you summarize how he "[broke] desktops with PulseAudio"? From my perspective as an end user it didn't seem to go badly at all. Was I insulated by my distribution maintainers?
  - wtbob 10y ago
    > Can you summarize how he "[broke] desktops with PulseAudio"? For a long while it was flaky and buggy, with audio periodically failing. It was never the end of the world, but it was annoying, and I think it runs decently enough now.
- justinsaccount 10y ago
  > how many systems have broken due to nohup no longer doing its job Zero? As far as I know the change that defaulted KillUserProcesses to yes was reverted by every distribution that ships systemd.
pfranz 10y ago
Not only that, but zcat is standard on every system I've tried (which, admittedly, isn't very many) and vim reads gzipped files as if they're plaintext. Last time I was writing a logging system the only reason I couldn't write directly to gzipped files was that I couldn't append to a gzipped file using Python (fairly important in logging). It looked like the reason was a Window's specific workaround having to do with seeking from the end of the file.
parenthephobia 10y ago
The options aren't freeform text or an opaque proprietary binary format. The problems discussed in the article could be solved with any structured data format. Logs could be streams of objects encoded using e.g. JSON, bencoding, or protocol buffers.

kiallmacinnes 10y ago

This kinda feels like a really really minor point to the article, the rest of the article - revolving around parseability of logs is a REALLY important part of running any production infrastructure. Anyone who's ran production infra for long enough has written all the regexes, mostly from scratch, mostly badly. Myself included. None of my parsers would have caught the ssh 'root from 8.8.8.8'@server log. Applications that offer structured logging are superior (in terms of logging), and we need to ensure we build applications that support this going forward. For those in the python world, https://pypi.python.org/pypi/python-logstash https://pypi.python.org/pypi/python-logstash is great. It's somewhat undocumented, but it can be used without logstash, writing python logs with their original arguments to JSON on disk (which we then had an agent ship to logstash, because, disk turns out to be a decent buffer for handling network glitches). e.g. LOG.error("Invalid user %(user)s", {"user": "Kiall"}) can be logged as "ERROR: Invalid user Kiall", or: {"message": "Invalid user Kiall", "level": "ERROR", "extras": {"user": "Kiall"}}. I'd love to get the raw uninterpolated message in there too, so that I could match on that, then show the formatted message, while allowing filtering on the extras.

wtbob 10y ago

> But I know that when I pull a 3 year old log out of the archive, I can gunzip it and will still be able to read it, no matter how much the file format has changed since then, and I don't have to dig up a 3 year old magic decoder program (that may only run on one operating system) to be able to see what's in the file. Oh, but you can rest assured that systemd will never, ever have a breaking change. Lennart Poettering can be trsuted to never do something that would cause anybody any trouble! /s And that's the problem: systemd is a great idea, and it does a lot of really great things, but the man in charge simply can't be trusted not to break things. It was one thing when he was breaking desktops with PulseAudio; it's an entirely different thing now that his decisions can break servers (how many systems have broken due to nohup no longer doing its job?), and break the ability to retrieve logs over time. I honestly think that systemd is going to arrive at a good place eventually, and a great deal of credit will belong to Poettering. But it will also cause a great deal of harm before it gets there, and he deserves a great deal of the blame.

brainfire 10y ago
Can you summarize how he "[broke] desktops with PulseAudio"? From my perspective as an end user it didn't seem to go badly at all. Was I insulated by my distribution maintainers?
- wtbob 10y ago
  > Can you summarize how he "[broke] desktops with PulseAudio"? For a long while it was flaky and buggy, with audio periodically failing. It was never the end of the world, but it was annoying, and I think it runs decently enough now.
justinsaccount 10y ago
> how many systems have broken due to nohup no longer doing its job Zero? As far as I know the change that defaulted KillUserProcesses to yes was reverted by every distribution that ships systemd.

pfranz 10y ago

Not only that, but zcat is standard on every system I've tried (which, admittedly, isn't very many) and vim reads gzipped files as if they're plaintext. Last time I was writing a logging system the only reason I couldn't write directly to gzipped files was that I couldn't append to a gzipped file using Python (fairly important in logging). It looked like the reason was a Window's specific workaround having to do with seeking from the end of the file.

parenthephobia 10y ago

The options aren't freeform text or an opaque proprietary binary format. The problems discussed in the article could be solved with any structured data format. Logs could be streams of objects encoded using e.g. JSON, bencoding, or protocol buffers.