4 ms·

Holy balls, talk about going out of your way to avoid syslog.

tptacek 15y ago
Syslog is a pile of shit, Ted. It's a relic. You clearly happen to love that relic, and I think you should find a way to place it just-so in a nicely lit alcove in your apartment. The rest of us should move on from it. I don't think less of you for admiring it. I have useless old things on display in my house too. * Freeform text is a terrible way to track system events. * Periodically rotated flat files are not a great way to store log information. * Goofy little UDP messages are not a good way to convey system events * The syslog PRI field dates back to when we exchanged messages with UUCP. I could keep going, but since you're just going to reply with "lolwut umad?", I'll leave it at that.
- zbailey 15y ago
  We use syslog where I work, and I've always felt the same way, but never heard any suggestions for better options with as wide adoption, support, and background as syslog. Out of pure curiosity, what do you see as the tool most likely to displace syslog in the future? Is there any alternative available that fixes most of these problems without rolling your own from pieces and parts?
  - tptacek 15y ago
    I think we're stuck with syslog, but for app logging, ad-hoc database storage --- especially if you have either (a) a database optimized for message queueing or (b) a schemaless database --- is going to tend to beat syslog. You're not going to realize it until you need the information you're logging, though; until then, it's going to seem like syslog is everything you could reasonably need.
- tedjdziuba 15y ago
  OK Thomas. First of all: u mad bro? Second of all: maybe on the day when you build something substantive, useful, and valuable, you can lecture me. Until then, take a knee and lend your ear. You know what else is a "relic of the past"? The internal combustion engine. It's wildly inefficient, but somehow, people keep on using it because it solves the problem. Maybe not optimally, but satisfactory. More to the point: * freeform text is an _excellent_ way to track system events. Know why? It's human readable. Anything more is a micro-optimization that is perpetrated by people who have never had to debug a serious production system. * Periodic rotation is fine, if you are storing a lot of "chatter" type logs. If you have critical events that need more careful tracking, then wowie zowie, syslog can do that too. It's almost like those Unix guys knew what they were doing. * Modern syslog implementations can use TCP, if that's your thing. Dumb if you ask me, but whatever. * So? What problem does that create? If you want to chest-thump about technical masturbation, then you're in the right place. Hacker News Wantrepreneurs love that kind of stuff. However, the rest of us who have work to do will use the tools available that solve the problem. Nothing is perfect, but I've got better ways to spend my time than worry about making my own centralized logging system. I ain't even mad.
- Andys 15y ago
  There's syslog the protocol, syslog the API, and syslog the daemon. There are syslog daemon and protocol replacements that are much better, but retain the API for wide-spread compatibility.
  - tptacek 15y ago
    I think my contention would be that if you shake off the early-90s-Unixisms and adopt any modern store (forget Redis even and just "use sqlite instead of fopen"), the machinery required to do 98% of what the best syslog daemons do is trivial. The syslog infrastructure just doesn't add much... except compatibility with a very dubious ecosystem of tools.
- moe 15y ago
  Yes, syslog is a pile of shit. It's a relic. And I'll add it tends to ship in a horrible default configuration with events scattered randomly over multiple files, no safe-guards against filling up the disk and no safeguards to ensure the stupid daemon is actually running. However... Freeform text is a terrible way to track system events. Nothing stops you from logging structured text. Periodically rotated flat files are not a great way to store log information. Modern syslog daemons will write to pretty much anything you want. Goofy little UDP messages are not a good way to convey system events Modern syslog daemons offer tcp transport. Some even try to offer some delivery guarantees (disk-backed spool), although personally I wouldn't rely on that for truly critical stuff. The syslog PRI field dates back to when we exchanged messages with UUCP. Thanks, I always wondered where those were from... And, well, you forgot a couple bullets: * syslog() is available everywhere, out of the box * It's trivial to move from file-based logging to syslog * We have mature syslog-daemons that dispatch events pretty reliably * Unless you're facebook you probably don't need anything more fancy. So, I'd say syslog gets the job done quite well, as long as you don't mistake it for a message queue.
  - tptacek 15y ago
    1. It's not just that logs tend not to be structured, it's that the metadata is all inband. And syslog in particular exacerbates the problem by decoupling logging clients from log storage policy: the log generator has no idea if its syslog has a super-smart storage policy and so has to assume everything needs fine grained human readable timestamps, a custom facility/severity notation, the proctitle and pid, and so on. 2. Most syslog daemons will store anywhere. Sure. The last syslog daemon I actually read had stack overflows in it, so it's been awhile for me. But if you've got syslog writing to a real store, what value is syslog adding? The standardized network protocol? Redis has a trivial network protocol too. 3. The issue with syslog facilities isn't that the "UUCP" facility harms anything directly; it's that one the few bits of out-of-band metadata syslog truly offers forces applications to decide whether they're "kern" or "daemon" or "local2". Whose application actually breaks down cleanly into syslog facilities? 4. I dispute that syslog is "good enough if you're not Facebook". I think you do too: you probably aren't doing all your metrics stuff with syslog logs; you probably aren't even getting your web stats though syslog. You may have even delegated your web logs to client side Javascript, because that is how bad text logs stored in flat files are to work with.
m0nastic 15y ago
The only time in my life I've ever been envious of syslog was when I had to build an aggregator/event correlator for a bunch of telco equipment that only talked TL1. Pretty much any other time I've had to work with it, I've wished for something better, so I, for one, am very happy by the thought of people starting to "go out of their way to avoid syslog".

Syslog is a pile of shit, Ted. It's a relic. You clearly happen to love that relic, and I think you should find a way to place it just-so in a nicely lit alcove in your apartment. The rest of us should move on from it. I don't think less of you for admiring it. I have useless old things on display in my house too. * Freeform text is a terrible way to track system events. * Periodically rotated flat files are not a great way to store log information. * Goofy little UDP messages are not a good way to convey system events * The syslog PRI field dates back to when we exchanged messages with UUCP. I could keep going, but since you're just going to reply with "lolwut umad?", I'll leave it at that.

zbailey 15y ago
We use syslog where I work, and I've always felt the same way, but never heard any suggestions for better options with as wide adoption, support, and background as syslog. Out of pure curiosity, what do you see as the tool most likely to displace syslog in the future? Is there any alternative available that fixes most of these problems without rolling your own from pieces and parts?
- tptacek 15y ago
  I think we're stuck with syslog, but for app logging, ad-hoc database storage --- especially if you have either (a) a database optimized for message queueing or (b) a schemaless database --- is going to tend to beat syslog. You're not going to realize it until you need the information you're logging, though; until then, it's going to seem like syslog is everything you could reasonably need.
tedjdziuba 15y ago
OK Thomas. First of all: u mad bro? Second of all: maybe on the day when you build something substantive, useful, and valuable, you can lecture me. Until then, take a knee and lend your ear. You know what else is a "relic of the past"? The internal combustion engine. It's wildly inefficient, but somehow, people keep on using it because it solves the problem. Maybe not optimally, but satisfactory. More to the point: * freeform text is an _excellent_ way to track system events. Know why? It's human readable. Anything more is a micro-optimization that is perpetrated by people who have never had to debug a serious production system. * Periodic rotation is fine, if you are storing a lot of "chatter" type logs. If you have critical events that need more careful tracking, then wowie zowie, syslog can do that too. It's almost like those Unix guys knew what they were doing. * Modern syslog implementations can use TCP, if that's your thing. Dumb if you ask me, but whatever. * So? What problem does that create? If you want to chest-thump about technical masturbation, then you're in the right place. Hacker News Wantrepreneurs love that kind of stuff. However, the rest of us who have work to do will use the tools available that solve the problem. Nothing is perfect, but I've got better ways to spend my time than worry about making my own centralized logging system. I ain't even mad.
Andys 15y ago
There's syslog the protocol, syslog the API, and syslog the daemon. There are syslog daemon and protocol replacements that are much better, but retain the API for wide-spread compatibility.
- tptacek 15y ago
  I think my contention would be that if you shake off the early-90s-Unixisms and adopt any modern store (forget Redis even and just "use sqlite instead of fopen"), the machinery required to do 98% of what the best syslog daemons do is trivial. The syslog infrastructure just doesn't add much... except compatibility with a very dubious ecosystem of tools.
moe 15y ago
Yes, syslog is a pile of shit. It's a relic. And I'll add it tends to ship in a horrible default configuration with events scattered randomly over multiple files, no safe-guards against filling up the disk and no safeguards to ensure the stupid daemon is actually running. However... Freeform text is a terrible way to track system events. Nothing stops you from logging structured text. Periodically rotated flat files are not a great way to store log information. Modern syslog daemons will write to pretty much anything you want. Goofy little UDP messages are not a good way to convey system events Modern syslog daemons offer tcp transport. Some even try to offer some delivery guarantees (disk-backed spool), although personally I wouldn't rely on that for truly critical stuff. The syslog PRI field dates back to when we exchanged messages with UUCP. Thanks, I always wondered where those were from... And, well, you forgot a couple bullets: * syslog() is available everywhere, out of the box * It's trivial to move from file-based logging to syslog * We have mature syslog-daemons that dispatch events pretty reliably * Unless you're facebook you probably don't need anything more fancy. So, I'd say syslog gets the job done quite well, as long as you don't mistake it for a message queue.
- tptacek 15y ago
  1. It's not just that logs tend not to be structured, it's that the metadata is all inband. And syslog in particular exacerbates the problem by decoupling logging clients from log storage policy: the log generator has no idea if its syslog has a super-smart storage policy and so has to assume everything needs fine grained human readable timestamps, a custom facility/severity notation, the proctitle and pid, and so on. 2. Most syslog daemons will store anywhere. Sure. The last syslog daemon I actually read had stack overflows in it, so it's been awhile for me. But if you've got syslog writing to a real store, what value is syslog adding? The standardized network protocol? Redis has a trivial network protocol too. 3. The issue with syslog facilities isn't that the "UUCP" facility harms anything directly; it's that one the few bits of out-of-band metadata syslog truly offers forces applications to decide whether they're "kern" or "daemon" or "local2". Whose application actually breaks down cleanly into syslog facilities? 4. I dispute that syslog is "good enough if you're not Facebook". I think you do too: you probably aren't doing all your metrics stuff with syslog logs; you probably aren't even getting your web stats though syslog. You may have even delegated your web logs to client side Javascript, because that is how bad text logs stored in flat files are to work with.

m0nastic 15y ago

The only time in my life I've ever been envious of syslog was when I had to build an aggregator/event correlator for a bunch of telco equipment that only talked TL1. Pretty much any other time I've had to work with it, I've wished for something better, so I, for one, am very happy by the thought of people starting to "go out of their way to avoid syslog".