11 ms·

Guide to Logging

205 points by makaimc 11y ago

mostlyjason 11y ago
Hey guys I'm one of the technical editors for the guide. We created it because logging is really valuable for troubleshooting and monitoring systems. However, it's a complex topic and so many resources today are spread around on different forums. We wanted to have a place to consolidate all that useful information into something much easier to read. If it helps a dev or ops person solve a problem faster or easier than we've done our job! There is still a lot of content to be written for languages like python, ruby, and more. We'd love to have more contributors, or even just suggestions for the site. I can also get in touch with any of the authors if you have questions for them.
- kordless 11y ago
  > many resources today are spread around on different forums Centralizing the all the knowledge that's been written about a given topic isn't scalable in terms of contributions. It's far better to focus on enabling contributions than it is to focus on building a solution in which to put the contributions. There's a reason products like StackExchange are more successful at providing features for encouraging contributions than topic specific sites. > There is still a lot of content to be written for languages like python, ruby, and more.. We'd love to have more contributors, There is already a ton of contributions to logging solutions on the Internet. Why not build a solution categorizing those, instead of trying to build new content in a centralized (and I might add commercially curated) solution? Also, where's the source to this "open source" project? I'm not seeing any links to code.
  - mostlyjason 11y ago
    I think readability is a concern. Often when I'm learning a new technology I'd rather sit down with a good book on the topic rather than reading a bunch of forum posts. I can be pretty confident that it will cover the most important concepts, building up from basics to more advanced ones, and showing all the connections in between. A lot of times reading regular posts, you have to build all these connections yourself. We do link to really good resources and documentation where they are available though so if you have any to suggest please let me know! I think the open source part of the title refers to the fact that we cover all different technologies from a variety of perspectives, not just a single vendor solution. Also, we allow contributions from anyone in the community. I couldn't find an easy way for WordPress to share or collaborate on source code, but if you think of a way, let me know. In the meantime, we're giving access to people who want to contribute.
    - kordless 11y ago
      I'd suggest building the content off a Github repo, rendering from markdown files. This solves the problem of contribution enablement. Rackspace has a docs/developer project that I was involved with for a very short time which has its source code available for use. Start there and get your content moved off Wordpress. Nothing against Wordpress, but it's not built for this. Also, code snippets. Rackspace site on Github: https://github.com/rackerlabs/developer.rackspace.com https://github.com/rackerlabs/developer.rackspace.com I agree with the readability part. It's important to curate from the stance of educating the multitudes quickly. Having disparate content can be an issue in starting the education, but variety is the spice of life - and use cases as time goes on. I'd recommend content + reference as a solution here. Cookbook style, if you will. Keep in mind I've spent a fair amount of clock cycles thinking about this than most people. I've rejected things like logging standards as well. That doesn't mean I'm right, but it is, at the least, a data point for you.
jedberg 11y ago
I have a long rant about logging, but the TL;DR is: aggregated logging in general is useless don't bother, everything you need to know should be in your monitoring system. Basically, if something is broken and it's a problem, it will happen again, so you just start logging when something breaks. How do you know something breaks? Log your application metrics. Log your customer experience in the form of metrics and monitoring. But don't aggregate every code exception and warning. If they aren't affecting your customer experience, who cares? And if they are affecting it, then it should be expressed as an application metric in your monitoring system. Edit: To make my point a little clearer, I believe that central logging and aggregating should be available, just not turned on all the time.
- teacup50 11y ago
  What? The only way to find many issues is by looking at logs of unexpected (and especially unhandled) exceptions. If you have so many of those that they're impossible to dig through, FIX YOUR CODE.
  - jedberg 11y ago
    You can keep the logs on the local box and take a peek at them once in a while if you want to clean up the code (which you should), but there is no reason to spend a ton of money and time aggregating the logs.
    - seiji 11y ago
      Centralized logging is critical to managing security. No real world production systems should be logging locally.
      - jedberg 11y ago
        The longer version of my rant stipulates that you have a good application firewall that is looking at the incoming traffic and you are using immutable infrastructure for the fronted so an attacker can't do much damage if they do get in. Keep in mind I consider the existence of exceptions to be an application metric that should be logged, so if there is a security issue causing exceptions that should show up in the monitoring, and then you can look at the exceptions that happen going forward. If the box is compromised, your local IDS should catch that (assuming you are allowing writes on the box at all)
mirimir 11y ago
I find nothing on tor or lighttpd :(
avn2109 11y ago
Was I the only one who was really hoping this was going to be about best practices for cutting down trees?
acheron 11y ago
My four-year-old watches this show, whence I've learned all about logging: http://www.amazon.com/Mighty-Machines-In-The-Forest/dp/B00008YZ0E http://www.amazon.com/Mighty-Machines-In-The-Forest/dp/B0000...
- 0xdeadbeefbabe 11y ago
  The subtext that I've just started to notice, as I watch with my four year old, is that Canada is awesome.
  - DKnoll 11y ago
    I'm going to take this even further off-topic but if you enjoy cheesy vintage Canadian children's television Hammy Hamster AKA Tales of the Riverbank is required viewing.
deleted 11y ago
[deleted]
- gk1 11y ago
  It's built on Wordpress, so yes. http://www.wordpress.org http://www.wordpress.org
  - deleted 11y ago
    [deleted]
mugsie 11y ago
https://www.loggly.com/ultimate-guide/?s=parse+WSGI+logs&post_type=post https://www.loggly.com/ultimate-guide/?s=parse+WSGI+logs&pos... 0 results https://www.loggly.com/ultimate-guide/?s=parse+python+logs&post_type=post https://www.loggly.com/ultimate-guide/?s=parse+python+logs&p... Gives you a page on NodeJS Ultimate ... not so much.
- mentat 11y ago
  For even "haproxy". Surely no one uses that right?
  - mugsie 11y ago
    did a quick scan - nginx just returns generic "linux" / "systemd" (and node for some reason). ruby - you guessed it - node.
    - mostlyjason 11y ago
      I'm one of the tech editors for the guide and we'd love to have contributors for nginx and ruby! http://www.loggly.com/ultimate-guide/become-a-contributor/ http://www.loggly.com/ultimate-guide/become-a-contributor/
- jt2190 11y ago
  I've downvoted you because the world doesn't need yet another post about how some open source project that has the audacity to call themselves "the ultimate resource" for their initial release doesn't live up to some-random-person-on-the-internet's idea of "ultimate". I'm guessing from the specifics of your example queries that you already have some knowledge about WSGI and Python that you could contribute to the guide. Why not do that instead?
  - mostlyjason 11y ago
    Great suggestion! I'm one of the tech editors for the guide and we'd love to have contributors for python and other languages! http://www.loggly.com/ultimate-guide/become-a-contributor/ http://www.loggly.com/ultimate-guide/become-a-contributor/
strickjb9 11y ago
Your site's readability would be greatly improved if you darkened the text color. #444 or darker is good enough for a white background.
DKnoll 11y ago
Loggly is crazy expensive for what they have on offer (at least given my use case which is aggregating syslog and Windows event logs). You're much better setting up Logstash or Graylog2 on-premises or in public cloud.
packetized 11y ago
'Become a Contributor' seems to mean 'please give us your email address to be added to our mailing list.' This seems moderately suspect.

205 points by makaimc 11y ago

Hey guys I'm one of the technical editors for the guide. We created it because logging is really valuable for troubleshooting and monitoring systems. However, it's a complex topic and so many resources today are spread around on different forums. We wanted to have a place to consolidate all that useful information into something much easier to read. If it helps a dev or ops person solve a problem faster or easier than we've done our job! There is still a lot of content to be written for languages like python, ruby, and more. We'd love to have more contributors, or even just suggestions for the site. I can also get in touch with any of the authors if you have questions for them.

kordless 11y ago
> many resources today are spread around on different forums Centralizing the all the knowledge that's been written about a given topic isn't scalable in terms of contributions. It's far better to focus on enabling contributions than it is to focus on building a solution in which to put the contributions. There's a reason products like StackExchange are more successful at providing features for encouraging contributions than topic specific sites. > There is still a lot of content to be written for languages like python, ruby, and more.. We'd love to have more contributors, There is already a ton of contributions to logging solutions on the Internet. Why not build a solution categorizing those, instead of trying to build new content in a centralized (and I might add commercially curated) solution? Also, where's the source to this "open source" project? I'm not seeing any links to code.
- mostlyjason 11y ago
  I think readability is a concern. Often when I'm learning a new technology I'd rather sit down with a good book on the topic rather than reading a bunch of forum posts. I can be pretty confident that it will cover the most important concepts, building up from basics to more advanced ones, and showing all the connections in between. A lot of times reading regular posts, you have to build all these connections yourself. We do link to really good resources and documentation where they are available though so if you have any to suggest please let me know! I think the open source part of the title refers to the fact that we cover all different technologies from a variety of perspectives, not just a single vendor solution. Also, we allow contributions from anyone in the community. I couldn't find an easy way for WordPress to share or collaborate on source code, but if you think of a way, let me know. In the meantime, we're giving access to people who want to contribute.
  - kordless 11y ago
    I'd suggest building the content off a Github repo, rendering from markdown files. This solves the problem of contribution enablement. Rackspace has a docs/developer project that I was involved with for a very short time which has its source code available for use. Start there and get your content moved off Wordpress. Nothing against Wordpress, but it's not built for this. Also, code snippets. Rackspace site on Github: https://github.com/rackerlabs/developer.rackspace.com https://github.com/rackerlabs/developer.rackspace.com I agree with the readability part. It's important to curate from the stance of educating the multitudes quickly. Having disparate content can be an issue in starting the education, but variety is the spice of life - and use cases as time goes on. I'd recommend content + reference as a solution here. Cookbook style, if you will. Keep in mind I've spent a fair amount of clock cycles thinking about this than most people. I've rejected things like logging standards as well. That doesn't mean I'm right, but it is, at the least, a data point for you.

jedberg 11y ago

I have a long rant about logging, but the TL;DR is: aggregated logging in general is useless don't bother, everything you need to know should be in your monitoring system. Basically, if something is broken and it's a problem, it will happen again, so you just start logging when something breaks. How do you know something breaks? Log your application metrics. Log your customer experience in the form of metrics and monitoring. But don't aggregate every code exception and warning. If they aren't affecting your customer experience, who cares? And if they are affecting it, then it should be expressed as an application metric in your monitoring system. Edit: To make my point a little clearer, I believe that central logging and aggregating should be available, just not turned on all the time.

teacup50 11y ago
What? The only way to find many issues is by looking at logs of unexpected (and especially unhandled) exceptions. If you have so many of those that they're impossible to dig through, FIX YOUR CODE.
- jedberg 11y ago
  You can keep the logs on the local box and take a peek at them once in a while if you want to clean up the code (which you should), but there is no reason to spend a ton of money and time aggregating the logs.
  - seiji 11y ago
    Centralized logging is critical to managing security. No real world production systems should be logging locally.
    - jedberg 11y ago
      The longer version of my rant stipulates that you have a good application firewall that is looking at the incoming traffic and you are using immutable infrastructure for the fronted so an attacker can't do much damage if they do get in. Keep in mind I consider the existence of exceptions to be an application metric that should be logged, so if there is a security issue causing exceptions that should show up in the monitoring, and then you can look at the exceptions that happen going forward. If the box is compromised, your local IDS should catch that (assuming you are allowing writes on the box at all)

mirimir 11y ago

I find nothing on tor or lighttpd :(

avn2109 11y ago

Was I the only one who was really hoping this was going to be about best practices for cutting down trees?

acheron 11y ago

My four-year-old watches this show, whence I've learned all about logging: http://www.amazon.com/Mighty-Machines-In-The-Forest/dp/B00008YZ0E http://www.amazon.com/Mighty-Machines-In-The-Forest/dp/B0000...

0xdeadbeefbabe 11y ago
The subtext that I've just started to notice, as I watch with my four year old, is that Canada is awesome.
- DKnoll 11y ago
  I'm going to take this even further off-topic but if you enjoy cheesy vintage Canadian children's television Hammy Hamster AKA Tales of the Riverbank is required viewing.

deleted 11y ago

[deleted]

gk1 11y ago
It's built on Wordpress, so yes. http://www.wordpress.org http://www.wordpress.org
- deleted 11y ago
  [deleted]

mugsie 11y ago

https://www.loggly.com/ultimate-guide/?s=parse+WSGI+logs&post_type=post https://www.loggly.com/ultimate-guide/?s=parse+WSGI+logs&pos... 0 results https://www.loggly.com/ultimate-guide/?s=parse+python+logs&post_type=post https://www.loggly.com/ultimate-guide/?s=parse+python+logs&p... Gives you a page on NodeJS Ultimate ... not so much.

mentat 11y ago
For even "haproxy". Surely no one uses that right?
- mugsie 11y ago
  did a quick scan - nginx just returns generic "linux" / "systemd" (and node for some reason). ruby - you guessed it - node.
  - mostlyjason 11y ago
    I'm one of the tech editors for the guide and we'd love to have contributors for nginx and ruby! http://www.loggly.com/ultimate-guide/become-a-contributor/ http://www.loggly.com/ultimate-guide/become-a-contributor/
jt2190 11y ago
I've downvoted you because the world doesn't need yet another post about how some open source project that has the audacity to call themselves "the ultimate resource" for their initial release doesn't live up to some-random-person-on-the-internet's idea of "ultimate". I'm guessing from the specifics of your example queries that you already have some knowledge about WSGI and Python that you could contribute to the guide. Why not do that instead?
- mostlyjason 11y ago
  Great suggestion! I'm one of the tech editors for the guide and we'd love to have contributors for python and other languages! http://www.loggly.com/ultimate-guide/become-a-contributor/ http://www.loggly.com/ultimate-guide/become-a-contributor/

strickjb9 11y ago

Your site's readability would be greatly improved if you darkened the text color. #444 or darker is good enough for a white background.

DKnoll 11y ago

Loggly is crazy expensive for what they have on offer (at least given my use case which is aggregating syslog and Windows event logs). You're much better setting up Logstash or Graylog2 on-premises or in public cloud.

packetized 11y ago

'Become a Contributor' seems to mean 'please give us your email address to be added to our mailing list.' This seems moderately suspect.