5 ms·
When you have fleets of thousands of machines performing a given service, and thousands of services implementing your product, you simply must centralize and in
by mypalmike 5y ago
When you have fleets of thousands of machines performing a given service, and thousands of services implementing your product, you simply must centralize and index logs if you want to have any chance of managing outages.
- jeffbee 5y agoExperience suggests otherwise.
- mypalmike 5y agoInteresting. What company have you worked for that runs hundreds of thousands of servers without centralized logging?
- RhodesianHunter 5y agoNot the person you replied to, but basically airlines, banks and other financial institutions, most things government or DOD are largely run this way.
- avl999 5y agoI worked on a tier 1 service at Amazon with over 1500 hosts in the fleet. Logs within the hour would be on the host and we would literally just grep through them, to grep through logs on a subset of the fleet or the entire fleet there was a simple utility that would ssh into multiple prod host and run grep in parallel. For logs older than an hour the logs would go gzipped into storage and the we zgrep'd though them. No centralized logging (unless you call logs being rotated off the host at end of the hour) and definitely no indexing. Never had any problems managing outages. Grep alone can go a long way in managing outages.
- jeffbee 5y agoAll the resources are at the edges of your infrastructure, so distributed grep makes more and more sense the larger your installation becomes. Centralized logging looks less and less economic as things expand.
- romantomjak 5y agoThanks, this is a great insight. I haven’t thought about it in this way and it sounds like a very pragmatic approach without over complicating things.
- closeparen 5y agoDid you ever worry about those greps interfering with production performance?
- jeffbee 5y agoDo you ever worry that your logs exporting agent interferes with service performance? After all, that goes on continuously in a setup like the one in the article, rather than as-needed in a distributed predicate evaluation setup.
- closeparen 5y agoA constant stable performance drag is going to show up in your load tests, capacity planning, etc. rather than in surprising intermittent degradations. Also our log collector agents run in containers like everything else, so there is some amount of resource isolation (not perfect of course).