4 ms·
The whole architecture looks insanely expensive. Putting your debug logs through scribe, through Kafka, then into Splunk, and indexing them, has got to cost som
by jeffbee 5y ago
The whole architecture looks insanely expensive. Putting your debug logs through scribe, through Kafka, then into Splunk, and indexing them, has got to cost something like fifty dollars per gigabyte.
I've always been a proponent over leaving logs where they were produced, not collecting them, and not indexing them, so an architecture like this I find just shocking.
- closeparen 5y agoCentralized logging is one of the many problems you sign up for once you opt for microservices on a cluster scheduler. Since service instances are ephemeral and cluster nodes only slightly less so, it doesn’t really work to leave the logs in place.
- mypalmike 5y agoWhen you have fleets of thousands of machines performing a given service, and thousands of services implementing your product, you simply must centralize and index logs if you want to have any chance of managing outages.
- jeffbee 5y agoExperience suggests otherwise.
- mypalmike 5y agoInteresting. What company have you worked for that runs hundreds of thousands of servers without centralized logging?
- RhodesianHunter 5y agoNot the person you replied to, but basically airlines, banks and other financial institutions, most things government or DOD are largely run this way.
- avl999 5y agoI worked on a tier 1 service at Amazon with over 1500 hosts in the fleet. Logs within the hour would be on the host and we would literally just grep through them, to grep through logs on a subset of the fleet or the entire fleet there was a simple utility that would ssh into multiple prod host and run grep in parallel. For logs older than an hour the logs would go gzipped into storage and the we zgrep'd though them. No centralized logging (unless you call logs being rotated off the host at end of the hour) and definitely no indexing. Never had any problems managing outages. Grep alone can go a long way in managing outages.
- jeffbee 5y agoAll the resources are at the edges of your infrastructure, so distributed grep makes more and more sense the larger your installation becomes. Centralized logging looks less and less economic as things expand.
- romantomjak 5y agoThanks, this is a great insight. I haven’t thought about it in this way and it sounds like a very pragmatic approach without over complicating things.
- closeparen 5y agoDid you ever worry about those greps interfering with production performance?
- jeffbee 5y agoDo you ever worry that your logs exporting agent interferes with service performance? After all, that goes on continuously in a setup like the one in the article, rather than as-needed in a distributed predicate evaluation setup.
- closeparen 5y agoA constant stable performance drag is going to show up in your load tests, capacity planning, etc. rather than in surprising intermittent degradations. Also our log collector agents run in containers like everything else, so there is some amount of resource isolation (not perfect of course).
- Aeolun 5y agoIn enterprises of this size, it’s quite possible you get a request to debug something that happened a month ago. Your instance may be gone, so you need to debug going only by whatever was logged.
- dilyevsky 5y agoCan’t do that in cloud (i know twitter doesnt run in cloud) and autoscale - machine goes poof and so do the logs. One of the reasons why autoscaling to save $ is bs.
- jeffbee 5y agoThat’s simplistic. For one thing, at scale not every debug statement is a precious snowflake. Secondly, in an orchestration scheme like k8s there’s no reason why your container lifecycle can’t include a cleanup container that runs and either exports the logs or just sleeps for an hour, so ephemeral data lives a little longer.
- closeparen 5y agoExports your logs to where? I would call that place centralized logging.
- dilyevsky 5y agoI’m assuming neighboring nodes. Now you’re basically designing replicated (best effort), distributed index - a mini version of google search. This is a clever design but number of companies that can diy it with sufficient quality I can probably count on one hand
- jeffbee 5y agoFor the price of even a modest splunk enterprise setup you can hire half a dozen top notch backend/distributed systems engineers. That’s the opportunity cost of a setup like the one in the article.
- closeparen 5y agoI’ve heard this before and it sounds like Splunk’s margins are just way too high. Why doesn’t someone sell a logging platform that’s cheaper than writing your own at scale?