5 ms·
I fundamentally agree with your fundamental disagreement with the article, but then I also fundamentally disagree with: > ship log events out of the box and in
by jeffbee 6y ago
I fundamentally agree with your fundamental disagreement with the article, but then I also fundamentally disagree with:
> ship log events out of the box and into something searchable, indexable, and can derive metrics
That's just a way to spend a ton of money. There's really not a reason to ship or index logs. Just leave them where they are produced and instead of shipping them to a central thing that inevitably becomes an expensive bottleneck, instead develop ways to access them in place (other than SSH) and frameworks for large-scale distributed searching where you push the predicate out to the machines where the logs exist.
- jakupovic 6y agoInception of fundamental disagreements ... On the point, what do you do if the machine is not there anymore? Your model fails immediately. Also, some processes operate on an assumption that the "machine" will stand-up -> do-something()- > die(), which your model also doesn't support. Not to be messing with you, but I don't think there is the "right" model, but whatever works for us. I do like your model for immediate accessibility of the logs.
- viraptor 6y agoThis way you lose them every time when something goes wrong with the infra, or when your fleet autoscales. Or if you do blue-green deployment with infra, then on every deployment. Your storage/indexing money spend scales with your company. If you have < 10 hosts you can probably handle logs with a cost that's a rounding error with something like loki+grafana. If you have a bigger environment and more people, the cost will be more significant, but you'll also likely have more revenue to justify it.
- hnlmorg 6y agoThat's fine if you don't have many servers and treat them as pets[1] but once you start scaling sideways, auto-scaling, using docker / Kubernetes or any of the other tooling around build pipelines or clustering and you quickly run into issues if those logs aren't streamed to a central repository. This is the age old problem of simple problems being harder to solve at scale. Running a blog on Digital Ocean is very different to running a million dollar application on AWS. [1] https://medium.com/@Joachim8675309/devops-concepts-pets-vs-cattle-2380b5aab313 https://medium.com/@Joachim8675309/devops-concepts-pets-vs-c...
- jeffbee 6y agoPeople always -- always -- pull cattle-vs-pets on me after I advocate for this position, but I've slaughtered more machines than any of you. The problem is that you are treating your logs as pets when the truth is the logs are also cattle. Virtually all debug logs will pass through their life cycle without being read, so indexing them is just a flagrant waste of energy.
- hnlmorg 6y ago> I've slaughtered more machines than any of you. I'd rather not engage in pointless unprovable arguments, if you don't mind. > The problem is that you are treating your logs as pets when the truth is the logs are also cattle. Virtually all debug logs will pass through their life cycle without being read, so indexing them is just a flagrant waste of energy. You shouldn't have debugging enabled on production systems unless you have an interim process that filters out debug messages before they get indexed (and thus you can toggle which logs get indexed there rather than reconfiguring / redeploying all of your application nodes). Also nobody is suggesting logs should be treated as "pets". You still want to purge out older logs however the problem is you cannot always replicate reported errors so if you don't have those log messages captured then you're sod out of luck. Don't get me wrong, there is a certain allure to the traditional method of systems administration - I've been on both sides of the fence - but central logging services have so many other benefits such as security (tamper proof logs, users don't require SSH), ease of use, persistent logging, etc. The only real downside is cost but that quickly becomes absorbed in your pricing plan when customers start asking for SLAs.
- R0b0t1 6y agoI have to agree with the other guy. Going too far into "Don't touch the machine" leads you to ridiculous places. Designing your system to be instrumented is invaluable, and sometimes that means logging on to a machine. Should you minimize this? Yes, but not to the point you invest tens of thousands or more into a way to not use SSH.
- 6y ago
- Narkov 6y ago> There's really not a reason to ship or index logs. Proper Information Security policy is one very good reason. If a box is compromised, you don't want the logs being tampered with. You ship the logs to a central, dedicated server so that a clear audit trail can be established.
- jdxcode 6y agoI imagine basically any form of compliance would require centralizing logs in some form
- f0rfun 6y agoand for audit. Audit/Compliance is one of the worst yet valid reasons to introduce complexity, redundancy and all sorts of costs into projects. It's also why some people earn a living out of it purely.
- tyldum 6y agoAuthentication, authorization and access logs typically. Generic application logs can often be treated differently.
- nurettin 6y agoShipping software logs is just useless bureaucracy. Don't confuse it with shipping firewall/ssh/rdp logs which is an absolute must for telling how/when you were pwned. Hacker will just disable app/db logs, update their user as admin, dump the db and proceed with the cryptofuckery.
- throwawayinfo 6y agoEh? In my experience logs are some of the most valuable data produced by software. Debugging, experiments, software quality, auth trails, analytics, customer support, privacy, security, audit/compliance and abuse are all seriously useful applications for logs. Throwing them away is like dumping gold.
- throwawayinfo 6y agoI've seen centralised logging scale to FANG level, so I really wouldn't worry about bottlenecking.
- spicyramen 6y agoIn production systems when something goes wrong is very common to ask customers/support team for logs. In most cases customer need SSh access or some role to collect this info. By the time they get to those, they are gone. In my experience to really scale a service you need OOB logging tools and automation to perform actions around errors in logs
- gumby 6y agoYou could just have an sshable account without shell privileges, only able to run grep or some small set of controlled utilities and with filesystem access restricted to read only access to the log sirs. Anyone who needs access could remotely run a command that would return the results of whatever permitted query they submitted.