4 ms·
That's fine if you don't have many servers and treat them as pets[1] but once you start scaling sideways, auto-scaling, using docker / Kubernetes or any of the
by hnlmorg 6y ago
That's fine if you don't have many servers and treat them as pets[1] but once you start scaling sideways, auto-scaling, using docker / Kubernetes or any of the other tooling around build pipelines or clustering and you quickly run into issues if those logs aren't streamed to a central repository.
This is the age old problem of simple problems being harder to solve at scale. Running a blog on Digital Ocean is very different to running a million dollar application on AWS.
[1] https://medium.com/@Joachim8675309/devops-concepts-pets-vs-cattle-2380b5aab313 https://medium.com/@Joachim8675309/devops-concepts-pets-vs-c...
- jeffbee 6y agoPeople always -- always -- pull cattle-vs-pets on me after I advocate for this position, but I've slaughtered more machines than any of you. The problem is that you are treating your logs as pets when the truth is the logs are also cattle. Virtually all debug logs will pass through their life cycle without being read, so indexing them is just a flagrant waste of energy.
- hnlmorg 6y ago> I've slaughtered more machines than any of you. I'd rather not engage in pointless unprovable arguments, if you don't mind. > The problem is that you are treating your logs as pets when the truth is the logs are also cattle. Virtually all debug logs will pass through their life cycle without being read, so indexing them is just a flagrant waste of energy. You shouldn't have debugging enabled on production systems unless you have an interim process that filters out debug messages before they get indexed (and thus you can toggle which logs get indexed there rather than reconfiguring / redeploying all of your application nodes). Also nobody is suggesting logs should be treated as "pets". You still want to purge out older logs however the problem is you cannot always replicate reported errors so if you don't have those log messages captured then you're sod out of luck. Don't get me wrong, there is a certain allure to the traditional method of systems administration - I've been on both sides of the fence - but central logging services have so many other benefits such as security (tamper proof logs, users don't require SSH), ease of use, persistent logging, etc. The only real downside is cost but that quickly becomes absorbed in your pricing plan when customers start asking for SLAs.
- R0b0t1 6y agoI have to agree with the other guy. Going too far into "Don't touch the machine" leads you to ridiculous places. Designing your system to be instrumented is invaluable, and sometimes that means logging on to a machine. Should you minimize this? Yes, but not to the point you invest tens of thousands or more into a way to not use SSH.
- Aeolun 6y agoSince we already know you can put a more or less adequate elasticsearch cluster on DO for $60 per month, I don’t think that argument holds a lot of water. By the time you want centralized logs, you are probably already spending much more than the logging will cost on infrastructure.
- majkinetor 6y agoES cluster might be easy to install but doing all other chores with your log, along with learning ES search language and quirks is far from trivial. I personally spent months making viable solution on premise and its not something I want to do again. Way less hassle to access server directly for daily logs and just move them during the night on some archival storage.
- hnlmorg 6y agoNobody said "you should never log into the machine!" We just said reading log files shouldn't require SSH access. What you're doing here is constructing a straw man argument while agreeing with the same point I was making.
- jdxcode 6y ago> I've slaughtered more machines than any of you. Not only is that vanishingly unlikely to be true with this crowd it’s also not at all the kind of rhetoric that they’ll listen to either.
- markbnj 6y agoWhen I read that comment I was thinking about our auto-scaling nodepools on GKE that have been running for a couple of years and scale up/down every business day by 20-40 nodes or so depending on load... but I am not sure I get to take credit for slaughtering them.
- gabereiser 6y agothe logs in my case are audit trails required for compliance and for investigation by 3rd parties.
- tristor 6y ago> I've slaughtered more machines than any of you. I guarantee that you haven't, as the number of people that can make that claim in relation to my background is vanishingly small and I know almost all of them by name. I centralize logs. Not only does it make more sense for administration at scale, it's invaluable for security reasons and assists in compliance by providing a controllable guaranteed audit trail. You may have a valid argument about cost here for some applications, but it's unwise to make arrogant claims you cannot back up.
- seized 6y agoSo that's the new replacement for pointless uptime boasting.... Except even less provable.