15 ms·

>Every developer needs access to some servers for example to check the application logs. I fundamentally disagree with this. I’ve been writing software a long

>Every developer needs access to some servers for example to check the application logs. I fundamentally disagree with this. I’ve been writing software a long time and I used to demand I have server access so I can tail logs, creating the problem this article talks about resolving (good read btw). But I can’t help but wonder why we keep teaching this mindset. We have log analysis tools available on pretty much every cloud now. Docker has an aws log driver, a gcp log driver... etc. Backend developers should make a conscious decision on how to ship log events out of the box and into something searchable, indexable, and can derive metrics from. I ran a cloud infrastructure group that vehemently said “No” to any dev requesting ssh access. Not because they couldn’t, but because we don’t have access either. We created our platform without the ability to modify it. Infrastructure as code. Only way is to redeploy the stack. You’d be surprised at how much less stress there is when you can have alerts on log events from your application when things break instead of a support call or a support ticket. Proactive > Reactive debugging. I also understand not all shops are at that level of maturity. I’d love it if the community as a whole stopped teaching people to treat their app and server as a second home, as a pet, that must be nurtured. Obviously these are my opinions and the article itself addresses how to handle ssh keys for server access in a logical way, my only issue is why create that mess in the first place?

mpaepper 6y ago
Thank you, I agree that this should be avoided as much as possible in general. Actually, in my company, we are pushing the logs to an external service, so I guess that line I wrote there was not totally thought out. ;) However, it still might be useful for certain people to check servers for health or other metrics and you might not always have everything as perfect as you wish. I guess you already stated that yourself. So in general I'd agree. Good comment!
- gabereiser 6y ago
  Health is a function of the application, not it’s host. A health check endpoint is paramount to ensuring the app is healthy. I still disagree with having _any_ access to a “box”. Local dev, console log, deployed debug? Better make sure you are logging events and not non-sense. Actionable events with request tracing (preferably). But yeah, it was a good article. Bitwarden is something I’ve used to share privileged keys before but the whole signing stuff was the right way to go. Also, if you aren’t on “cloud”, odds are you are still using something like Kubernetes or DC/OS or Swarm or the like. If you aren’t then well, wordpress sites aren’t really in the same ballpark technically. (Joke, Wordpress sites get traffic, some lots of traffic, I don’t discriminate against the PHP tribe).
  - holoduke 6y ago
    I see your point. But you are incorrect and too much reasoning probably from your current position. In many new companies practicality is very important. There are maybe no resources to setup a giant set of utilities which replaces info you would normally get by SSH to a server. In a new company you wanna make this a graduate process. The security aspect is always important. But it's a balance. You cannot over engineer.
all_usernames 6y ago
Agreed, and what you're saying is really the proper way do things in a cloud-native environment where all hosts are ephemeral, containerized apps are idempotent, and especially a serverless environment where no one has access to the invisible host. However, for anything shy of full serverless, just s/developer/ops and you still have to solve for the same problem. At the end of the day you're always going to want someone to be able to get a shell on the box for troubleshooting some critical issue. If there is a box.
ThePhysicist 6y ago
We also manage our whole infrastructure automatically via Ansible, but if there's a problem that goes beyond an error in the deployed code it's still necessary to access a machine via SSH in order to debug it. Logs are arguably something that can be easily centralized as most application support logging via syslog and there are a lot of great tools available for it like greylog.
- gabereiser 6y ago
  Sure, having a privileged key you use with ansible to actually do the thing... but we’re talking about giving the shop access to the box. Not your team of highly trained infrastructure engineers (or bob, who has all his playbooks on a thumb drive for his office). If ansible and non-cloud is your deal, an ssh key (or two: deployments) is fine. I’m just against giving others access to things they could break or worse, leave the door open...
- pnutjam 6y ago
  IMHO, using ansible to manage keys is a way simpler process. You won't miss things because your inventory file has everything.
jeffbee 6y ago
I fundamentally agree with your fundamental disagreement with the article, but then I also fundamentally disagree with: > ship log events out of the box and into something searchable, indexable, and can derive metrics That's just a way to spend a ton of money. There's really not a reason to ship or index logs. Just leave them where they are produced and instead of shipping them to a central thing that inevitably becomes an expensive bottleneck, instead develop ways to access them in place (other than SSH) and frameworks for large-scale distributed searching where you push the predicate out to the machines where the logs exist.
- jakupovic 6y ago
  Inception of fundamental disagreements ... On the point, what do you do if the machine is not there anymore? Your model fails immediately. Also, some processes operate on an assumption that the "machine" will stand-up -> do-something()- > die(), which your model also doesn't support. Not to be messing with you, but I don't think there is the "right" model, but whatever works for us. I do like your model for immediate accessibility of the logs.
- viraptor 6y ago
  This way you lose them every time when something goes wrong with the infra, or when your fleet autoscales. Or if you do blue-green deployment with infra, then on every deployment. Your storage/indexing money spend scales with your company. If you have < 10 hosts you can probably handle logs with a cost that's a rounding error with something like loki+grafana. If you have a bigger environment and more people, the cost will be more significant, but you'll also likely have more revenue to justify it.
- hnlmorg 6y ago
  That's fine if you don't have many servers and treat them as pets[1] but once you start scaling sideways, auto-scaling, using docker / Kubernetes or any of the other tooling around build pipelines or clustering and you quickly run into issues if those logs aren't streamed to a central repository. This is the age old problem of simple problems being harder to solve at scale. Running a blog on Digital Ocean is very different to running a million dollar application on AWS. [1] https://medium.com/@Joachim8675309/devops-concepts-pets-vs-cattle-2380b5aab313 https://medium.com/@Joachim8675309/devops-concepts-pets-vs-c...
kyran_adept 6y ago
What if you encounter an nginx bug, or a kernel bug? At some point, when you reach a low enough level you will need some deeper investigation tools and finally access to the machine to get enough data and fix the problem.
- paranoidrobot 6y ago
  That should be the exception, rather than the rule.
  - anamexis 6y ago
    And it is deeply frustrating when you run up against one of these exceptions and need to wade through some bureaucracy before you can investigate further.
    - gabereiser 6y ago
      the point people are trying to make is that if you are at the scale where a kernel bug or an nginx bug is borking your app, it's not the developers job to go poking around the system for a fix. It's the devops/infra people's job. In my world, if you want to investigate an nginx bug... "docker run -it nginx:latest /bin/bash" and go for it... find the issue, reproduce it, then fix it in the pipeline and deploy again. You didn't touch production at all. If your debugging relies on being ON PRODUCTION, you don't suffer from the scale you need to be on there in the first place.
      - jclulow 6y ago
        Not all bugs are sufficiently cheaply reproducible outside the environment in which they are observed. It seems silly to tie your hands behind your back when you could just inspect what the computer is doing and then fix it.
      - shawnz 6y ago
        Why can't your developers be "devops people"?
recursive 6y ago
My application runs on my customers' server(s). I don't generally have any access to anything.
kevindong 6y ago
The main counterpoint is what do you do when the logging system goes down at the same time there's an incident? Do you then just rely purely on metrics/customer reports to try to debug?
- zachrose 6y ago
  When you’re tailing logs on a server there’s still a “logging system“ (your ssh connection, tail -f, a disk with finite space) that can still “fail”. Probably in either case you’d work to fix logging and you’d hope that the system is simple enough to fix.
  - Spivak 6y ago
    Logging systems are annoyingly complicated and there are a lot of moving pieces that can fail. Way more than reading a file over SSH. This kind of access shouldn’t be your go-to but it’s important to have a battleshort.
    - chousuke 6y ago
      What's complicated about a syslog server? I suppose some applications like to spew out multiline logs (which are something you just can't process outside the source while maintaining sanity) but seriously, when you actually consider these things while writing the application, you don't need all that much to have something workable.
      - Spivak 6y ago
        I was going to make a note that an rsyslog collector would be my only exception to this because of how simple it is but you still have to be careful because by default rsyslog is okay with losing logs in-flight. But I would be lying if the typical logging setup was this simple. It’s basically all remixes of ELK and while they’re fantastic when they work there are a lot many moving parts that can individually fail.
stjohnswarts 6y ago
I think the point is that there's often a lot more to server access than just looking at logs. The article offers a solution to that.
mrmonkeyman 6y ago
This has nothing to do with "maturity", this is introducing sickening levels of complexity that hide behind walls of configuration JSON just to get cheaper programmers.
secondcoming 6y ago
> Backend developers should make a conscious decision on how to ship log events out of the box Great! hand waving. What are you going to do is this log streaming process stops working, and cannot restart? Kill the machine and lose the logs? That might be ok for your webapp, it could be trouble for more serious applications. > You’d be surprised at how much less stress there is when you can have alerts on log events from your application when things break instead of a support call or a support ticket. Giving SSH access doesn't mean you don't have event logs. > stopped teaching people to treat their app and server as a second home, as a pet, that must be nurtured In our case, nobody is treating anything as a pet. Our servers are mostly spot/preemptibles, but we still have SSH access. Because it makes it really easy to debug if things do go badly wrong. From being able to see htop output, to being able to upload custom binaries to investigate an issue, or to smoke test a code change... it's a huge time saver. Give me SSH or give me death.

mpaepper 6y ago

Thank you, I agree that this should be avoided as much as possible in general. Actually, in my company, we are pushing the logs to an external service, so I guess that line I wrote there was not totally thought out. ;) However, it still might be useful for certain people to check servers for health or other metrics and you might not always have everything as perfect as you wish. I guess you already stated that yourself. So in general I'd agree. Good comment!

gabereiser 6y ago
Health is a function of the application, not it’s host. A health check endpoint is paramount to ensuring the app is healthy. I still disagree with having _any_ access to a “box”. Local dev, console log, deployed debug? Better make sure you are logging events and not non-sense. Actionable events with request tracing (preferably). But yeah, it was a good article. Bitwarden is something I’ve used to share privileged keys before but the whole signing stuff was the right way to go. Also, if you aren’t on “cloud”, odds are you are still using something like Kubernetes or DC/OS or Swarm or the like. If you aren’t then well, wordpress sites aren’t really in the same ballpark technically. (Joke, Wordpress sites get traffic, some lots of traffic, I don’t discriminate against the PHP tribe).
- holoduke 6y ago
  I see your point. But you are incorrect and too much reasoning probably from your current position. In many new companies practicality is very important. There are maybe no resources to setup a giant set of utilities which replaces info you would normally get by SSH to a server. In a new company you wanna make this a graduate process. The security aspect is always important. But it's a balance. You cannot over engineer.

all_usernames 6y ago

Agreed, and what you're saying is really the proper way do things in a cloud-native environment where all hosts are ephemeral, containerized apps are idempotent, and especially a serverless environment where no one has access to the invisible host. However, for anything shy of full serverless, just s/developer/ops and you still have to solve for the same problem. At the end of the day you're always going to want someone to be able to get a shell on the box for troubleshooting some critical issue. If there is a box.

ThePhysicist 6y ago

We also manage our whole infrastructure automatically via Ansible, but if there's a problem that goes beyond an error in the deployed code it's still necessary to access a machine via SSH in order to debug it. Logs are arguably something that can be easily centralized as most application support logging via syslog and there are a lot of great tools available for it like greylog.

gabereiser 6y ago
Sure, having a privileged key you use with ansible to actually do the thing... but we’re talking about giving the shop access to the box. Not your team of highly trained infrastructure engineers (or bob, who has all his playbooks on a thumb drive for his office). If ansible and non-cloud is your deal, an ssh key (or two: deployments) is fine. I’m just against giving others access to things they could break or worse, leave the door open...
pnutjam 6y ago
IMHO, using ansible to manage keys is a way simpler process. You won't miss things because your inventory file has everything.

jeffbee 6y ago

I fundamentally agree with your fundamental disagreement with the article, but then I also fundamentally disagree with: > ship log events out of the box and into something searchable, indexable, and can derive metrics That's just a way to spend a ton of money. There's really not a reason to ship or index logs. Just leave them where they are produced and instead of shipping them to a central thing that inevitably becomes an expensive bottleneck, instead develop ways to access them in place (other than SSH) and frameworks for large-scale distributed searching where you push the predicate out to the machines where the logs exist.

jakupovic 6y ago
Inception of fundamental disagreements ... On the point, what do you do if the machine is not there anymore? Your model fails immediately. Also, some processes operate on an assumption that the "machine" will stand-up -> do-something()- > die(), which your model also doesn't support. Not to be messing with you, but I don't think there is the "right" model, but whatever works for us. I do like your model for immediate accessibility of the logs.
viraptor 6y ago
This way you lose them every time when something goes wrong with the infra, or when your fleet autoscales. Or if you do blue-green deployment with infra, then on every deployment. Your storage/indexing money spend scales with your company. If you have < 10 hosts you can probably handle logs with a cost that's a rounding error with something like loki+grafana. If you have a bigger environment and more people, the cost will be more significant, but you'll also likely have more revenue to justify it.
hnlmorg 6y ago
That's fine if you don't have many servers and treat them as pets[1] but once you start scaling sideways, auto-scaling, using docker / Kubernetes or any of the other tooling around build pipelines or clustering and you quickly run into issues if those logs aren't streamed to a central repository. This is the age old problem of simple problems being harder to solve at scale. Running a blog on Digital Ocean is very different to running a million dollar application on AWS. [1] https://medium.com/@Joachim8675309/devops-concepts-pets-vs-cattle-2380b5aab313 https://medium.com/@Joachim8675309/devops-concepts-pets-vs-c...

kyran_adept 6y ago

What if you encounter an nginx bug, or a kernel bug? At some point, when you reach a low enough level you will need some deeper investigation tools and finally access to the machine to get enough data and fix the problem.

paranoidrobot 6y ago
That should be the exception, rather than the rule.
- anamexis 6y ago
  And it is deeply frustrating when you run up against one of these exceptions and need to wade through some bureaucracy before you can investigate further.
  - gabereiser 6y ago
    the point people are trying to make is that if you are at the scale where a kernel bug or an nginx bug is borking your app, it's not the developers job to go poking around the system for a fix. It's the devops/infra people's job. In my world, if you want to investigate an nginx bug... "docker run -it nginx:latest /bin/bash" and go for it... find the issue, reproduce it, then fix it in the pipeline and deploy again. You didn't touch production at all. If your debugging relies on being ON PRODUCTION, you don't suffer from the scale you need to be on there in the first place.
    - jclulow 6y ago
      Not all bugs are sufficiently cheaply reproducible outside the environment in which they are observed. It seems silly to tie your hands behind your back when you could just inspect what the computer is doing and then fix it.
    - shawnz 6y ago
      Why can't your developers be "devops people"?

recursive 6y ago

My application runs on my customers' server(s). I don't generally have any access to anything.

kevindong 6y ago

The main counterpoint is what do you do when the logging system goes down at the same time there's an incident? Do you then just rely purely on metrics/customer reports to try to debug?

zachrose 6y ago
When you’re tailing logs on a server there’s still a “logging system“ (your ssh connection, tail -f, a disk with finite space) that can still “fail”. Probably in either case you’d work to fix logging and you’d hope that the system is simple enough to fix.
- Spivak 6y ago
  Logging systems are annoyingly complicated and there are a lot of moving pieces that can fail. Way more than reading a file over SSH. This kind of access shouldn’t be your go-to but it’s important to have a battleshort.
  - chousuke 6y ago
    What's complicated about a syslog server? I suppose some applications like to spew out multiline logs (which are something you just can't process outside the source while maintaining sanity) but seriously, when you actually consider these things while writing the application, you don't need all that much to have something workable.
    - Spivak 6y ago
      I was going to make a note that an rsyslog collector would be my only exception to this because of how simple it is but you still have to be careful because by default rsyslog is okay with losing logs in-flight. But I would be lying if the typical logging setup was this simple. It’s basically all remixes of ELK and while they’re fantastic when they work there are a lot many moving parts that can individually fail.

stjohnswarts 6y ago

I think the point is that there's often a lot more to server access than just looking at logs. The article offers a solution to that.

mrmonkeyman 6y ago

This has nothing to do with "maturity", this is introducing sickening levels of complexity that hide behind walls of configuration JSON just to get cheaper programmers.

secondcoming 6y ago

> Backend developers should make a conscious decision on how to ship log events out of the box Great! hand waving. What are you going to do is this log streaming process stops working, and cannot restart? Kill the machine and lose the logs? That might be ok for your webapp, it could be trouble for more serious applications. > You’d be surprised at how much less stress there is when you can have alerts on log events from your application when things break instead of a support call or a support ticket. Giving SSH access doesn't mean you don't have event logs. > stopped teaching people to treat their app and server as a second home, as a pet, that must be nurtured In our case, nobody is treating anything as a pet. Our servers are mostly spot/preemptibles, but we still have SSH access. Because it makes it really easy to debug if things do go badly wrong. From being able to see htop output, to being able to upload custom binaries to investigate an issue, or to smoke test a code change... it's a huge time saver. Give me SSH or give me death.