6 ms·

What are they meant to do if the world starts burning at 4am? Wait until a reasonable hour? Oncall fires at 4am suck but they're part of running critical servic

What are they meant to do if the world starts burning at 4am? Wait until a reasonable hour? Oncall fires at 4am suck but they're part of running critical services.

lallysingh 7y ago
You have a runbook. Anything not covered by the runbook escalates to someone who already understands the code.
- LargeWu 7y ago
  Are you saying you've never written code and then later came back to it, saying "Who wrote this shit?" and did a git blame, only to find out it was you?
  - lallysingh 7y ago
    If you can't read your own code, then the language isn't your problem.
    - h91wka 7y ago
      Seems like someone never solved a complex problem.
      - lallysingh 7y ago
        Full on ad-hominem, eh?
        detaro 7y ago
        and in comparison your comment wasn't?
        lallysingh 7y ago
        Who was I attacking? Writing unreadable code isn't a language problem. Even in doc-poor environments, the names and sparse comments should be enough. If not, the language can't help you choose good names.
        dmitriid 7y ago
        Good luck remembering complex business logic in your perfectly readable code and all external dependencies (DBs, other microservices, etc.) a few months after you last touched it, and in the middle of the night. And after a few refactorings. Most errors have nothing to do with “unreadable code”.
- h91wka 7y ago
  Obviously, I was talking about the situation when it escalated to that person. And if you are going to claim that you know exactly how your code works after a month, then I call bs on that. Not to mention that 1) people leave companies sometimes 2) 3rd party code can break too.
  - lallysingh 7y ago
    Not obvious. Look, you've already fucked yourself by needing a 4am custom fix. That isn't a language problem.
    - magicalhippo 7y ago
      It's not a language problem that you need a 4am custom fix. It can be a language problem if the person who needs to do a 4am custom fix can't understand what the code is supposed to do. Or to put it another way: I'd much rather be debugging some BASIC code at 4am than some Brainfuck code.
      - lallysingh 7y ago
        I think the same person who writes unreadable code in Haskell would write unreadable code in most any other adult language.
      - bpyne 7y ago
        Only a person from the team who wrote the code to begin with should create an emergency fix. Since any developer from that team already knows Haskell, from having worked on the code base, the criticism is ridiculous.
        magicalhippo 7y ago
        We're all on the same team at work (there's only the six of us), but that doesn't mean I know all the details of my coworkers work. It's also not uncommon that the dev available at 4am is not the one who wrote the code himself. As I pointed out in my previous post, I do feel the language can be a non-trivial factor in allowing a dev like myself to be confident in developing a fix for my coworkers code at 4am. However I don't know Haskell, so I have no idea how it fares in this regard.
- deleted 7y ago
  [deleted]
- ukj 7y ago
  In the DevOps world, the person who understands the code is the person who got paged for it being broken.
  - kbr2000 7y ago
    How does that work in practice? In order to page, you should find out if it is the code, and it is indeed broken. I'm guessing all tests passed, so that won't be of much help anymore neither... In reality, it would be a sysadmin confirming dependencies for this application to run (in the system, network, storage, ...) are functioning as required. If there turns out to be a problem with the application itself, there's no time for development at that point: you roll back. I don't see how the paging developers thing would be able to provide any stability. It's too late for that when you're in production.
    - ukj 7y ago
      In my reality I am the "sysadmin". I am also the "developer". I am the guy who wrote tests (unit and integration). I also configured the CI/CD pipeline. I am also the guy who puts metrics in place to monitor the health of the system and if any metrics breach alarming thresholds the owner of the service (me!) is automatically paged. How does a sysadmin know that something is broken anyway, and how does a sysadmin know who needs to get paged? If a sysadmin can make this decision - so can an algorithm. Much of this is in Google's SRE handbook[1]. The entire notion behind the DevOps concept was to make sure that you don't segregate development and operations. The people who write crappy code must be the people who wake up at 3am when the crappy code breaks. >It's too late for that when you're in production. Bugs will always slip through testing, and things will break even in production. Nobody is perfect - not even Google. So what do you do when rollback doesn't work, you've tried everything in the playbook but the system/service is still down? Whose job is it to understand how to recover your service? Surely you don't expect the sysadmins to be doing that? They don't understand the system. How could they? They didn't build it. [1] https://landing.google.com/sre/books/ https://landing.google.com/sre/books/

What are they meant to do if the world starts burning at 4am? Wait until a reasonable hour? Oncall fires at 4am suck but they're part of running critical services.

lallysingh 7y ago

You have a runbook. Anything not covered by the runbook escalates to someone who already understands the code.

LargeWu 7y ago
Are you saying you've never written code and then later came back to it, saying "Who wrote this shit?" and did a git blame, only to find out it was you?
- lallysingh 7y ago
  If you can't read your own code, then the language isn't your problem.
  - h91wka 7y ago
    Seems like someone never solved a complex problem.
    - lallysingh 7y ago
      Full on ad-hominem, eh?
      - detaro 7y ago
        and in comparison your comment wasn't?
        lallysingh 7y ago
        Who was I attacking? Writing unreadable code isn't a language problem. Even in doc-poor environments, the names and sparse comments should be enough. If not, the language can't help you choose good names.
        dmitriid 7y ago
        Good luck remembering complex business logic in your perfectly readable code and all external dependencies (DBs, other microservices, etc.) a few months after you last touched it, and in the middle of the night. And after a few refactorings. Most errors have nothing to do with “unreadable code”.
h91wka 7y ago
Obviously, I was talking about the situation when it escalated to that person. And if you are going to claim that you know exactly how your code works after a month, then I call bs on that. Not to mention that 1) people leave companies sometimes 2) 3rd party code can break too.
- lallysingh 7y ago
  Not obvious. Look, you've already fucked yourself by needing a 4am custom fix. That isn't a language problem.
  - magicalhippo 7y ago
    It's not a language problem that you need a 4am custom fix. It can be a language problem if the person who needs to do a 4am custom fix can't understand what the code is supposed to do. Or to put it another way: I'd much rather be debugging some BASIC code at 4am than some Brainfuck code.
    - lallysingh 7y ago
      I think the same person who writes unreadable code in Haskell would write unreadable code in most any other adult language.
    - bpyne 7y ago
      Only a person from the team who wrote the code to begin with should create an emergency fix. Since any developer from that team already knows Haskell, from having worked on the code base, the criticism is ridiculous.
      - magicalhippo 7y ago
        We're all on the same team at work (there's only the six of us), but that doesn't mean I know all the details of my coworkers work. It's also not uncommon that the dev available at 4am is not the one who wrote the code himself. As I pointed out in my previous post, I do feel the language can be a non-trivial factor in allowing a dev like myself to be confident in developing a fix for my coworkers code at 4am. However I don't know Haskell, so I have no idea how it fares in this regard.
deleted 7y ago
[deleted]
ukj 7y ago
In the DevOps world, the person who understands the code is the person who got paged for it being broken.
- kbr2000 7y ago
  How does that work in practice? In order to page, you should find out if it is the code, and it is indeed broken. I'm guessing all tests passed, so that won't be of much help anymore neither... In reality, it would be a sysadmin confirming dependencies for this application to run (in the system, network, storage, ...) are functioning as required. If there turns out to be a problem with the application itself, there's no time for development at that point: you roll back. I don't see how the paging developers thing would be able to provide any stability. It's too late for that when you're in production.
  - ukj 7y ago
    In my reality I am the "sysadmin". I am also the "developer". I am the guy who wrote tests (unit and integration). I also configured the CI/CD pipeline. I am also the guy who puts metrics in place to monitor the health of the system and if any metrics breach alarming thresholds the owner of the service (me!) is automatically paged. How does a sysadmin know that something is broken anyway, and how does a sysadmin know who needs to get paged? If a sysadmin can make this decision - so can an algorithm. Much of this is in Google's SRE handbook[1]. The entire notion behind the DevOps concept was to make sure that you don't segregate development and operations. The people who write crappy code must be the people who wake up at 3am when the crappy code breaks. >It's too late for that when you're in production. Bugs will always slip through testing, and things will break even in production. Nobody is perfect - not even Google. So what do you do when rollback doesn't work, you've tried everything in the playbook but the system/service is still down? Whose job is it to understand how to recover your service? Surely you don't expect the sysadmins to be doing that? They don't understand the system. How could they? They didn't build it. [1] https://landing.google.com/sre/books/ https://landing.google.com/sre/books/