7 ms·

Some notes on running new software in production

84 points by eurg 8y ago

dswalter 8y ago
The wisdom that comes from running things in production makes open-ended questions about those gotchas useful interview questions. 'looks like you used kubernetes for a major project recently. What were some of the unexpected warts?' Someone with experience running a service on k8s will have at least a few stories, and how someone describes their own responses to those hurdles can shed light on their personality.
- quickthrower2 8y ago
  Don't make me relive those stories!
- taeric 8y ago
  Agreed, though it saddens me how many of those hurdles will be poor rediscovery of previously well documented problems in older systems. :(
- NikolaeVarius 8y ago
  I tend to like these sorts of questions, really shows you how much a person understands about a system. I also like it when its asked of me, though I always wonder how much a rant about the things I've experienced setting up Nomad/Consul/Vault would be productive.
zamazingo 8y ago
Try working at a place where all fixes are tested in production...
- hnmonkey 8y ago
  Can you describe what you guys do and your infrastructure? Is there a need to test in production or has it just not been important enough to prevent it?
  - user5994461 8y ago
    Everybody has a test environment, it's rare to also have a production environment.
    - hnmonkey 8y ago
      Your statement doesn't match up at all with what they said nor even make sense. What's the point of a business if you only have a test environment but rarely have a production environment? You seem to be assuming to know what their development process and infrastructure is like.
      - nevon 8y ago
        The joke is that if you don't have a testing environment, production is your testing environment, which in effect means that you don't have a production environment.
  - zamazingo 8y ago
    Sorry, cannot describe infrastructure (not a throwaway account). Prod and Test are out of sync in terms of mission critical software, Test is not powerful enough to handle Prod's tasks, and there is no formal procedure to use Test for testing. So Test is never actually used. When a bug is found in Prod, it's fixed on Prod, while it's live. By fix, of course I mean an iteration on and combinations of: qc: we found bug dev: it's not a bug qc: it is a bug: [insert evidence] dev: it's fixed now qc: no it's not dev: how about now? qc: no dev: and now? qc: yes but now this other thing is broken dev: here's another attempt, please check again qc: Prod is down dev: try it now qc: it's still broken, along with the other broken part, and now these three other parts are broken too ceo: it's fine, close this ticket
shay_ker 8y ago
> Envoy’s active healthchecking means that you services get healthchecked by every client. This is mostly okay but (again) services with many clients can get overwhelmed by it. Man that's brutal. Is that normal in service-mesh frameworks?

84 points by eurg 8y ago

The wisdom that comes from running things in production makes open-ended questions about those gotchas useful interview questions. 'looks like you used kubernetes for a major project recently. What were some of the unexpected warts?' Someone with experience running a service on k8s will have at least a few stories, and how someone describes their own responses to those hurdles can shed light on their personality.

quickthrower2 8y ago
Don't make me relive those stories!
taeric 8y ago
Agreed, though it saddens me how many of those hurdles will be poor rediscovery of previously well documented problems in older systems. :(
NikolaeVarius 8y ago
I tend to like these sorts of questions, really shows you how much a person understands about a system. I also like it when its asked of me, though I always wonder how much a rant about the things I've experienced setting up Nomad/Consul/Vault would be productive.

zamazingo 8y ago

Try working at a place where all fixes are tested in production...

hnmonkey 8y ago
Can you describe what you guys do and your infrastructure? Is there a need to test in production or has it just not been important enough to prevent it?
- user5994461 8y ago
  Everybody has a test environment, it's rare to also have a production environment.
  - hnmonkey 8y ago
    Your statement doesn't match up at all with what they said nor even make sense. What's the point of a business if you only have a test environment but rarely have a production environment? You seem to be assuming to know what their development process and infrastructure is like.
    - nevon 8y ago
      The joke is that if you don't have a testing environment, production is your testing environment, which in effect means that you don't have a production environment.
- zamazingo 8y ago
  Sorry, cannot describe infrastructure (not a throwaway account). Prod and Test are out of sync in terms of mission critical software, Test is not powerful enough to handle Prod's tasks, and there is no formal procedure to use Test for testing. So Test is never actually used. When a bug is found in Prod, it's fixed on Prod, while it's live. By fix, of course I mean an iteration on and combinations of: qc: we found bug dev: it's not a bug qc: it is a bug: [insert evidence] dev: it's fixed now qc: no it's not dev: how about now? qc: no dev: and now? qc: yes but now this other thing is broken dev: here's another attempt, please check again qc: Prod is down dev: try it now qc: it's still broken, along with the other broken part, and now these three other parts are broken too ceo: it's fine, close this ticket

shay_ker 8y ago

> Envoy’s active healthchecking means that you services get healthchecked by every client. This is mostly okay but (again) services with many clients can get overwhelmed by it. Man that's brutal. Is that normal in service-mesh frameworks?