13 ms·

What I Learned Managing Site Reliability for Some of the Busiest Gambling Sites

220 points by slyall 10y ago

rb808 10y ago
Great article. Documentation is always tough - I still haven't found the right solution. Its nice to see some SRE people that care, most 1st/2nd line support I've worked seem to just escalate anything non-trivial which is frustrating, but also useful. If SREs are too good developers lose touch with production and get lazy.
- zhengyi13 10y ago
  Reflecting on my time in frontline support, 1st level folk have limited skills, limited resources in terms of time and tooling, and in particular tend to have pretty tight metrics applied to them. They pretty much have to escalate quickly, or they'll be yelled at, or worse. Some of them flat out don't belong in Support. But poor management and poor metrics drive behavior in unhelpful directions, too.
- zwischenzug 10y ago
  Yeah, we were lucky in that we only had engineers with years of experience on call. They had to have had dev experience, too. This followed an attempt to run a more 'standard' support service (before my time) with specialist support staff. That failed badly, mostly because customers and devs hated that they added little value to the process. It was a long time before people were allowed to be recruited direct to support again, and by then the leaders were all former devs->tech leads.
jldugger 10y ago
> The team I joined was around 5 engineers (all former developers and technical leaders), which grew to around 50 of more mixed experience across multiple locations by the time I left. Unfortunately, it's difficult for the audience to determine whether the author's success is attributable to the philosophy in the post, or the 10x growth in staff.
- stephengillie 10y ago
  The author attributes both his success, and the team's growth, to his documentation philosophy. It does make some sense - improving process documentation for the most common issues will help ensure new team members resolve these issues consistently and effectively, both increasing their utility to the team and their personal morale. How much is directly attributable to him is open to debate.
  - zwischenzug 10y ago
    The team's growth was partly because of its perceived success at managing operations, but also because of large growth in customer base and application sprawl. The business model we had depended on fast releases and customers who refused to pay for much-needed testing or QA, but were happy to pay for the support we provided. Not saying that's a good thing, but it was the plate we were served.
- zwischenzug 10y ago
  The quantity of issues dealt with went up more than 10x over that time, and the experience of the people dealing with them dropped. Whether that was due to me, the things we did, or other reasons is of course open to debate. This is just my view from where I sat.
elorant 10y ago
I wish there were more articles like this for sites that are considered gray areas like gambling or porn.
- kbenson 10y ago
  For porn, Youporn has had some technical (or technical PR) submissions multiple times over the years. "Youporn.com is now a 100% Redis Site" (126 comments, 5 years ago)[1] "How YouPorn Uses Redis: SFW Edition" (95 comments, 4 years ago)[2] "YouPorn: Symfony2, Redis, Varnish, HA Proxy... (Keynote at ConFoo 2012)" (49 comments, 5 years ago) There's more in the HN search. 1: https://news.ycombinator.com/item?id=3597891 https://news.ycombinator.com/item?id=3597891 2: https://news.ycombinator.com/item?id=6137087 https://news.ycombinator.com/item?id=6137087 3: https://news.ycombinator.com/item?id=3750060 https://news.ycombinator.com/item?id=3750060
systems 10y ago
i am always surprised, that gambling sites and similar business with questionable morality dont face hard time recruiting top engineers , i always though this would be a major concern
- fasteo 10y ago
  Honest question. Do you think that they have "questionable morality" by the business nature (gambling, porn, etc) or for how they run the business (ex. dark patterns) ?
  - systems 10y ago
    i had business nature in mind
- jtreminio 10y ago
  Gambling itself isn't immoral.
  - solotronics 10y ago
    some people think it is morality is subjective
    - akvadrako 10y ago
      If morality is subjective is itself up for debate. I personally think morality is absolute. There is only one system of morality which isn't arbitrary and allowing morals to be arbitrary strips the word of any meaning.
      - brokenmachine 10y ago
        Do a google search for "moral dilemma" and you will find many examples where it is very difficult to come up with simplistic black and white opinions. Such dilemmas leave me with the idea that it is very naive to think there is some kind of absolute morality.
jedberg 10y ago
I'm going to have to respectfully disagree with a big chunk of this article. Documentation is generally a waste of time unless you have a very static infrastructure, and run books are the devil. You should never use a run book -- instead you should spend the time you were going to write a run book writing code to execute the steps automatically. This will reduce human error and make things faster and more repeatable. Even better is if the person who wrote the code also writes the automation to fix it so that it stays up to date with changes in the code. At Netflix we tried to avoid spending a lot of time on documentation because by the time the document was done, it was out of date. Almost inevitably any time you needed the documentation, it no longer applied. I wish the author had spent more time on talking about incident reviews. Those were our key to success. After every event, you review the event with everyone involved, including the developers, and then come up with an action plan that at a minimum prevents the same problem from happening again, but even better, prevents an entire class of problems from happening again. Then you have to follow through and make sure the changes are actually getting implemented. I agree with the author on the point about culture. That was absolutely critical. You need a culture that isn't about placing blame but finding solutions. One where people feel comfortable, and even eager, to come out and say "It was my fault, here's the problem, and here's how I'm going to fix it!"
- zwischenzug 10y ago
  'Documentation is generally a waste of time unless you have a very static infrastructure' I definitely agree with that, and it's partly a corollary of 'documentation is expensive and requires costly maintenance'. Run books/checklists are mostly implemented really really badly. Automation is the ideal, but is costly, and itself requires maintenance. Most of the steps we had to perform did not lend themselves to automation, also.
  - jedberg 10y ago
    > Automation is the ideal, but is costly, and itself requires maintenance. I would contend that the cost of automation is about the same as the cost of documentation plus the cost of having to manually do the work over an over. It's just a cost borne up front instead of over time. But to your point in the article, you have to have a culture that supports bearing that up front cost. > Most of the steps we had to perform did not lend themselves to automation, also. I don't understand how that is possible? Could you give an example of a task that can't be automated?
mfonda 10y ago
I'm a big fan of checklists as well; however, working through a checklist is still a manual process with room for human error. I've gone down the road of creating checklists, then realized that many of the items would be much better automated. For example, suppose a list item was "Ensure command X is called with arguments -a, -b, and -c, then command Y is called". This could all be wrapped into a simple script that calls these commands, eliminating the potential for human error. I've found that as I create checklists, they often turn into a list of things I really need to automate.
- Cyranix 10y ago
  Definitely agree — and that's not a knock against checklists. Externalizing a process that previously existed only in someone's head is a win. Creating a checklist is a straightforward framework for such externalizing, and allows you to separate the question "how do I accomplish my goal?" from the question "how do I express this with code?" (like writing pseudocode before real code). Whether a process is automated or manual, there's always room for error if the surrounding context changes. When an automated process is annotated _like_ a checklist, I find that I get the best of both worlds: minimal affordance for human error with a clearly described thought process to fall back on in the event of a problem. (It's also not terribly uncommon to have steps that can't be fully automated, like authenticating to a VPN with 2FA under certain security frameworks...!)
- MaulingMonkey 10y ago
  > I'm a big fan of checklists as well; however, working through a checklist is still a manual process with room for human error. I've had at least one checklist where, by following it religiously, human error was rare - and by paying proper attention, what errors did occur were noticeable and correctable. Handed off the checklist to someone else (at management's request - it was eating up some of my time) and the human error rate went to nearly 100%. Great guy, good at his main job, just not fastidious about checklist discipline. I ended up creating an incredibly brittle, constantly breaking, broken-as-hell time sinking set of scripts to automate the process on our build servers, because that was less work. Sigh.

220 points by slyall 10y ago

rb808 10y ago

Great article. Documentation is always tough - I still haven't found the right solution. Its nice to see some SRE people that care, most 1st/2nd line support I've worked seem to just escalate anything non-trivial which is frustrating, but also useful. If SREs are too good developers lose touch with production and get lazy.

zhengyi13 10y ago
Reflecting on my time in frontline support, 1st level folk have limited skills, limited resources in terms of time and tooling, and in particular tend to have pretty tight metrics applied to them. They pretty much have to escalate quickly, or they'll be yelled at, or worse. Some of them flat out don't belong in Support. But poor management and poor metrics drive behavior in unhelpful directions, too.
zwischenzug 10y ago
Yeah, we were lucky in that we only had engineers with years of experience on call. They had to have had dev experience, too. This followed an attempt to run a more 'standard' support service (before my time) with specialist support staff. That failed badly, mostly because customers and devs hated that they added little value to the process. It was a long time before people were allowed to be recruited direct to support again, and by then the leaders were all former devs->tech leads.

jldugger 10y ago

> The team I joined was around 5 engineers (all former developers and technical leaders), which grew to around 50 of more mixed experience across multiple locations by the time I left. Unfortunately, it's difficult for the audience to determine whether the author's success is attributable to the philosophy in the post, or the 10x growth in staff.

stephengillie 10y ago
The author attributes both his success, and the team's growth, to his documentation philosophy. It does make some sense - improving process documentation for the most common issues will help ensure new team members resolve these issues consistently and effectively, both increasing their utility to the team and their personal morale. How much is directly attributable to him is open to debate.
- zwischenzug 10y ago
  The team's growth was partly because of its perceived success at managing operations, but also because of large growth in customer base and application sprawl. The business model we had depended on fast releases and customers who refused to pay for much-needed testing or QA, but were happy to pay for the support we provided. Not saying that's a good thing, but it was the plate we were served.
zwischenzug 10y ago
The quantity of issues dealt with went up more than 10x over that time, and the experience of the people dealing with them dropped. Whether that was due to me, the things we did, or other reasons is of course open to debate. This is just my view from where I sat.

elorant 10y ago

I wish there were more articles like this for sites that are considered gray areas like gambling or porn.

kbenson 10y ago
For porn, Youporn has had some technical (or technical PR) submissions multiple times over the years. "Youporn.com is now a 100% Redis Site" (126 comments, 5 years ago)[1] "How YouPorn Uses Redis: SFW Edition" (95 comments, 4 years ago)[2] "YouPorn: Symfony2, Redis, Varnish, HA Proxy... (Keynote at ConFoo 2012)" (49 comments, 5 years ago) There's more in the HN search. 1: https://news.ycombinator.com/item?id=3597891 https://news.ycombinator.com/item?id=3597891 2: https://news.ycombinator.com/item?id=6137087 https://news.ycombinator.com/item?id=6137087 3: https://news.ycombinator.com/item?id=3750060 https://news.ycombinator.com/item?id=3750060

systems 10y ago

i am always surprised, that gambling sites and similar business with questionable morality dont face hard time recruiting top engineers , i always though this would be a major concern

fasteo 10y ago
Honest question. Do you think that they have "questionable morality" by the business nature (gambling, porn, etc) or for how they run the business (ex. dark patterns) ?
- systems 10y ago
  i had business nature in mind
jtreminio 10y ago
Gambling itself isn't immoral.
- solotronics 10y ago
  some people think it is morality is subjective
  - akvadrako 10y ago
    If morality is subjective is itself up for debate. I personally think morality is absolute. There is only one system of morality which isn't arbitrary and allowing morals to be arbitrary strips the word of any meaning.
    - brokenmachine 10y ago
      Do a google search for "moral dilemma" and you will find many examples where it is very difficult to come up with simplistic black and white opinions. Such dilemmas leave me with the idea that it is very naive to think there is some kind of absolute morality.

jedberg 10y ago

I'm going to have to respectfully disagree with a big chunk of this article. Documentation is generally a waste of time unless you have a very static infrastructure, and run books are the devil. You should never use a run book -- instead you should spend the time you were going to write a run book writing code to execute the steps automatically. This will reduce human error and make things faster and more repeatable. Even better is if the person who wrote the code also writes the automation to fix it so that it stays up to date with changes in the code. At Netflix we tried to avoid spending a lot of time on documentation because by the time the document was done, it was out of date. Almost inevitably any time you needed the documentation, it no longer applied. I wish the author had spent more time on talking about incident reviews. Those were our key to success. After every event, you review the event with everyone involved, including the developers, and then come up with an action plan that at a minimum prevents the same problem from happening again, but even better, prevents an entire class of problems from happening again. Then you have to follow through and make sure the changes are actually getting implemented. I agree with the author on the point about culture. That was absolutely critical. You need a culture that isn't about placing blame but finding solutions. One where people feel comfortable, and even eager, to come out and say "It was my fault, here's the problem, and here's how I'm going to fix it!"

zwischenzug 10y ago
'Documentation is generally a waste of time unless you have a very static infrastructure' I definitely agree with that, and it's partly a corollary of 'documentation is expensive and requires costly maintenance'. Run books/checklists are mostly implemented really really badly. Automation is the ideal, but is costly, and itself requires maintenance. Most of the steps we had to perform did not lend themselves to automation, also.
- jedberg 10y ago
  > Automation is the ideal, but is costly, and itself requires maintenance. I would contend that the cost of automation is about the same as the cost of documentation plus the cost of having to manually do the work over an over. It's just a cost borne up front instead of over time. But to your point in the article, you have to have a culture that supports bearing that up front cost. > Most of the steps we had to perform did not lend themselves to automation, also. I don't understand how that is possible? Could you give an example of a task that can't be automated?

mfonda 10y ago

I'm a big fan of checklists as well; however, working through a checklist is still a manual process with room for human error. I've gone down the road of creating checklists, then realized that many of the items would be much better automated. For example, suppose a list item was "Ensure command X is called with arguments -a, -b, and -c, then command Y is called". This could all be wrapped into a simple script that calls these commands, eliminating the potential for human error. I've found that as I create checklists, they often turn into a list of things I really need to automate.

Cyranix 10y ago
Definitely agree — and that's not a knock against checklists. Externalizing a process that previously existed only in someone's head is a win. Creating a checklist is a straightforward framework for such externalizing, and allows you to separate the question "how do I accomplish my goal?" from the question "how do I express this with code?" (like writing pseudocode before real code). Whether a process is automated or manual, there's always room for error if the surrounding context changes. When an automated process is annotated _like_ a checklist, I find that I get the best of both worlds: minimal affordance for human error with a clearly described thought process to fall back on in the event of a problem. (It's also not terribly uncommon to have steps that can't be fully automated, like authenticating to a VPN with 2FA under certain security frameworks...!)
MaulingMonkey 10y ago
> I'm a big fan of checklists as well; however, working through a checklist is still a manual process with room for human error. I've had at least one checklist where, by following it religiously, human error was rare - and by paying proper attention, what errors did occur were noticeable and correctable. Handed off the checklist to someone else (at management's request - it was eating up some of my time) and the human error rate went to nearly 100%. Great guy, good at his main job, just not fastidious about checklist discipline. I ended up creating an incredibly brittle, constantly breaking, broken-as-hell time sinking set of scripts to automate the process on our build servers, because that was less work. Sigh.