6 ms·

A great anecdote in the 'we can't test this code' discussion -- for an aggressive two-week release cycle of this massively popular game, this level of testing h

by swanson 10y ago

A great anecdote in the 'we can't test this code' discussion -- for an aggressive two-week release cycle of this massively popular game, this level of testing has probably paid for itself many times over.

dimino 10y ago
In this case? Probably. But the idea that catching a bug in production is a huge problem is a myth, pushed by research that was later completely withdrawn.
- debaserab2 10y ago
  When customers discover bugs before you do, it reflects poorly on your reputation and harms the trust you have established with the consumer of your application. That's not a myth.
  - im_down_w_otp 10y ago
    Or you might also work in a hyper-regulated marketplace dealing with huge amounts of trust and money, and so finding a bug in production can be a very, very, very big deal.
    - dimino 10y ago
      Obviously, but not always, or even usually.
  - dimino 10y ago
    Depends on your domain. Facebook bugs are less costly to deploy fixes for than router firmware bugs. The specific thing I referred to as a myth, though? That is completely fabricated, the guy lied in his research. Total myth.
    - meowface 10y ago
      A particular study being debunked does not necessarily make its hypothesis a "myth".
    - MichaelBurge 10y ago
      I could publish a study saying smoking is bad for you, intentionally make some mistakes, and have the study withdrawn using strong words like 'fabricated' or 'lied'. The result would be a dimino-compliant way of establishing that smoking is not bad for you. Or at least, that "smoking is bad for you" is a myth. Of course, for smoking we have dozens of studies and a strong understanding of the mechanisms, so this trick wouldn't fool anyone. But, just be careful about letting fraudsters determine your opinion on a topic, whether for or against, whether they're caught or not. As for bugs specifically, there are huge benefits to being able to reason about systems as if they are bug-free. It's a PITA when your libraries, kernel, compiler, upstream API, etc. don't behave as expected. As professional software developers, we can work around them; users can sometimes get really confused when things don't work as expected. Sometimes they're scared to tell you. Smaller companies can probably bear more bugs, since they can give individual users more attention. Larger ones tend to need to work more reliably.
      - dimino 10y ago
        I don't think the characterization you're making of my argument is accurate, and I don't get the sense you're interested in understanding what I'm explaining here, so I'm probably done. I'll leave you with a recommendation to read The Startup Owner's Handbook, and listen to the Stanford lectures given by Sam Altman and friends, who talk a lot more about this.
- swanson 10y ago
  Are you referring to research on the 'cost of change curve' or research on the economic impact of production bugs?
  - dimino 10y ago
    I am on mobile, so it's hard to provide links, but I'm referring to the "cost of bug in production" chart that's one of the more famous studies, from a citations perspective. Completely fabricated research.
- jnardiello 10y ago
  > But the idea that catching a bug in production is a huge problem is a myth You clearly have never worked in fintech, banks or payments. I personally witnessed the moment we catched a race condition which costed the company n-k euros in missed transactions. And this isn't even considering mission critical software where bugs can (literally) kill.
  - dimino 10y ago
    Relax, I don't think I said bugs are never bad, or devastating. I said the idea that catching a bug in production is a huge problem is a myth. It is not, by default, true. It is true given additional information for a specific case, but is not the general rule.
    - jerf 10y ago
      Bug impacts are highly non-linearly and non-Gaussianly distributed. It doesn't matter if the "average" bug is not that big a deal to find in production, what matters is what the worst thing you can find in production is. Even if the study is not literally correct in the average case, it doesn't take much modeling or much real-world experience to see one is still wise to develop with very similar ideas in mind, if not an even more intense focus on getting bugs identified before production because the paper probably understates the impact given real bug distributions. (I say "identified" because you don't always fix identified bugs. But you really, really want to have as much as possible identified before production. Production is a terrible place to find bugs for the first time. Yeah, every once in a while an identified bug might have a far worse than expected impact, I've had that happen, but at least in my experience the completely unidentified bugs are what kill you the hardest.)
      - dimino 10y ago
        This is an older style of thinking that is, while safe, slower. Competition who doesn't follow this model and pushes fast, failing often will generally run faster to market than a company who follows this line of thinking. Their software might be buggier, but the concept of "good enough" applies. If you're building rocket engines that carry people, you think like this. If you're building a social networking website, or a food ordering website, or a home sharing website, the time it takes to go from "oh there's a bug" to "that bug is fixed in production" is a matter of hours, if not minutes.
- developer2 10y ago
  Depends on the bug. Sticking to League of Legends as an example, is it really the worst thing if they have to disable a champion for a day or two to fix a bug? Not really. It still happens fairly often, and the impact is inconvenient but it's not the end of the world. Now how about a bug that prevents logins or crashes the backend? This completely take the game offline. These sort of downtimes hurt customer satisfaction and the bottom line. If serious issues arise often enough, customers lose confidence and patience and may leave your product for a competitor. And while you're offline trying desperately to rush a fix, you're losing revenue. If the service/product you're offering has competitors, quality matters. Sometimes it's not even the measurable quality, but the perception your customers have. Find me a business owner who thinks major bugs in production are not "a huge problem".
  - dimino 10y ago
    > Find me a business owner who thinks major bugs in production are not "a huge problem". "Move fast and break things" - Mark Zuckerberg.
    - Ralfp 10y ago
      They moved away from that, didn't they?
      - developer2 10y ago
        Indeed. Quote from Mark: >> We used to have this famous mantra... and the idea here is that as developers, moving quickly is so important that we were even willing to tolerate a few bugs in order to do it. What we realized over time is that it wasn't helping us to move faster because we had to slow down to fix these bugs and it wasn't improving our speed.
        dimino 10y ago
        They grew as a company to a place where they could slow down.
    - 10y ago
jakevoytko 10y ago
It definitely has. League of Legends is considered a serious e-sport game. Individual tournaments for League of Legends have $2,000,000+ prize pools. If a new bug decided a round of the tournament, it would undermine LOL as a serious contender for this type of attention. There is a terrific business case for investing the time to make this game as stable as possible.
- mashlol 10y ago
  This has actually happened before at official League of Legends tournaments (that is, a fairly serious bug in a serious match). I believe their policy in the case of a serious bug is to re-do the match, but I am not 100% sure.
  - chacham15 10y ago
    It depends. If the bug was caught before game of record, you just restart the game. Afterwards, it must qualify as a game deciding bug. If so, the game gets scrapped and the match is redone.
- ReadingInBed 10y ago
  This actually happened last year at their biggest tournament. A well known bug caused several champions to be disabled midway through the tournament. This happens regularly that champions are disabled due to bugs at international events.

by swanson 10y ago

dimino 10y ago

In this case? Probably. But the idea that catching a bug in production is a huge problem is a myth, pushed by research that was later completely withdrawn.

debaserab2 10y ago
When customers discover bugs before you do, it reflects poorly on your reputation and harms the trust you have established with the consumer of your application. That's not a myth.
- im_down_w_otp 10y ago
  Or you might also work in a hyper-regulated marketplace dealing with huge amounts of trust and money, and so finding a bug in production can be a very, very, very big deal.
  - dimino 10y ago
    Obviously, but not always, or even usually.
- dimino 10y ago
  Depends on your domain. Facebook bugs are less costly to deploy fixes for than router firmware bugs. The specific thing I referred to as a myth, though? That is completely fabricated, the guy lied in his research. Total myth.
  - meowface 10y ago
    A particular study being debunked does not necessarily make its hypothesis a "myth".
  - MichaelBurge 10y ago
    I could publish a study saying smoking is bad for you, intentionally make some mistakes, and have the study withdrawn using strong words like 'fabricated' or 'lied'. The result would be a dimino-compliant way of establishing that smoking is not bad for you. Or at least, that "smoking is bad for you" is a myth. Of course, for smoking we have dozens of studies and a strong understanding of the mechanisms, so this trick wouldn't fool anyone. But, just be careful about letting fraudsters determine your opinion on a topic, whether for or against, whether they're caught or not. As for bugs specifically, there are huge benefits to being able to reason about systems as if they are bug-free. It's a PITA when your libraries, kernel, compiler, upstream API, etc. don't behave as expected. As professional software developers, we can work around them; users can sometimes get really confused when things don't work as expected. Sometimes they're scared to tell you. Smaller companies can probably bear more bugs, since they can give individual users more attention. Larger ones tend to need to work more reliably.
    - dimino 10y ago
      I don't think the characterization you're making of my argument is accurate, and I don't get the sense you're interested in understanding what I'm explaining here, so I'm probably done. I'll leave you with a recommendation to read The Startup Owner's Handbook, and listen to the Stanford lectures given by Sam Altman and friends, who talk a lot more about this.
swanson 10y ago
Are you referring to research on the 'cost of change curve' or research on the economic impact of production bugs?
- dimino 10y ago
  I am on mobile, so it's hard to provide links, but I'm referring to the "cost of bug in production" chart that's one of the more famous studies, from a citations perspective. Completely fabricated research.
jnardiello 10y ago
> But the idea that catching a bug in production is a huge problem is a myth You clearly have never worked in fintech, banks or payments. I personally witnessed the moment we catched a race condition which costed the company n-k euros in missed transactions. And this isn't even considering mission critical software where bugs can (literally) kill.
- dimino 10y ago
  Relax, I don't think I said bugs are never bad, or devastating. I said the idea that catching a bug in production is a huge problem is a myth. It is not, by default, true. It is true given additional information for a specific case, but is not the general rule.
  - jerf 10y ago
    Bug impacts are highly non-linearly and non-Gaussianly distributed. It doesn't matter if the "average" bug is not that big a deal to find in production, what matters is what the worst thing you can find in production is. Even if the study is not literally correct in the average case, it doesn't take much modeling or much real-world experience to see one is still wise to develop with very similar ideas in mind, if not an even more intense focus on getting bugs identified before production because the paper probably understates the impact given real bug distributions. (I say "identified" because you don't always fix identified bugs. But you really, really want to have as much as possible identified before production. Production is a terrible place to find bugs for the first time. Yeah, every once in a while an identified bug might have a far worse than expected impact, I've had that happen, but at least in my experience the completely unidentified bugs are what kill you the hardest.)
    - dimino 10y ago
      This is an older style of thinking that is, while safe, slower. Competition who doesn't follow this model and pushes fast, failing often will generally run faster to market than a company who follows this line of thinking. Their software might be buggier, but the concept of "good enough" applies. If you're building rocket engines that carry people, you think like this. If you're building a social networking website, or a food ordering website, or a home sharing website, the time it takes to go from "oh there's a bug" to "that bug is fixed in production" is a matter of hours, if not minutes.
developer2 10y ago
Depends on the bug. Sticking to League of Legends as an example, is it really the worst thing if they have to disable a champion for a day or two to fix a bug? Not really. It still happens fairly often, and the impact is inconvenient but it's not the end of the world. Now how about a bug that prevents logins or crashes the backend? This completely take the game offline. These sort of downtimes hurt customer satisfaction and the bottom line. If serious issues arise often enough, customers lose confidence and patience and may leave your product for a competitor. And while you're offline trying desperately to rush a fix, you're losing revenue. If the service/product you're offering has competitors, quality matters. Sometimes it's not even the measurable quality, but the perception your customers have. Find me a business owner who thinks major bugs in production are not "a huge problem".
- dimino 10y ago
  > Find me a business owner who thinks major bugs in production are not "a huge problem". "Move fast and break things" - Mark Zuckerberg.
  - Ralfp 10y ago
    They moved away from that, didn't they?
    - developer2 10y ago
      Indeed. Quote from Mark: >> We used to have this famous mantra... and the idea here is that as developers, moving quickly is so important that we were even willing to tolerate a few bugs in order to do it. What we realized over time is that it wasn't helping us to move faster because we had to slow down to fix these bugs and it wasn't improving our speed.
      - dimino 10y ago
        They grew as a company to a place where they could slow down.
  - 10y ago

jakevoytko 10y ago

It definitely has. League of Legends is considered a serious e-sport game. Individual tournaments for League of Legends have $2,000,000+ prize pools. If a new bug decided a round of the tournament, it would undermine LOL as a serious contender for this type of attention. There is a terrific business case for investing the time to make this game as stable as possible.

mashlol 10y ago
This has actually happened before at official League of Legends tournaments (that is, a fairly serious bug in a serious match). I believe their policy in the case of a serious bug is to re-do the match, but I am not 100% sure.
- chacham15 10y ago
  It depends. If the bug was caught before game of record, you just restart the game. Afterwards, it must qualify as a game deciding bug. If so, the game gets scrapped and the match is redone.
ReadingInBed 10y ago
This actually happened last year at their biggest tournament. A well known bug caused several champions to be disabled midway through the tournament. This happens regularly that champions are disabled due to bugs at international events.