10 ms·

Automated Testing for League of Legends

246 points by jsnell 11y ago

swanson 11y ago
A great anecdote in the 'we can't test this code' discussion -- for an aggressive two-week release cycle of this massively popular game, this level of testing has probably paid for itself many times over.
- dimino 11y ago
  In this case? Probably. But the idea that catching a bug in production is a huge problem is a myth, pushed by research that was later completely withdrawn.
  - debaserab2 11y ago
    When customers discover bugs before you do, it reflects poorly on your reputation and harms the trust you have established with the consumer of your application. That's not a myth.
    - im_down_w_otp 11y ago
      Or you might also work in a hyper-regulated marketplace dealing with huge amounts of trust and money, and so finding a bug in production can be a very, very, very big deal.
      - dimino 11y ago
        Obviously, but not always, or even usually.
    - dimino 11y ago
      Depends on your domain. Facebook bugs are less costly to deploy fixes for than router firmware bugs. The specific thing I referred to as a myth, though? That is completely fabricated, the guy lied in his research. Total myth.
      - meowface 11y ago
        A particular study being debunked does not necessarily make its hypothesis a "myth".
      - MichaelBurge 11y ago
        I could publish a study saying smoking is bad for you, intentionally make some mistakes, and have the study withdrawn using strong words like 'fabricated' or 'lied'. The result would be a dimino-compliant way of establishing that smoking is not bad for you. Or at least, that "smoking is bad for you" is a myth. Of course, for smoking we have dozens of studies and a strong understanding of the mechanisms, so this trick wouldn't fool anyone. But, just be careful about letting fraudsters determine your opinion on a topic, whether for or against, whether they're caught or not. As for bugs specifically, there are huge benefits to being able to reason about systems as if they are bug-free. It's a PITA when your libraries, kernel, compiler, upstream API, etc. don't behave as expected. As professional software developers, we can work around them; users can sometimes get really confused when things don't work as expected. Sometimes they're scared to tell you. Smaller companies can probably bear more bugs, since they can give individual users more attention. Larger ones tend to need to work more reliably.
        dimino 11y ago
        I don't think the characterization you're making of my argument is accurate, and I don't get the sense you're interested in understanding what I'm explaining here, so I'm probably done. I'll leave you with a recommendation to read The Startup Owner's Handbook, and listen to the Stanford lectures given by Sam Altman and friends, who talk a lot more about this.
  - swanson 11y ago
    Are you referring to research on the 'cost of change curve' or research on the economic impact of production bugs?
    - dimino 11y ago
      I am on mobile, so it's hard to provide links, but I'm referring to the "cost of bug in production" chart that's one of the more famous studies, from a citations perspective. Completely fabricated research.
  - jnardiello 11y ago
    > But the idea that catching a bug in production is a huge problem is a myth You clearly have never worked in fintech, banks or payments. I personally witnessed the moment we catched a race condition which costed the company n-k euros in missed transactions. And this isn't even considering mission critical software where bugs can (literally) kill.
    - dimino 11y ago
      Relax, I don't think I said bugs are never bad, or devastating. I said the idea that catching a bug in production is a huge problem is a myth. It is not, by default, true. It is true given additional information for a specific case, but is not the general rule.
      - jerf 11y ago
        Bug impacts are highly non-linearly and non-Gaussianly distributed. It doesn't matter if the "average" bug is not that big a deal to find in production, what matters is what the worst thing you can find in production is. Even if the study is not literally correct in the average case, it doesn't take much modeling or much real-world experience to see one is still wise to develop with very similar ideas in mind, if not an even more intense focus on getting bugs identified before production because the paper probably understates the impact given real bug distributions. (I say "identified" because you don't always fix identified bugs. But you really, really want to have as much as possible identified before production. Production is a terrible place to find bugs for the first time. Yeah, every once in a while an identified bug might have a far worse than expected impact, I've had that happen, but at least in my experience the completely unidentified bugs are what kill you the hardest.)
        dimino 11y ago
        This is an older style of thinking that is, while safe, slower. Competition who doesn't follow this model and pushes fast, failing often will generally run faster to market than a company who follows this line of thinking. Their software might be buggier, but the concept of "good enough" applies. If you're building rocket engines that carry people, you think like this. If you're building a social networking website, or a food ordering website, or a home sharing website, the time it takes to go from "oh there's a bug" to "that bug is fixed in production" is a matter of hours, if not minutes.
  - developer2 11y ago
    Depends on the bug. Sticking to League of Legends as an example, is it really the worst thing if they have to disable a champion for a day or two to fix a bug? Not really. It still happens fairly often, and the impact is inconvenient but it's not the end of the world. Now how about a bug that prevents logins or crashes the backend? This completely take the game offline. These sort of downtimes hurt customer satisfaction and the bottom line. If serious issues arise often enough, customers lose confidence and patience and may leave your product for a competitor. And while you're offline trying desperately to rush a fix, you're losing revenue. If the service/product you're offering has competitors, quality matters. Sometimes it's not even the measurable quality, but the perception your customers have. Find me a business owner who thinks major bugs in production are not "a huge problem".
    - dimino 11y ago
      > Find me a business owner who thinks major bugs in production are not "a huge problem". "Move fast and break things" - Mark Zuckerberg.
      - Ralfp 11y ago
        They moved away from that, didn't they?
        developer2 11y ago
        Indeed. Quote from Mark: >> We used to have this famous mantra... and the idea here is that as developers, moving quickly is so important that we were even willing to tolerate a few bugs in order to do it. What we realized over time is that it wasn't helping us to move faster because we had to slow down to fix these bugs and it wasn't improving our speed.
        dimino 11y ago
        They grew as a company to a place where they could slow down.
      - 11y ago
- jakevoytko 11y ago
  It definitely has. League of Legends is considered a serious e-sport game. Individual tournaments for League of Legends have $2,000,000+ prize pools. If a new bug decided a round of the tournament, it would undermine LOL as a serious contender for this type of attention. There is a terrific business case for investing the time to make this game as stable as possible.
  - mashlol 11y ago
    This has actually happened before at official League of Legends tournaments (that is, a fairly serious bug in a serious match). I believe their policy in the case of a serious bug is to re-do the match, but I am not 100% sure.
    - chacham15 11y ago
      It depends. If the bug was caught before game of record, you just restart the game. Afterwards, it must qualify as a game deciding bug. If so, the game gets scrapped and the match is redone.
  - ReadingInBed 11y ago
    This actually happened last year at their biggest tournament. A well known bug caused several champions to be disabled midway through the tournament. This happens regularly that champions are disabled due to bugs at international events.
amenod 11y ago
Nice writeup! Sounds pretty similar to the way serious web apps are being tested (using Selenium & co.). I find it interesting that they built their own testing system though - couldn't they have used some existing framework?
- badloginagain 11y ago
  A third party solution would be as hard, or harder, than developing an in-house solution. The challenge is setting up the interactions between all gameplay elements- it's going to very unique to each game. The amount you'd need to abstract to make it reusable for other developers would make it nearly useless.
kyberias 11y ago
Are these all the automated test they run? The article doesn't seem to mention unit tests for example. Do they write and run unit tests? The test class example seems to assert a bunch of stuff that might be easier (and cleaner) to test in a set of unit tests. Update: In particular this is weird: > "Tests make use of remote procedure call (RPC) endpoints exposed on the client and the server in order to issue commands and monitor game state. For the most part, tests consist of a fairly linear set of instructions and queries—existing tests cover everything from champion abilities to vision rules to the expected rewards for a minion kill. " They test "rules for expected rewards" with an out-of-process python test program that connects to the client and server via RPC. Seems unnecessary complex way to test specific game rules. Is this a symptom of a separate QA team and no developer-written unit-tests?
- crdoconnor 11y ago
  It's a more loosely coupled way to test than unit tests. That has certain advantages.
- chipsy 11y ago
  No, it's because game engines don't unit test well. There is too much I/O, latency dependent behavior, and shared mutable state to make it feasible outside of some math and low level format or protocol tests. Many game bugs are purely data bugs and result from a misconfiguration or a malformed asset. Manual testing of the result, following a checklist and creatively hammering on the system, is thus the default method. What Riot is using is a form of integration test where manual user input is emulated to produce a result data set. What you aren't seeing in the test code is "everything else" that was needed to set up a running game state. This technique makes it easy for QA to jump in and see what's happening visually when a failure occurs, eliminating the need to slave away at a checklist.
- badloginagain 11y ago
  It's useful as a framework for testing all game interactions. When you think about the amount of unique abilities for the hundred(?) unique characters and how they interact against eachother, the test matrix that results is massive. Additionally, you want to use production server/client code as much has possible, while getting through the tests as quickly as possible (skip front end flow, matchmaking, etc.). Using client/server endpoint calls are great because it allows these integration tests to be a list of instructions (the same instructions used by production code) to create a very deep test suite. Integration testing is critical for games, where you have so many independent units interacting and changing each other's states.
yokohummer7 11y ago
A release cycle of two weeks looks miracle to an outsider like me. Good to see what's going on internally. I loved the video showing automated champion moves, show me more! Also, > In Wood 5 we don't use wards anyway, so I see no problem with this critical failure 100% agreed. There's no point in changing to lens because nobody buys a ward!
cpeterso 11y ago
I'd love to hear more about automated stress testing and LoL's release schedule. Do they have beta releases or code freeze before the two week release dates?
- badloginagain 11y ago
  If using an svn/p4 source control method, they probably branch mainline into a release branch, test, then branch to a staging, test against production data, etc, all the while fixing any necessary bugs. This allows development work to continue on mainline while keeping the release testing continuous and stable.
- huac 11y ago
  They have a public beta environment that is set up for feature testing, rather than stress testing. The PBE does see code rollbacks (if features are incomplete) before releases, so I believe the game logic on the PBE represents basically what will be moved to the live server.
finishingmove 11y ago
We see the test checks three things: Verifies: - KogMaw deals less damage to non-lane minions - KogMaw deals percentile magic damage - KogMaw deals normal damage to lane minions and the verify method has three assertions. In my opinion, this should be three tests, each with only one assertion.
- swanson 11y ago
  I tend to agree, but sometimes in these kind of instrumented test environments, the setup cost of each test case might be very high so maybe the team found it reasonable to group all the assertions into a single test (especially given that the suite is already taking 1-2 hours to give feedback to developers).
  - badloginagain 11y ago
    It appears they're grouping by ability.
  - finishingmove 11y ago
    I am well aware of that and I agree on being pragmatic, but more often than not, the reason for multiple assertions per test is developer laziness and/or unawareness of the benefits of isolating failures by having just one assertion per test.
    - 0x0 11y ago
      Seems to me you actually want all three assertions checked in the same run. Otherwise you could get three runs where only the first assertion was true in the first run; only the second assertion succeeding in the second run with the others failing; etc. I bet the tests aren't pixel-by-pixel frame-by-frame reproducible (they are talking about a one week staging period for tests to prove themselves, so surely there's some non-deterministic wiggle room in remote controlling the full graphical client), so you really want to cover all assertions together for each run.
      - 11y ago
derFunk 11y ago
Wow, this is pretty sophisticated and commendable. I'd love to have the resources to do something similar. In contrast to 'standard' application development, automated testing is really rare in the non-AAA games industry. At least in terms of logic-/active gameplay testing. It's pure luxury, you can only do it if you can afford it. In a project-based work for hire game shop this is an almost unthinkable thing to do, because you don't get it sold to your contractors/customers. They just won't pay for the effort you don't our directly into the game. The only thing you can do is to develop your own automated testing framework over time and over projects, which is a tedious thing to do because you cannot really focus on it (because it's not a first class citizen in your project schedule).
- badloginagain 11y ago
  I hope it started as a much more humble tool, that kept growing in size as the company did. Each release it gets a bit more useful, until you hit that critical mass point where's it's a pillar of development.
- andrepd 11y ago
  And yet they can't rewrite their ridiculously inefficient launcher properly :^)
  - quaunaut 11y ago
    Actually, the launcher was replaced under 2 years ago. The matchmaking client has entered alpha, and is accompanied with rewrites to a lot of Riot systems to help it all work together.
- huac 11y ago
  On the other hand, just because you have a great testing framework, it doesn't mean you should launch new game features/changes left and right. (LoL is structured around roughly yearly 'seasons' for competitive play and for ranked play rewards. This season they have released a far greater amount of game mechanic changes, which make it difficult to plan competitive strategy and difficult for regular players to keep up.)
JamieAli 11y ago
Honestly I don’t know why some people look at these pictures, especially when they could be getting better at the game. No wonder 90% of people stay at BRONZE level while the big boys-including me, rank up and get DIAMOND level. It wasn't hard once I started using this: http://gamesinfun.com/league-of-legends http://gamesinfun.com/league-of-legends
mangix 11y ago
too bad dota 2 is a better game
- dylz 11y ago
  really, this stuff goes on even on HN?
  - errantspark 11y ago
    Maybe we extend the benefit of the doubt and perceive the comment as a reminder that having code that passes all the tests doesn't necessarily equate to having a good product? It's obviously a stretch as that guy's username is the name of a DotA hero, but hey, rose colored glasses are fun sometimes.
- kawsper 11y ago
  Funnily enough, the latest patch was rushed out by Valve, and it contained a lot of bugs and strange behavior. Lots of bugs in Dota 2 seems to be happening because of manual testing.
  - errantspark 11y ago
    Not to say that DotA wouldn't benefit from more rigorous testing, but in Valve's defense I'd say that the complexity of interactions in DotA is far beyond LoL. Some of the bugs you're seeing might only arise in extremely rare situations. DotA is balanced with creativity in mind. When an exploitative strategy is found the game is not changed to remove it but instead through some combination of tweaks and modifications, often to seemingly unrelated aspects it's made into something that's only situationally viable. The reason the game is so deep/interesting is because this philosophy has been in place for many (10+) years now resulting in a game that's much more complex than pretty much anything else out there. DotA 2 retains edge case behavior that was present in the original Warcraft 3 mod. Things that were once engine limitations are now part essential parts of gameplay. Preserving these weird interactions instead of attempting to make the game more regular/comprehensible is one of the things that allows such a high level of creativity in competitive play. There's always a way you can outplay your opponent that's clever. In go parlance: there's always new tesuji to be found. All that being said I think Riot's efforts here are quite laudable and DotA could certainly find benefit from more rigorous testing.
amleszk 11y ago
Wow this is the first time I've seen automation testing applied to a video game. 2 things struck me as really great from this: - Staging area for their tests, so many times I've lost confidence in our test suite because of a flaky test. We then delete that test forever. Adding tests to a staging area to very that they are stable is a great compromise - each test is a class, with 'setup', 'execute' 'verify' - makes the test a lot easier to read and refactor
bruxis 11y ago
This is beautiful.
Ameo 11y ago
Oh man with that kind of control of the game they could /certainly/ create some kind of sandbox mode. The APIs that they have for testing alone are pretty incredible; I'd love to be able to play in a mode where you could change level on-demand, create dummy enemies, etc.
BuckRogers 11y ago
With LoL being the most popular video game in the world right now, it's also the most relevant game of all time. How they build and test it is fascinating. I've played this game for 7+ years now and as a programmer love to hear about all this. A Rioter said a couple years ago that they used Erlang[0] and as a longtime Erlang (and now Elixir) admirer and tinkerer that was great to know my favorite game uses it. [0]https://erlangcentral.org/scaling-league-of-legends-chat-to-70-million-players-by-michal-ptaszek/ https://erlangcentral.org/scaling-league-of-legends-chat-to-...

246 points by jsnell 11y ago

A great anecdote in the 'we can't test this code' discussion -- for an aggressive two-week release cycle of this massively popular game, this level of testing has probably paid for itself many times over.

dimino 11y ago
In this case? Probably. But the idea that catching a bug in production is a huge problem is a myth, pushed by research that was later completely withdrawn.
- debaserab2 11y ago
  When customers discover bugs before you do, it reflects poorly on your reputation and harms the trust you have established with the consumer of your application. That's not a myth.
  - im_down_w_otp 11y ago
    Or you might also work in a hyper-regulated marketplace dealing with huge amounts of trust and money, and so finding a bug in production can be a very, very, very big deal.
    - dimino 11y ago
      Obviously, but not always, or even usually.
  - dimino 11y ago
    Depends on your domain. Facebook bugs are less costly to deploy fixes for than router firmware bugs. The specific thing I referred to as a myth, though? That is completely fabricated, the guy lied in his research. Total myth.
    - meowface 11y ago
      A particular study being debunked does not necessarily make its hypothesis a "myth".
    - MichaelBurge 11y ago
      I could publish a study saying smoking is bad for you, intentionally make some mistakes, and have the study withdrawn using strong words like 'fabricated' or 'lied'. The result would be a dimino-compliant way of establishing that smoking is not bad for you. Or at least, that "smoking is bad for you" is a myth. Of course, for smoking we have dozens of studies and a strong understanding of the mechanisms, so this trick wouldn't fool anyone. But, just be careful about letting fraudsters determine your opinion on a topic, whether for or against, whether they're caught or not. As for bugs specifically, there are huge benefits to being able to reason about systems as if they are bug-free. It's a PITA when your libraries, kernel, compiler, upstream API, etc. don't behave as expected. As professional software developers, we can work around them; users can sometimes get really confused when things don't work as expected. Sometimes they're scared to tell you. Smaller companies can probably bear more bugs, since they can give individual users more attention. Larger ones tend to need to work more reliably.
      - dimino 11y ago
        I don't think the characterization you're making of my argument is accurate, and I don't get the sense you're interested in understanding what I'm explaining here, so I'm probably done. I'll leave you with a recommendation to read The Startup Owner's Handbook, and listen to the Stanford lectures given by Sam Altman and friends, who talk a lot more about this.
- swanson 11y ago
  Are you referring to research on the 'cost of change curve' or research on the economic impact of production bugs?
  - dimino 11y ago
    I am on mobile, so it's hard to provide links, but I'm referring to the "cost of bug in production" chart that's one of the more famous studies, from a citations perspective. Completely fabricated research.
- jnardiello 11y ago
  > But the idea that catching a bug in production is a huge problem is a myth You clearly have never worked in fintech, banks or payments. I personally witnessed the moment we catched a race condition which costed the company n-k euros in missed transactions. And this isn't even considering mission critical software where bugs can (literally) kill.
  - dimino 11y ago
    Relax, I don't think I said bugs are never bad, or devastating. I said the idea that catching a bug in production is a huge problem is a myth. It is not, by default, true. It is true given additional information for a specific case, but is not the general rule.
    - jerf 11y ago
      Bug impacts are highly non-linearly and non-Gaussianly distributed. It doesn't matter if the "average" bug is not that big a deal to find in production, what matters is what the worst thing you can find in production is. Even if the study is not literally correct in the average case, it doesn't take much modeling or much real-world experience to see one is still wise to develop with very similar ideas in mind, if not an even more intense focus on getting bugs identified before production because the paper probably understates the impact given real bug distributions. (I say "identified" because you don't always fix identified bugs. But you really, really want to have as much as possible identified before production. Production is a terrible place to find bugs for the first time. Yeah, every once in a while an identified bug might have a far worse than expected impact, I've had that happen, but at least in my experience the completely unidentified bugs are what kill you the hardest.)
      - dimino 11y ago
        This is an older style of thinking that is, while safe, slower. Competition who doesn't follow this model and pushes fast, failing often will generally run faster to market than a company who follows this line of thinking. Their software might be buggier, but the concept of "good enough" applies. If you're building rocket engines that carry people, you think like this. If you're building a social networking website, or a food ordering website, or a home sharing website, the time it takes to go from "oh there's a bug" to "that bug is fixed in production" is a matter of hours, if not minutes.
- developer2 11y ago
  Depends on the bug. Sticking to League of Legends as an example, is it really the worst thing if they have to disable a champion for a day or two to fix a bug? Not really. It still happens fairly often, and the impact is inconvenient but it's not the end of the world. Now how about a bug that prevents logins or crashes the backend? This completely take the game offline. These sort of downtimes hurt customer satisfaction and the bottom line. If serious issues arise often enough, customers lose confidence and patience and may leave your product for a competitor. And while you're offline trying desperately to rush a fix, you're losing revenue. If the service/product you're offering has competitors, quality matters. Sometimes it's not even the measurable quality, but the perception your customers have. Find me a business owner who thinks major bugs in production are not "a huge problem".
  - dimino 11y ago
    > Find me a business owner who thinks major bugs in production are not "a huge problem". "Move fast and break things" - Mark Zuckerberg.
    - Ralfp 11y ago
      They moved away from that, didn't they?
      - developer2 11y ago
        Indeed. Quote from Mark: >> We used to have this famous mantra... and the idea here is that as developers, moving quickly is so important that we were even willing to tolerate a few bugs in order to do it. What we realized over time is that it wasn't helping us to move faster because we had to slow down to fix these bugs and it wasn't improving our speed.
        dimino 11y ago
        They grew as a company to a place where they could slow down.
    - 11y ago
jakevoytko 11y ago
It definitely has. League of Legends is considered a serious e-sport game. Individual tournaments for League of Legends have $2,000,000+ prize pools. If a new bug decided a round of the tournament, it would undermine LOL as a serious contender for this type of attention. There is a terrific business case for investing the time to make this game as stable as possible.
- mashlol 11y ago
  This has actually happened before at official League of Legends tournaments (that is, a fairly serious bug in a serious match). I believe their policy in the case of a serious bug is to re-do the match, but I am not 100% sure.
  - chacham15 11y ago
    It depends. If the bug was caught before game of record, you just restart the game. Afterwards, it must qualify as a game deciding bug. If so, the game gets scrapped and the match is redone.
- ReadingInBed 11y ago
  This actually happened last year at their biggest tournament. A well known bug caused several champions to be disabled midway through the tournament. This happens regularly that champions are disabled due to bugs at international events.

amenod 11y ago

Nice writeup! Sounds pretty similar to the way serious web apps are being tested (using Selenium & co.). I find it interesting that they built their own testing system though - couldn't they have used some existing framework?

badloginagain 11y ago
A third party solution would be as hard, or harder, than developing an in-house solution. The challenge is setting up the interactions between all gameplay elements- it's going to very unique to each game. The amount you'd need to abstract to make it reusable for other developers would make it nearly useless.

kyberias 11y ago

Are these all the automated test they run? The article doesn't seem to mention unit tests for example. Do they write and run unit tests? The test class example seems to assert a bunch of stuff that might be easier (and cleaner) to test in a set of unit tests. Update: In particular this is weird: > "Tests make use of remote procedure call (RPC) endpoints exposed on the client and the server in order to issue commands and monitor game state. For the most part, tests consist of a fairly linear set of instructions and queries—existing tests cover everything from champion abilities to vision rules to the expected rewards for a minion kill. " They test "rules for expected rewards" with an out-of-process python test program that connects to the client and server via RPC. Seems unnecessary complex way to test specific game rules. Is this a symptom of a separate QA team and no developer-written unit-tests?

crdoconnor 11y ago
It's a more loosely coupled way to test than unit tests. That has certain advantages.
chipsy 11y ago
No, it's because game engines don't unit test well. There is too much I/O, latency dependent behavior, and shared mutable state to make it feasible outside of some math and low level format or protocol tests. Many game bugs are purely data bugs and result from a misconfiguration or a malformed asset. Manual testing of the result, following a checklist and creatively hammering on the system, is thus the default method. What Riot is using is a form of integration test where manual user input is emulated to produce a result data set. What you aren't seeing in the test code is "everything else" that was needed to set up a running game state. This technique makes it easy for QA to jump in and see what's happening visually when a failure occurs, eliminating the need to slave away at a checklist.
badloginagain 11y ago
It's useful as a framework for testing all game interactions. When you think about the amount of unique abilities for the hundred(?) unique characters and how they interact against eachother, the test matrix that results is massive. Additionally, you want to use production server/client code as much has possible, while getting through the tests as quickly as possible (skip front end flow, matchmaking, etc.). Using client/server endpoint calls are great because it allows these integration tests to be a list of instructions (the same instructions used by production code) to create a very deep test suite. Integration testing is critical for games, where you have so many independent units interacting and changing each other's states.

yokohummer7 11y ago

A release cycle of two weeks looks miracle to an outsider like me. Good to see what's going on internally. I loved the video showing automated champion moves, show me more! Also, > In Wood 5 we don't use wards anyway, so I see no problem with this critical failure 100% agreed. There's no point in changing to lens because nobody buys a ward!

cpeterso 11y ago

I'd love to hear more about automated stress testing and LoL's release schedule. Do they have beta releases or code freeze before the two week release dates?

badloginagain 11y ago
If using an svn/p4 source control method, they probably branch mainline into a release branch, test, then branch to a staging, test against production data, etc, all the while fixing any necessary bugs. This allows development work to continue on mainline while keeping the release testing continuous and stable.
huac 11y ago
They have a public beta environment that is set up for feature testing, rather than stress testing. The PBE does see code rollbacks (if features are incomplete) before releases, so I believe the game logic on the PBE represents basically what will be moved to the live server.

finishingmove 11y ago

We see the test checks three things: Verifies: - KogMaw deals less damage to non-lane minions - KogMaw deals percentile magic damage - KogMaw deals normal damage to lane minions and the verify method has three assertions. In my opinion, this should be three tests, each with only one assertion.

swanson 11y ago
I tend to agree, but sometimes in these kind of instrumented test environments, the setup cost of each test case might be very high so maybe the team found it reasonable to group all the assertions into a single test (especially given that the suite is already taking 1-2 hours to give feedback to developers).
- badloginagain 11y ago
  It appears they're grouping by ability.
- finishingmove 11y ago
  I am well aware of that and I agree on being pragmatic, but more often than not, the reason for multiple assertions per test is developer laziness and/or unawareness of the benefits of isolating failures by having just one assertion per test.
  - 0x0 11y ago
    Seems to me you actually want all three assertions checked in the same run. Otherwise you could get three runs where only the first assertion was true in the first run; only the second assertion succeeding in the second run with the others failing; etc. I bet the tests aren't pixel-by-pixel frame-by-frame reproducible (they are talking about a one week staging period for tests to prove themselves, so surely there's some non-deterministic wiggle room in remote controlling the full graphical client), so you really want to cover all assertions together for each run.
    - 11y ago

derFunk 11y ago

Wow, this is pretty sophisticated and commendable. I'd love to have the resources to do something similar. In contrast to 'standard' application development, automated testing is really rare in the non-AAA games industry. At least in terms of logic-/active gameplay testing. It's pure luxury, you can only do it if you can afford it. In a project-based work for hire game shop this is an almost unthinkable thing to do, because you don't get it sold to your contractors/customers. They just won't pay for the effort you don't our directly into the game. The only thing you can do is to develop your own automated testing framework over time and over projects, which is a tedious thing to do because you cannot really focus on it (because it's not a first class citizen in your project schedule).

badloginagain 11y ago
I hope it started as a much more humble tool, that kept growing in size as the company did. Each release it gets a bit more useful, until you hit that critical mass point where's it's a pillar of development.
andrepd 11y ago
And yet they can't rewrite their ridiculously inefficient launcher properly :^)
- quaunaut 11y ago
  Actually, the launcher was replaced under 2 years ago. The matchmaking client has entered alpha, and is accompanied with rewrites to a lot of Riot systems to help it all work together.
huac 11y ago
On the other hand, just because you have a great testing framework, it doesn't mean you should launch new game features/changes left and right. (LoL is structured around roughly yearly 'seasons' for competitive play and for ranked play rewards. This season they have released a far greater amount of game mechanic changes, which make it difficult to plan competitive strategy and difficult for regular players to keep up.)

JamieAli 11y ago

Honestly I don’t know why some people look at these pictures, especially when they could be getting better at the game. No wonder 90% of people stay at BRONZE level while the big boys-including me, rank up and get DIAMOND level. It wasn't hard once I started using this: http://gamesinfun.com/league-of-legends http://gamesinfun.com/league-of-legends

mangix 11y ago

too bad dota 2 is a better game

dylz 11y ago
really, this stuff goes on even on HN?
- errantspark 11y ago
  Maybe we extend the benefit of the doubt and perceive the comment as a reminder that having code that passes all the tests doesn't necessarily equate to having a good product? It's obviously a stretch as that guy's username is the name of a DotA hero, but hey, rose colored glasses are fun sometimes.
kawsper 11y ago
Funnily enough, the latest patch was rushed out by Valve, and it contained a lot of bugs and strange behavior. Lots of bugs in Dota 2 seems to be happening because of manual testing.
- errantspark 11y ago
  Not to say that DotA wouldn't benefit from more rigorous testing, but in Valve's defense I'd say that the complexity of interactions in DotA is far beyond LoL. Some of the bugs you're seeing might only arise in extremely rare situations. DotA is balanced with creativity in mind. When an exploitative strategy is found the game is not changed to remove it but instead through some combination of tweaks and modifications, often to seemingly unrelated aspects it's made into something that's only situationally viable. The reason the game is so deep/interesting is because this philosophy has been in place for many (10+) years now resulting in a game that's much more complex than pretty much anything else out there. DotA 2 retains edge case behavior that was present in the original Warcraft 3 mod. Things that were once engine limitations are now part essential parts of gameplay. Preserving these weird interactions instead of attempting to make the game more regular/comprehensible is one of the things that allows such a high level of creativity in competitive play. There's always a way you can outplay your opponent that's clever. In go parlance: there's always new tesuji to be found. All that being said I think Riot's efforts here are quite laudable and DotA could certainly find benefit from more rigorous testing.

amleszk 11y ago

Wow this is the first time I've seen automation testing applied to a video game. 2 things struck me as really great from this: - Staging area for their tests, so many times I've lost confidence in our test suite because of a flaky test. We then delete that test forever. Adding tests to a staging area to very that they are stable is a great compromise - each test is a class, with 'setup', 'execute' 'verify' - makes the test a lot easier to read and refactor

bruxis 11y ago

This is beautiful.

Ameo 11y ago

Oh man with that kind of control of the game they could /certainly/ create some kind of sandbox mode. The APIs that they have for testing alone are pretty incredible; I'd love to be able to play in a mode where you could change level on-demand, create dummy enemies, etc.

BuckRogers 11y ago

With LoL being the most popular video game in the world right now, it's also the most relevant game of all time. How they build and test it is fascinating. I've played this game for 7+ years now and as a programmer love to hear about all this. A Rioter said a couple years ago that they used Erlang[0] and as a longtime Erlang (and now Elixir) admirer and tinkerer that was great to know my favorite game uses it. [0]https://erlangcentral.org/scaling-league-of-legends-chat-to-70-million-players-by-michal-ptaszek/ https://erlangcentral.org/scaling-league-of-legends-chat-to-...