8 ms·

“What Went Right and What Went Wrong”: Analysis of 155 Postmortems from Gamedev

95 points by winterismute 11y ago

jacques_chester 11y ago
PDF is on the right, for anyone else wondering what to do about an abstract that doesn't contain any spoilers.
- brudgers 11y ago
  Link: http://research.microsoft.com/pubs/262301/washburn-icse-2016.pdf http://research.microsoft.com/pubs/262301/washburn-icse-2016...
teddyh 11y ago
> 9. CONCLUSIONS > > We find that we were able to identify both best practices and pitfalls in game development using the information present in the postmortems. Such information on the development of all kinds of software would be highly useful too. Therefore we urge the research community to provide a forum where postmortems on general software development can be presented, and practitioners to report their retrospective thoughts in a postmortem. > > Finally, based on our analysis of the data we collected, we make a few recommendations to game developers. First, be sure to practice good risk management techniques. This will help avoid some of the adverse effects of obstacles that you may encounter during development. Second, prescribe to an iterative development process, and utilize prototypes as a method of proving features and concepts before committing them to your design. Third, don't be overly ambitious in your design. Be reasonable, and take into account your schedule and budget before adding something to your design. Building off of that, don't be overly optimistic with your scheduling. If you make an estimate that initially feels optimistic to you, don't give that estimate to your stakeholders. Revisit and reassess your design to form a better estimation.
- padobson 11y ago
  There's some world-shaking insight. The best piece of advice I can give is to never, ever give an estimate to stakeholders until you've worked on the estimate first. And don't finish an estimate until you've actually built some small proofs of concept. Your estimates will still be wrong, but they'll be much more accurate than some off-the-cuff number that your stakeholders will be building their various plans around. It all sounds straightforward, but the hardest discipline for me as a developer is to keep from saying things in stakeholder meetings that make my stakeholders happy in the moment, but ultimately needed more thought and effort. It's something I struggle with in every such meeting, to varying degrees of success.
  - hluska 11y ago
    I think this is a symptom of a fear of "I don't know." If I could change one thing in dev cultures, I would instill a healthy respect for the sentence: "I don't know, but here's $how_I'll_find_out and here's $when_I'll_have_an_answer."
    - ChuckMcM 11y ago
      Exactly right. Too often I've seen people commit to a schedule where the milestones were of unknown difficulty. When cutting through a new jungle, you have no idea what is between you and your goal, so really all you can say is that the best you can hope for is X if this jungle happens to be like previous ones. This is especially true in startup situations where you are learning technologies that may themselves not be complete, with people who are learning their own new things, to achieve a result which is only hypothetically possible. It drove my CFO nuts but rather than commit to a schedule for a big deliverable I would walk backwards from the end point and say, "These are the stops between where we are, and where we are going. We measure our progress by getting to each stop, but like a subway map there isn't a known amount of time between stops, only what the stops are." And he would come back with "well we only have money to get to this date, will it be done by then?" and then we would talk about the uncertainty between each of the milestones. At some point you can reach a common understanding of what the unknowns are and how discovering their value will inform on the difficulty of the next step. That said, I've met managers who just say "Oh we will be done by %x date." and then basically worked the problem the same way I have. If you're agile you can break down the intermediate stops as sprints but estimating the backlog is still the killer step.
- bottled_poe 11y ago
  Turns out software estimates are hard. Who knew?
ekianjo 11y ago
The recommendations at the end of the pdf seem to be written by captain Obvious. They are so generic they can apply to about ANY project even non gamedev related.
- Kristine1975 11y ago
  >The recommendations at the end of the pdf seem to be written by captain Obvious. Doesn't that simply mean that this research confirms other people's experience/gut feeling?
  - ekianjo 11y ago
    It simply means the research is not insightful.
- yread 11y ago
  Well they weren't that obvious to 155 projects it seems...
  - deleted 11y ago
    [deleted]
  - quakenul 11y ago
    I would assume it was (to most of them) but knowing and doing are two very different things (i.e. everyone knows smoking is bad for you and yet people still do it / start doing it). So to me the far more interesting and helpful question to answer is: Are there most efficient ways to keep people from making the same mistake over and over, other than repeating what everyone already knows, over and over? Identify the forces that drive people away from best practices most often and give us ideas and tools to tackle them.
    - tslug 11y ago
      The best approach is to apprentice with someone who knows not to make them. If you can't do that, check in with someone regularly who: 1. Won't sugar coat feedback, 2. Has shipped games, 3. Has a healthy sense of humility, and 3. Is genuinely concerned about your success. I really like sitting on advisory boards in this role, but my secret to being good at it is that I regularly ask for feedback from others who are smarter than me at the things I don't get (which is most things). In the long run, it's like most things. It takes practice. To the criticism that the game industry is behind the rest of the software industry in software engineering practices, I do believe the rest of the software industry can eat something phallic. :) I'd like to see them pull off what we have to with about 10X the competition, juggling massive game assets, a fraction of the budget, finance, and exit options, needing to meet a tough frame rate budget (even more exacting now with VR), all issues GPU-ish (when was the last time you wrote a shader, Mr. Website Developer?), and having people with a huge range of technical skills requiring direct access to the source/asset repo.
msane 11y ago
Apparently they didn't achieve any insights suitable for mentioning in the summary.
Negative1 11y ago
The old post-mortems in Game Developer magazine were written almost in template form. Went right: 1) Great team/culture 2) Everyone put forth the effort (scheduled crunch) 3) Some amazing techwork or artwork or designwork etc... Went wrong: 1) Didn't anticipate x/y/z 2) Burnout (remember that crunch)? 3) Bad/lazy scheduling and/or capacity planning Kudos to the people who did this at MS but you're looking at tainted data. Most game developers know what really went wrong and do not publicly broadcast it in fear of losing future contracts or publisher trust. In the end the postmortem becomes a soundboard for 'shoutouts' and cheer-leading instead of that raw, unbiased feedback. I've been to all hands post-mortems that were like that and it can get really really ugly.
- pandaman 11y ago
  Exactly this. Only one game I've shipped had a Gamasutra postmortem but that one was so generic that it could have been applied to any other game, ignoring the specific issues that project had.
- ap22213 11y ago
  What are the real reasons for successes and failures? You left me hanging...
  - jonny_eh 11y ago
    People making mistakes, of one type or another. In these post mortems though, they rarely attribute mistakes to any individual, or even team.
- x0x0 11y ago
  I've been to an all-hands post-mortem where the director of engineering had to say "anyone with an opinion besides [name of director of engineering] sucks"? Said director of engineering literally watched us blow milestone after milestone yet somehow the knowledge we were going to miss our commits (and not by a little) never propagated upwards to the ceo. Either that or the ceo didn't want to listen; it could be any combination of those two. Either way, the engineers were pissed. Incredibly poor milestone management and a failure to triage didn't end up in our fucking learnings document somehow.
vvanders 11y ago
I found the Games Outcome Project to be much more rigorous(and anonymous) on the what practices help and hurt gamedev: http://gamasutra.com/blogs/PaulTozour/20141216/232023/The_Game_Outcomes_Project_Part_1_The_Best_and_the_Rest.php http://gamasutra.com/blogs/PaulTozour/20141216/232023/The_Ga...
- pandaman 11y ago
  If you compare the conclusions, this topic's research, while obvious, is closer to reality that the one, which concluded that crunch literally "makes games worse" despite the fact that every single game, which made a significant critical/financial impact had a period of crunch.
  - hnbro 11y ago
    > despite the fact that every single game, which made a significant critical/financial impact had a period of crunch this argument doesn't necessarily disprove the notion. it's plausible to suggest that whatever impact they had was mitigated to some degree.
    - pandaman 11y ago
      It sure does not. My concern is that in this research they correlate the amount of crunch with quality of a game and show that "better" games have statistically less crunch. Since I know the games, considered successful in the industry, all have a lot of crunch, it sounds like they use their own criteria for quality, which has nothing to do with the common meaning.
chipsy 11y ago
In the past, Gamasutra has also published data indicating correlations between "biggest problems/biggest successes" and marketplace success. The takeaways I remember about that: A good producer matters, a team of a least three people does better than one or two, and experience shipping stuff matters.

95 points by winterismute 11y ago

jacques_chester 11y ago

PDF is on the right, for anyone else wondering what to do about an abstract that doesn't contain any spoilers.

brudgers 11y ago
Link: http://research.microsoft.com/pubs/262301/washburn-icse-2016.pdf http://research.microsoft.com/pubs/262301/washburn-icse-2016...

teddyh 11y ago

> 9. CONCLUSIONS > > We find that we were able to identify both best practices and pitfalls in game development using the information present in the postmortems. Such information on the development of all kinds of software would be highly useful too. Therefore we urge the research community to provide a forum where postmortems on general software development can be presented, and practitioners to report their retrospective thoughts in a postmortem. > > Finally, based on our analysis of the data we collected, we make a few recommendations to game developers. First, be sure to practice good risk management techniques. This will help avoid some of the adverse effects of obstacles that you may encounter during development. Second, prescribe to an iterative development process, and utilize prototypes as a method of proving features and concepts before committing them to your design. Third, don't be overly ambitious in your design. Be reasonable, and take into account your schedule and budget before adding something to your design. Building off of that, don't be overly optimistic with your scheduling. If you make an estimate that initially feels optimistic to you, don't give that estimate to your stakeholders. Revisit and reassess your design to form a better estimation.

padobson 11y ago
There's some world-shaking insight. The best piece of advice I can give is to never, ever give an estimate to stakeholders until you've worked on the estimate first. And don't finish an estimate until you've actually built some small proofs of concept. Your estimates will still be wrong, but they'll be much more accurate than some off-the-cuff number that your stakeholders will be building their various plans around. It all sounds straightforward, but the hardest discipline for me as a developer is to keep from saying things in stakeholder meetings that make my stakeholders happy in the moment, but ultimately needed more thought and effort. It's something I struggle with in every such meeting, to varying degrees of success.
- hluska 11y ago
  I think this is a symptom of a fear of "I don't know." If I could change one thing in dev cultures, I would instill a healthy respect for the sentence: "I don't know, but here's $how_I'll_find_out and here's $when_I'll_have_an_answer."
  - ChuckMcM 11y ago
    Exactly right. Too often I've seen people commit to a schedule where the milestones were of unknown difficulty. When cutting through a new jungle, you have no idea what is between you and your goal, so really all you can say is that the best you can hope for is X if this jungle happens to be like previous ones. This is especially true in startup situations where you are learning technologies that may themselves not be complete, with people who are learning their own new things, to achieve a result which is only hypothetically possible. It drove my CFO nuts but rather than commit to a schedule for a big deliverable I would walk backwards from the end point and say, "These are the stops between where we are, and where we are going. We measure our progress by getting to each stop, but like a subway map there isn't a known amount of time between stops, only what the stops are." And he would come back with "well we only have money to get to this date, will it be done by then?" and then we would talk about the uncertainty between each of the milestones. At some point you can reach a common understanding of what the unknowns are and how discovering their value will inform on the difficulty of the next step. That said, I've met managers who just say "Oh we will be done by %x date." and then basically worked the problem the same way I have. If you're agile you can break down the intermediate stops as sprints but estimating the backlog is still the killer step.
bottled_poe 11y ago
Turns out software estimates are hard. Who knew?

ekianjo 11y ago

The recommendations at the end of the pdf seem to be written by captain Obvious. They are so generic they can apply to about ANY project even non gamedev related.

Kristine1975 11y ago
>The recommendations at the end of the pdf seem to be written by captain Obvious. Doesn't that simply mean that this research confirms other people's experience/gut feeling?
- ekianjo 11y ago
  It simply means the research is not insightful.
yread 11y ago
Well they weren't that obvious to 155 projects it seems...
- deleted 11y ago
  [deleted]
- quakenul 11y ago
  I would assume it was (to most of them) but knowing and doing are two very different things (i.e. everyone knows smoking is bad for you and yet people still do it / start doing it). So to me the far more interesting and helpful question to answer is: Are there most efficient ways to keep people from making the same mistake over and over, other than repeating what everyone already knows, over and over? Identify the forces that drive people away from best practices most often and give us ideas and tools to tackle them.
  - tslug 11y ago
    The best approach is to apprentice with someone who knows not to make them. If you can't do that, check in with someone regularly who: 1. Won't sugar coat feedback, 2. Has shipped games, 3. Has a healthy sense of humility, and 3. Is genuinely concerned about your success. I really like sitting on advisory boards in this role, but my secret to being good at it is that I regularly ask for feedback from others who are smarter than me at the things I don't get (which is most things). In the long run, it's like most things. It takes practice. To the criticism that the game industry is behind the rest of the software industry in software engineering practices, I do believe the rest of the software industry can eat something phallic. :) I'd like to see them pull off what we have to with about 10X the competition, juggling massive game assets, a fraction of the budget, finance, and exit options, needing to meet a tough frame rate budget (even more exacting now with VR), all issues GPU-ish (when was the last time you wrote a shader, Mr. Website Developer?), and having people with a huge range of technical skills requiring direct access to the source/asset repo.

msane 11y ago

Apparently they didn't achieve any insights suitable for mentioning in the summary.

Negative1 11y ago

The old post-mortems in Game Developer magazine were written almost in template form. Went right: 1) Great team/culture 2) Everyone put forth the effort (scheduled crunch) 3) Some amazing techwork or artwork or designwork etc... Went wrong: 1) Didn't anticipate x/y/z 2) Burnout (remember that crunch)? 3) Bad/lazy scheduling and/or capacity planning Kudos to the people who did this at MS but you're looking at tainted data. Most game developers know what really went wrong and do not publicly broadcast it in fear of losing future contracts or publisher trust. In the end the postmortem becomes a soundboard for 'shoutouts' and cheer-leading instead of that raw, unbiased feedback. I've been to all hands post-mortems that were like that and it can get really really ugly.

pandaman 11y ago
Exactly this. Only one game I've shipped had a Gamasutra postmortem but that one was so generic that it could have been applied to any other game, ignoring the specific issues that project had.
ap22213 11y ago
What are the real reasons for successes and failures? You left me hanging...
- jonny_eh 11y ago
  People making mistakes, of one type or another. In these post mortems though, they rarely attribute mistakes to any individual, or even team.
x0x0 11y ago
I've been to an all-hands post-mortem where the director of engineering had to say "anyone with an opinion besides [name of director of engineering] sucks"? Said director of engineering literally watched us blow milestone after milestone yet somehow the knowledge we were going to miss our commits (and not by a little) never propagated upwards to the ceo. Either that or the ceo didn't want to listen; it could be any combination of those two. Either way, the engineers were pissed. Incredibly poor milestone management and a failure to triage didn't end up in our fucking learnings document somehow.

vvanders 11y ago

I found the Games Outcome Project to be much more rigorous(and anonymous) on the what practices help and hurt gamedev: http://gamasutra.com/blogs/PaulTozour/20141216/232023/The_Game_Outcomes_Project_Part_1_The_Best_and_the_Rest.php http://gamasutra.com/blogs/PaulTozour/20141216/232023/The_Ga...

pandaman 11y ago
If you compare the conclusions, this topic's research, while obvious, is closer to reality that the one, which concluded that crunch literally "makes games worse" despite the fact that every single game, which made a significant critical/financial impact had a period of crunch.
- hnbro 11y ago
  > despite the fact that every single game, which made a significant critical/financial impact had a period of crunch this argument doesn't necessarily disprove the notion. it's plausible to suggest that whatever impact they had was mitigated to some degree.
  - pandaman 11y ago
    It sure does not. My concern is that in this research they correlate the amount of crunch with quality of a game and show that "better" games have statistically less crunch. Since I know the games, considered successful in the industry, all have a lot of crunch, it sounds like they use their own criteria for quality, which has nothing to do with the common meaning.

chipsy 11y ago

In the past, Gamasutra has also published data indicating correlations between "biggest problems/biggest successes" and marketplace success. The takeaways I remember about that: A good producer matters, a team of a least three people does better than one or two, and experience shipping stuff matters.