11 ms·

Confessions of an Unreal Engine 4 engineering firefighter

388 points by lfowles 9y ago

lfowles 9y ago
FTA, a take on entry/mid/senior level distinctions I haven't heard before: "If you are the guy asking for help from people below, to your side, and above you, you’re probably an entry-level engineer. If you frequently need support from those above you (excluding C-level and executive support, that my friends are a whole new level of hell), there’s a good chance you are a mid-level engineer. If you find yourself supporting and being supported by those at your side, you are likely a senior or lead level engineer."
- abritinthebay 9y ago
  I've heard a similar thing but phrased as what frustrates you in you work most. It's a bit more cynical. Basically: If you're frustrated by your work... probably junior. If you're frustrated by your colleagues... probably mid-level. If you're frustrated by too much management... probably senior. If you're frustrated by too little management... probably lead.
  - josephg 9y ago
    I think about it in relation to how much responsibility each person in the team is comfortable taking. If you're responsible to solve tickets, you're probably junior. If you're responsible for implementing features, mid. If you're responsible for making sure the whole project will work in time for delivery, senior. And if you're responsible for making sure your team can deliver the project on time, lead.
- dnomad 9y ago
  In fact it's very accurate. This is what distinguishes a true senior/10x engineer from a "ninja" or a "rockstar"; they are force mulitpliers and they make everybody around them 10x more productive. This also suggests what C-levels are supposed to be doing: providing the vision and culture that actually fosters this kind of mutual web of support.
georgeecollins 9y ago
The amazing thing to me is that now people can make games without a bunch of great engineers. Back in the day when we made Battlezone we had at least four really amazing engineers (one was a Masters in CS from MIT, one wrote early 3d workstation code, UCLA, Cal Tech CS degrees). In those days you needed people like that to just 3D to work in Windows. Direct X helped a lot. Unity and Unreal have made things we couldn't have dreamed of possible.
- deleted 9y ago
  [deleted]
- egypturnash 9y ago
  Oh hi co-worker of my old friend Ken :)
  - georgeecollins 9y ago
    OMG: Ken is one of the most amazing people I have ever worked with. You are lucky to count him as a friend.
    - egypturnash 9y ago
      I haven't seen him in person in years, what with me living in Seattle these days and him being in Austin. But yeah, he's a pretty good guy!
      - kennethdmiller3 9y ago
        "I'm not dead yet!" :D
        egypturnash 9y ago
        Oh hi there :)
        9y ago
dang 9y ago
Url changed from https://80.lv/articles/confessions-of-an-unreal-engine-4-engineering-firefighter/ https://80.lv/articles/confessions-of-an-unreal-engine-4-eng..., which points to this.
vvanders 9y ago
> Overtaxing your engineering effort is a surefire way to end up with hard-to-maintain work. Unless you are developing a throwaway prototype, hard-to-maintain work is one of the fastest ways to cause engineering fatigue Sadly this was still all par-for-the-course back when I was in the industry and all my old contacts still affirm that this is the case today across most studios. Easier to burn people, ship something and then re-hire/train a new batch. It's a shame the incentives of development aren't aligned with reasonable employment, the industry would be a lot better for it.
wyldfire 9y ago
> They immediately asked who was giving up this information and wanted to crack down on this perceived insubordination, rather than trying to address the issue. I was then dismissed for not specifying names, and my services were no longer required. Wow! That sounds terrible.
- pjc50 9y ago
  This is horribly routine in all hierarchical organisations across the world; those that manage to escape from it usually become runaway successes.
- mitchell_h 9y ago
  Fixing the blame is easier then fixing the problem.
- coldacid 9y ago
  If that company is still around, it's worth naming and shaming.
lkrubner 9y ago
About this: “responsibilities than they can handle, and development suffers from this person being fatigued, or worse, leaving altogether. Neither the engineer nor team can grow because of this, as new features or changes get increasingly more resource-demanding to implement” This is said about mid-level engineers who burn out, but I think it happens to senior level engineers as well.
banachtarski 9y ago
I don't understand why Unreal Engine 4 is in the title. It feels click-baity since you're expecting to read something about the "dark side" of UE4 or something.
- georgeecollins 9y ago
  I think it is because UE4 (and Unity) make it possible to get 50-80% there on projects without solid engineering. So people think they can get by without having the skills on the team.
  - banachtarski 9y ago
    I think most people with UE4 experience or running a studio know that you need solid programmers working on your project. The author is saying that even with that, project ambitions still exceed team bandwidth/capability.
- egypturnash 9y ago
  First sentence of the article: "I have been working professionally with Unreal Engine for over 9 years." Second paragraph: "If a game company in Los Angeles is having an Unreal Engine 4 problem that no one can solve, I usually end up getting that call. I am writing this article to explain why I get that call, how one would prevent the need for that call, and what I normally do after getting that call." So while a lot of these things are pretty useful strategies for any large project (we had asset naming conventions for parts of the Flash cartoons I was working on around the turn of the century), they are all common problems he's encountered in his career of being That Guy You Call When Your Big Unreal Project Is Broken.
  - banachtarski 9y ago
    Yea I read the article >.> I still read it as him trying to suggest what better practices you can adopt specifically pertaining to UE4 that would "prevent the need for that call"
_bxg1 9y ago
Most of this just sounds like standard software engineering best-practices, though from what I've gathered it seems like standard SE challenges get multiplied five-fold in game development. It takes an engineer braver than I to work at a triple-A game studio.
shoo 9y ago
tangentially, the "blueprint spaghetti" link is pretty great: https://blueprintsfromhell.tumblr.com/ https://blueprintsfromhell.tumblr.com/ at a former job i knew a team of devs who built and maintained a framework to help teams of researchers put together pipelines of simulation (often fluid simulation), animation, rendering, etc. This framework had its own blueprinting functionality. One of the cute features they had was nested blueprints -- e.g. you could define some blueprint, and abstract it behind an interface as its own component, then re-use the component in another blueprint. So i guess in a "blueprints from hell" scenario that lets you have "fractal blueprints from hell". edit: i'm also reminded of Chris Domas' defcon talk "Psychological Warfare in Reverse Engineering", where he sets out to construct binaries that cause despair when someone tries to reverse engineer them -- in particular he asks the question: "could i make a binary that draws a particular picture when someone views the binary's control flow graph in IDA" https://www.youtube.com/watch?v=HlUe0TUHOIc https://www.youtube.com/watch?v=HlUe0TUHOIc
AndyNemmity 9y ago
"Usually an outstanding problem or issue is already known by your staff, and often they have tried to make you aware of this problem, but usually it gets prioritized into nothingness, or filtered into white noise by your chain of command where no action gets done." This is the same as being a consultant. You come on site, you listen to everyone and make notes of it, and then you make a presentation and plan for direction. They pay you way more than their own people, to tell you what your own people are telling you. It's quite crazy.
- pas 9y ago
  As a consultant, they never change.
  - jeremiep 9y ago
    As an employee happy with his current employer, those who can change will and won't hire consultants in the process.
    - pas 9y ago
      Not that kind of consultant. :) And sort of agreed. Though, those who can change and will, should stop being a problem "soon" anyway.
gambler 9y ago
Good article, but I'm not sure who it is targeted at. Companies/teams that do bad things the author recommends against (or don't do good things he recommends) are usually ran by people who either don't care or are pathologically clueless. They're not going to read this. If they do, they are not going to change the right thing. They will instead go to some business conference, listen to overhyped process consultants and force-implement more "agile" techniques that have nothing to do with what actually hurts their productivity. Maybe, strip out most of the game dev stuff and write a small book? That might help. I don't know why, but books work on managers much better than articles. At least I can give or recommend someone a book without making it seem like an insult. "Hey, I've read this book by an engineer who worked on many large game projects. You should check it out."
- Allar 9y ago
  You're right. I wrote this article knowing this, which is what I tried opening with. I'm not qualified to write a book however because this article itself is proof my writing skills need work. Otherwise I'd be able to be more clear.
  - joshu 9y ago
    Are you kidding? It was a great article.
    - daveslash 9y ago
      Agreed - great article. That said, I think most authors of books like the one we're all imagining have support (in the form of friends and colleagues) in addition to an editor. I don't want to pressure you to write a book, but please don't let your own criticism of your writing abilities get in the way of a book.
  - sirspacey 9y ago
    I’d help in whatever way you’d like. My dream is to make a company engineers love to work in. The chance to learn from whatever you share would be amazing.
johnmcarthur 9y ago
"If you're reading this as an engineer (but this may apply to many fields), or as someone looking to grow an engineer, the most important thing you can do is identify current issues and weaknesses." I learned a very different things, but thank you nonetheless: there are companies that care about these things! I just got a full-time job that is basically like this, a hell on Earth (I've been freelancing before). So the lesson I learned is, I should be looking for a new job :) My coworker (we are 2 in the team) is already jumping ship even when he joined later than me.
FollowSteph3 9y ago
One of the most common things I’ve seen over and over is under powered hardware. It’s so cheap in comparison but since it’s easy for an accountant to total compared to lost time it’s very frequently done. It’s not a solve all but in so many cases it would make a significant difference ignoring all else.
ajeet_dhaliwal 9y ago
This was a great article and I recognize most of the issues described but I feel it doesn’t criticise leadership enough (even though it does somewhat). This is the biggest failure in the industry. A lot of the best people are too busy putting in the hard work to be put in leadership roles due to mismanagement from the current leadership. Why is crunch always a thing in games dev? It’s poor planning that endures for years and years.
martin_ky 9y ago
I'm sure this article resonated with many engineers, not just from the game industry. What resonated with me the most, was the part about hiring a consultant, only to tell management what their employees have been telling them all along (for a ridiculous hourly rate of course). The simplified executive line of reasoning, why to rather trust an external entity over their own people, goes usually along these lines: "We do not take advice from our employees. We hired them for grunt work and we only expect them to follow orders. What could they possibly know that could benefit our high-level decision making. Afterall we pay them so little". I consider the case, which the author describes, where the consultant actually talks to engineers, a happy day scenario. I was witness to a fast growing startup hiring a well known global consulting firm to help them set up company structure and processes. The startup was unable to do this on its own because of lack of leadership and unwillingness of the owners to empower their own people. The consulting firm sent over a couple of their junior consultants who consulted their 3-ring binder of best practices for the nearest matching industry category and gave their recommendations, which were followed quite strictly by a newly hired junior management. All without talking to a single engineer. More often than not, employees are considered a necessary means to an end and treated as such, with the least amount of reward and care for which they are willing to work. Their sense of professional honor, responsibility and guilt is often exploited in an attempt to achieve higher output. Does the company's products or services suffer because of this? Of course, but nobody at the top cares as long as there is a metric that tells a good story. And when not, the lowest ranks are the first to be pruned and restructured.
- direfungasaur 9y ago
  > The simplified executive line of reasoning, why to rather trust an external entity over their own people, goes usually along these lines: "We do not take advice from our employees. We hired them for grunt work and we only expect them to follow orders. What could they possibly know that could benefit our high-level decision making. Afterall we pay them so little". It's not always that leadership ignores line-engineers just because they consider them stupid. Even obvious recommendations get tangled into the web of politics and personal favors. Sometimes the 'obvious solution' is also a play to get more power or influence for someone you're opposed to. That factors into how the advice is taken. Part of the fiction management sells itself is consultants can cut across those organizational boundaries and find the 'right solution'. Sometimes having an 'impartial' voice echoing what others have already said is enough to convince you it's not just someone trying to poach your turf, it's actually good advice. Further, our tendencies towards tribalism influence this process. We tend not to think of other employees as part of the same group - they are devs, or ops, or product. That builds an 'us vs. them' mentality that's hard to break. Bringing in an outsider tends to unify the group into us (the company/team) vs. the outsider (the consultant). That psychology helps us to justify bitter medicine like our team getting more work (which will help the company). I tend to have a negative opinion of consultants, but someone pointed this aspect of group psychology to me in the past and it resonated.
golergka 9y ago
Wow, this hits way too close to home. On the last few projects I've been hired as a lead Unity Dev, my main job was to untangle the mess left by previous maintainers and finally put some good engineering practices in place. Project I'm working on right on had it's multiplayer "just added" on top of single player gameplay mechanics, and now my team finds itself in the middle of the complete architectural rewrite instead of making any new features.
EliRivers 9y ago
When I got the call to fix this dilemma literally a couple of weeks before release, the simple act of cleaning up the source control server, and purging these unneeded cache files from source control, fixed this audio issue that was plaguing them for months and had costed the company tons of money and time. This sort of thing drives me insane (or at least used to; my current colleagues, from junior to senior, are all actually pretty good). This is something that everyone knows, just by looking at it, is a bad thing. Everyone who saw this must have known it wasn't a great idea, even if they didn't know why. Even without being able to elucidate why, everyone (should) get the feeling that having these things in the repository is asking for trouble; I'd expect a junior to have at least a bad feeling about it, and a senior to be able to specify what kind of problems it could cause. I don't know about the exact circumstances, of course, but it's even surprising that nobody found this problem while they were investigating. I've certainly gone as far as "are we definitely using the exact same binary files on the different systems?" when chasing odd problems that only manifest on some installations and not others, so what went so wrong that nobody got that far in their debugging? So unless the company in question had by chance a room full of people with this particular massive blind spot, people saw this happening and they knew it was a bad idea and they didn't do anything about it. Culture failure. I don't know why. Maybe they were afraid to mention bad things because pointing out possible problems is seen as not being a team player. Maybe they were just so sick of it all they didn't care anymore. Maybe there were a couple of forcefully strong people in charge of the repository who saw any question as a personal attack and people were just sick of dealing with the egotistical prima donnas. Maybe the process for doing anything about it was laughable painful and went through someone who would shoot it down on the grounds that we're so busy firefighting things that are broken and you want to waste time tinkering with something that looks fine to me? If Allar is reading, could you say a few words about this? How many times do you get called in to fix something that could have been averted with a decent programming culture, even if the specific technical chops weren't there? We've all seen teams with good culture deliver software that they really didn't know how to build when they started through good culture, and we've all seen teams of people who all know what they're doing deliver a flaming wreck because the culture was a disaster. How often are you called in to deal with something that the existing team really couldn't have foreseen without some particular piece of specific knowledge or experience, and how often is it something that they could have dealt with months ago if they had just been allowed and guided to work sensibly and professionally?
laythea 9y ago
Good article and I agree with most points, however I am not sure I like the articles tone. Maybe I am a precious butterfly but it was a bit "all-knowing", with too much emphasis on how you save the day. I am sure your phone will start ringing after publications too :) Thanks anyway though, I read it :)
hypertexthero 9y ago
Great article that reminded me of John Cleese’s wonderful talk about creativity. > So, here's how to stamp out creativity in the rest of the organization and get a bit of respect going. > One: Allow subordinates no humour, it threatens your self-importance and especially your omniscience. Treat all humour as frivolous or subversive. > Because subversive is, of course, what humour will be in your setup, as it's the only way that people can express their opposition, since (if they express it openly) you're down on them like a ton of bricks. > So let's get this clear: blame humour for the resistance that your way of working creates. Then you don't have to blame your way of working. This is important. And I mean that solemnly. Your dignity is no laughing matter. > Second: keeping ourselves feeling irreplaceable involves cutting everybody else down to size, so don't miss an opportunity to undermine your employees' confidence. > A perfect opportunity comes when you're reviewing work that they've done. Use your authority to zero in immediately on all the things you can find wrong. Never never balance the negatives with positives, only criticize, just as your school teachers did. > Always remember: praise makes people uppity. > Third: Demand that people should always be actively doing things. If you catch anyone pondering, accuse them of laziness and/or indecision. This is to starve employees of thinking time because that leads to creativity and insurrection. So demand urgency at all times, use lots of fighting talk and war analogies, and establish a permanent atmosphere of stress, of breathless anxiety, and crisis. > In a phrase: keep that mode closed. > In this way we no-nonsense types can be sure that the tiny, tiny, microscopic quantity of creativity in our organization will all be ours! > But! Let your vigilance slip for one moment, and you could find yourself surrounded by happy, enthusiastic, and creative people whom you might never be able to completely control ever again! > So be careful. https://www.youtube.com/watch?v=Pb5oIIPO62g https://www.youtube.com/watch?v=Pb5oIIPO62g https://www.brainpickings.org/2012/04/12/john-cleese-on-creativity-1991/ https://www.brainpickings.org/2012/04/12/john-cleese-on-crea...
Kapura 9y ago
As somebody who writes code for games for a living, I found this article fascinating. Something that was coming up in my mind again and again as I was reading through it is the question of who has the power to affect the necessary changes. That seems to be the crux of a lot of the issues; is the studio leadership wrongminded, or have they put the wrong people at the gate? The parable of the producer ignoring engineering specialities when assigning tickets seemed especially gregarious. Why was that person unable to learn, even after the engineers raised the issue? Why were they still there? Were they friendly with the higher-ups, or what? Why was there no will to improve the process until external help had to be brought on? It also reminds me of stories I had heard from inside Microsoft in the pre-Nadella era. Tales of power brokers and entrenched lifers who could make things easy or impossible on a whim. What is the nature of the organisations that allow people like these to take root? It's funny how small scale engineering problems are rooted in code and large scale engineering problems are rooted in people.
- sounds 9y ago
  I think when you wrote "gregarious," you meant "egregious."
  - Kapura 9y ago
    lol you're definitely right. It was one of those words with Es and Rs and a couple of Gs.
    - sounds 8y ago
      :) No worries. But don't mistake it for "erogenous" or "gangrenous"
- Allar 9y ago
  I think I'mma have to use this quote somewhere, if permitted. 'It's funny how small scale engineering problems are rooted in code and large scale engineering problems are rooted in people.'
  - 9y ago
nudpiedo 9y ago
Since I've lived, almost to the letter, a few times many of the scenarios detailed here, I cannot upvote more this post. My question now is: is there any company mid-big size not guilty of those sins? Does it even exist or it is an utopia wished but not achievable? (just asking)

388 points by lfowles 9y ago

FTA, a take on entry/mid/senior level distinctions I haven't heard before: "If you are the guy asking for help from people below, to your side, and above you, you’re probably an entry-level engineer. If you frequently need support from those above you (excluding C-level and executive support, that my friends are a whole new level of hell), there’s a good chance you are a mid-level engineer. If you find yourself supporting and being supported by those at your side, you are likely a senior or lead level engineer."

abritinthebay 9y ago
I've heard a similar thing but phrased as what frustrates you in you work most. It's a bit more cynical. Basically: If you're frustrated by your work... probably junior. If you're frustrated by your colleagues... probably mid-level. If you're frustrated by too much management... probably senior. If you're frustrated by too little management... probably lead.
- josephg 9y ago
  I think about it in relation to how much responsibility each person in the team is comfortable taking. If you're responsible to solve tickets, you're probably junior. If you're responsible for implementing features, mid. If you're responsible for making sure the whole project will work in time for delivery, senior. And if you're responsible for making sure your team can deliver the project on time, lead.
dnomad 9y ago
In fact it's very accurate. This is what distinguishes a true senior/10x engineer from a "ninja" or a "rockstar"; they are force mulitpliers and they make everybody around them 10x more productive. This also suggests what C-levels are supposed to be doing: providing the vision and culture that actually fosters this kind of mutual web of support.

georgeecollins 9y ago

The amazing thing to me is that now people can make games without a bunch of great engineers. Back in the day when we made Battlezone we had at least four really amazing engineers (one was a Masters in CS from MIT, one wrote early 3d workstation code, UCLA, Cal Tech CS degrees). In those days you needed people like that to just 3D to work in Windows. Direct X helped a lot. Unity and Unreal have made things we couldn't have dreamed of possible.

deleted 9y ago
[deleted]
egypturnash 9y ago
Oh hi co-worker of my old friend Ken :)
- georgeecollins 9y ago
  OMG: Ken is one of the most amazing people I have ever worked with. You are lucky to count him as a friend.
  - egypturnash 9y ago
    I haven't seen him in person in years, what with me living in Seattle these days and him being in Austin. But yeah, he's a pretty good guy!
    - kennethdmiller3 9y ago
      "I'm not dead yet!" :D
      - egypturnash 9y ago
        Oh hi there :)
        9y ago

dang 9y ago

Url changed from https://80.lv/articles/confessions-of-an-unreal-engine-4-engineering-firefighter/ https://80.lv/articles/confessions-of-an-unreal-engine-4-eng..., which points to this.

vvanders 9y ago

> Overtaxing your engineering effort is a surefire way to end up with hard-to-maintain work. Unless you are developing a throwaway prototype, hard-to-maintain work is one of the fastest ways to cause engineering fatigue Sadly this was still all par-for-the-course back when I was in the industry and all my old contacts still affirm that this is the case today across most studios. Easier to burn people, ship something and then re-hire/train a new batch. It's a shame the incentives of development aren't aligned with reasonable employment, the industry would be a lot better for it.

wyldfire 9y ago

> They immediately asked who was giving up this information and wanted to crack down on this perceived insubordination, rather than trying to address the issue. I was then dismissed for not specifying names, and my services were no longer required. Wow! That sounds terrible.

pjc50 9y ago
This is horribly routine in all hierarchical organisations across the world; those that manage to escape from it usually become runaway successes.
mitchell_h 9y ago
Fixing the blame is easier then fixing the problem.
coldacid 9y ago
If that company is still around, it's worth naming and shaming.

lkrubner 9y ago

About this: “responsibilities than they can handle, and development suffers from this person being fatigued, or worse, leaving altogether. Neither the engineer nor team can grow because of this, as new features or changes get increasingly more resource-demanding to implement” This is said about mid-level engineers who burn out, but I think it happens to senior level engineers as well.

banachtarski 9y ago

I don't understand why Unreal Engine 4 is in the title. It feels click-baity since you're expecting to read something about the "dark side" of UE4 or something.

georgeecollins 9y ago
I think it is because UE4 (and Unity) make it possible to get 50-80% there on projects without solid engineering. So people think they can get by without having the skills on the team.
- banachtarski 9y ago
  I think most people with UE4 experience or running a studio know that you need solid programmers working on your project. The author is saying that even with that, project ambitions still exceed team bandwidth/capability.
egypturnash 9y ago
First sentence of the article: "I have been working professionally with Unreal Engine for over 9 years." Second paragraph: "If a game company in Los Angeles is having an Unreal Engine 4 problem that no one can solve, I usually end up getting that call. I am writing this article to explain why I get that call, how one would prevent the need for that call, and what I normally do after getting that call." So while a lot of these things are pretty useful strategies for any large project (we had asset naming conventions for parts of the Flash cartoons I was working on around the turn of the century), they are all common problems he's encountered in his career of being That Guy You Call When Your Big Unreal Project Is Broken.
- banachtarski 9y ago
  Yea I read the article >.> I still read it as him trying to suggest what better practices you can adopt specifically pertaining to UE4 that would "prevent the need for that call"

_bxg1 9y ago

Most of this just sounds like standard software engineering best-practices, though from what I've gathered it seems like standard SE challenges get multiplied five-fold in game development. It takes an engineer braver than I to work at a triple-A game studio.

shoo 9y ago

tangentially, the "blueprint spaghetti" link is pretty great: https://blueprintsfromhell.tumblr.com/ https://blueprintsfromhell.tumblr.com/ at a former job i knew a team of devs who built and maintained a framework to help teams of researchers put together pipelines of simulation (often fluid simulation), animation, rendering, etc. This framework had its own blueprinting functionality. One of the cute features they had was nested blueprints -- e.g. you could define some blueprint, and abstract it behind an interface as its own component, then re-use the component in another blueprint. So i guess in a "blueprints from hell" scenario that lets you have "fractal blueprints from hell". edit: i'm also reminded of Chris Domas' defcon talk "Psychological Warfare in Reverse Engineering", where he sets out to construct binaries that cause despair when someone tries to reverse engineer them -- in particular he asks the question: "could i make a binary that draws a particular picture when someone views the binary's control flow graph in IDA" https://www.youtube.com/watch?v=HlUe0TUHOIc https://www.youtube.com/watch?v=HlUe0TUHOIc

AndyNemmity 9y ago

"Usually an outstanding problem or issue is already known by your staff, and often they have tried to make you aware of this problem, but usually it gets prioritized into nothingness, or filtered into white noise by your chain of command where no action gets done." This is the same as being a consultant. You come on site, you listen to everyone and make notes of it, and then you make a presentation and plan for direction. They pay you way more than their own people, to tell you what your own people are telling you. It's quite crazy.

pas 9y ago
As a consultant, they never change.
- jeremiep 9y ago
  As an employee happy with his current employer, those who can change will and won't hire consultants in the process.
  - pas 9y ago
    Not that kind of consultant. :) And sort of agreed. Though, those who can change and will, should stop being a problem "soon" anyway.

gambler 9y ago

Good article, but I'm not sure who it is targeted at. Companies/teams that do bad things the author recommends against (or don't do good things he recommends) are usually ran by people who either don't care or are pathologically clueless. They're not going to read this. If they do, they are not going to change the right thing. They will instead go to some business conference, listen to overhyped process consultants and force-implement more "agile" techniques that have nothing to do with what actually hurts their productivity. Maybe, strip out most of the game dev stuff and write a small book? That might help. I don't know why, but books work on managers much better than articles. At least I can give or recommend someone a book without making it seem like an insult. "Hey, I've read this book by an engineer who worked on many large game projects. You should check it out."

Allar 9y ago
You're right. I wrote this article knowing this, which is what I tried opening with. I'm not qualified to write a book however because this article itself is proof my writing skills need work. Otherwise I'd be able to be more clear.
- joshu 9y ago
  Are you kidding? It was a great article.
  - daveslash 9y ago
    Agreed - great article. That said, I think most authors of books like the one we're all imagining have support (in the form of friends and colleagues) in addition to an editor. I don't want to pressure you to write a book, but please don't let your own criticism of your writing abilities get in the way of a book.
- sirspacey 9y ago
  I’d help in whatever way you’d like. My dream is to make a company engineers love to work in. The chance to learn from whatever you share would be amazing.

johnmcarthur 9y ago

"If you're reading this as an engineer (but this may apply to many fields), or as someone looking to grow an engineer, the most important thing you can do is identify current issues and weaknesses." I learned a very different things, but thank you nonetheless: there are companies that care about these things! I just got a full-time job that is basically like this, a hell on Earth (I've been freelancing before). So the lesson I learned is, I should be looking for a new job :) My coworker (we are 2 in the team) is already jumping ship even when he joined later than me.

FollowSteph3 9y ago

One of the most common things I’ve seen over and over is under powered hardware. It’s so cheap in comparison but since it’s easy for an accountant to total compared to lost time it’s very frequently done. It’s not a solve all but in so many cases it would make a significant difference ignoring all else.

ajeet_dhaliwal 9y ago

This was a great article and I recognize most of the issues described but I feel it doesn’t criticise leadership enough (even though it does somewhat). This is the biggest failure in the industry. A lot of the best people are too busy putting in the hard work to be put in leadership roles due to mismanagement from the current leadership. Why is crunch always a thing in games dev? It’s poor planning that endures for years and years.

martin_ky 9y ago

I'm sure this article resonated with many engineers, not just from the game industry. What resonated with me the most, was the part about hiring a consultant, only to tell management what their employees have been telling them all along (for a ridiculous hourly rate of course). The simplified executive line of reasoning, why to rather trust an external entity over their own people, goes usually along these lines: "We do not take advice from our employees. We hired them for grunt work and we only expect them to follow orders. What could they possibly know that could benefit our high-level decision making. Afterall we pay them so little". I consider the case, which the author describes, where the consultant actually talks to engineers, a happy day scenario. I was witness to a fast growing startup hiring a well known global consulting firm to help them set up company structure and processes. The startup was unable to do this on its own because of lack of leadership and unwillingness of the owners to empower their own people. The consulting firm sent over a couple of their junior consultants who consulted their 3-ring binder of best practices for the nearest matching industry category and gave their recommendations, which were followed quite strictly by a newly hired junior management. All without talking to a single engineer. More often than not, employees are considered a necessary means to an end and treated as such, with the least amount of reward and care for which they are willing to work. Their sense of professional honor, responsibility and guilt is often exploited in an attempt to achieve higher output. Does the company's products or services suffer because of this? Of course, but nobody at the top cares as long as there is a metric that tells a good story. And when not, the lowest ranks are the first to be pruned and restructured.

direfungasaur 9y ago
> The simplified executive line of reasoning, why to rather trust an external entity over their own people, goes usually along these lines: "We do not take advice from our employees. We hired them for grunt work and we only expect them to follow orders. What could they possibly know that could benefit our high-level decision making. Afterall we pay them so little". It's not always that leadership ignores line-engineers just because they consider them stupid. Even obvious recommendations get tangled into the web of politics and personal favors. Sometimes the 'obvious solution' is also a play to get more power or influence for someone you're opposed to. That factors into how the advice is taken. Part of the fiction management sells itself is consultants can cut across those organizational boundaries and find the 'right solution'. Sometimes having an 'impartial' voice echoing what others have already said is enough to convince you it's not just someone trying to poach your turf, it's actually good advice. Further, our tendencies towards tribalism influence this process. We tend not to think of other employees as part of the same group - they are devs, or ops, or product. That builds an 'us vs. them' mentality that's hard to break. Bringing in an outsider tends to unify the group into us (the company/team) vs. the outsider (the consultant). That psychology helps us to justify bitter medicine like our team getting more work (which will help the company). I tend to have a negative opinion of consultants, but someone pointed this aspect of group psychology to me in the past and it resonated.

golergka 9y ago

Wow, this hits way too close to home. On the last few projects I've been hired as a lead Unity Dev, my main job was to untangle the mess left by previous maintainers and finally put some good engineering practices in place. Project I'm working on right on had it's multiplayer "just added" on top of single player gameplay mechanics, and now my team finds itself in the middle of the complete architectural rewrite instead of making any new features.

EliRivers 9y ago

When I got the call to fix this dilemma literally a couple of weeks before release, the simple act of cleaning up the source control server, and purging these unneeded cache files from source control, fixed this audio issue that was plaguing them for months and had costed the company tons of money and time. This sort of thing drives me insane (or at least used to; my current colleagues, from junior to senior, are all actually pretty good). This is something that everyone knows, just by looking at it, is a bad thing. Everyone who saw this must have known it wasn't a great idea, even if they didn't know why. Even without being able to elucidate why, everyone (should) get the feeling that having these things in the repository is asking for trouble; I'd expect a junior to have at least a bad feeling about it, and a senior to be able to specify what kind of problems it could cause. I don't know about the exact circumstances, of course, but it's even surprising that nobody found this problem while they were investigating. I've certainly gone as far as "are we definitely using the exact same binary files on the different systems?" when chasing odd problems that only manifest on some installations and not others, so what went so wrong that nobody got that far in their debugging? So unless the company in question had by chance a room full of people with this particular massive blind spot, people saw this happening and they knew it was a bad idea and they didn't do anything about it. Culture failure. I don't know why. Maybe they were afraid to mention bad things because pointing out possible problems is seen as not being a team player. Maybe they were just so sick of it all they didn't care anymore. Maybe there were a couple of forcefully strong people in charge of the repository who saw any question as a personal attack and people were just sick of dealing with the egotistical prima donnas. Maybe the process for doing anything about it was laughable painful and went through someone who would shoot it down on the grounds that we're so busy firefighting things that are broken and you want to waste time tinkering with something that looks fine to me? If Allar is reading, could you say a few words about this? How many times do you get called in to fix something that could have been averted with a decent programming culture, even if the specific technical chops weren't there? We've all seen teams with good culture deliver software that they really didn't know how to build when they started through good culture, and we've all seen teams of people who all know what they're doing deliver a flaming wreck because the culture was a disaster. How often are you called in to deal with something that the existing team really couldn't have foreseen without some particular piece of specific knowledge or experience, and how often is it something that they could have dealt with months ago if they had just been allowed and guided to work sensibly and professionally?

laythea 9y ago

Good article and I agree with most points, however I am not sure I like the articles tone. Maybe I am a precious butterfly but it was a bit "all-knowing", with too much emphasis on how you save the day. I am sure your phone will start ringing after publications too :) Thanks anyway though, I read it :)

hypertexthero 9y ago

Great article that reminded me of John Cleese’s wonderful talk about creativity. > So, here's how to stamp out creativity in the rest of the organization and get a bit of respect going. > One: Allow subordinates no humour, it threatens your self-importance and especially your omniscience. Treat all humour as frivolous or subversive. > Because subversive is, of course, what humour will be in your setup, as it's the only way that people can express their opposition, since (if they express it openly) you're down on them like a ton of bricks. > So let's get this clear: blame humour for the resistance that your way of working creates. Then you don't have to blame your way of working. This is important. And I mean that solemnly. Your dignity is no laughing matter. > Second: keeping ourselves feeling irreplaceable involves cutting everybody else down to size, so don't miss an opportunity to undermine your employees' confidence. > A perfect opportunity comes when you're reviewing work that they've done. Use your authority to zero in immediately on all the things you can find wrong. Never never balance the negatives with positives, only criticize, just as your school teachers did. > Always remember: praise makes people uppity. > Third: Demand that people should always be actively doing things. If you catch anyone pondering, accuse them of laziness and/or indecision. This is to starve employees of thinking time because that leads to creativity and insurrection. So demand urgency at all times, use lots of fighting talk and war analogies, and establish a permanent atmosphere of stress, of breathless anxiety, and crisis. > In a phrase: keep that mode closed. > In this way we no-nonsense types can be sure that the tiny, tiny, microscopic quantity of creativity in our organization will all be ours! > But! Let your vigilance slip for one moment, and you could find yourself surrounded by happy, enthusiastic, and creative people whom you might never be able to completely control ever again! > So be careful. https://www.youtube.com/watch?v=Pb5oIIPO62g https://www.youtube.com/watch?v=Pb5oIIPO62g https://www.brainpickings.org/2012/04/12/john-cleese-on-creativity-1991/ https://www.brainpickings.org/2012/04/12/john-cleese-on-crea...

Kapura 9y ago

As somebody who writes code for games for a living, I found this article fascinating. Something that was coming up in my mind again and again as I was reading through it is the question of who has the power to affect the necessary changes. That seems to be the crux of a lot of the issues; is the studio leadership wrongminded, or have they put the wrong people at the gate? The parable of the producer ignoring engineering specialities when assigning tickets seemed especially gregarious. Why was that person unable to learn, even after the engineers raised the issue? Why were they still there? Were they friendly with the higher-ups, or what? Why was there no will to improve the process until external help had to be brought on? It also reminds me of stories I had heard from inside Microsoft in the pre-Nadella era. Tales of power brokers and entrenched lifers who could make things easy or impossible on a whim. What is the nature of the organisations that allow people like these to take root? It's funny how small scale engineering problems are rooted in code and large scale engineering problems are rooted in people.

sounds 9y ago
I think when you wrote "gregarious," you meant "egregious."
- Kapura 9y ago
  lol you're definitely right. It was one of those words with Es and Rs and a couple of Gs.
  - sounds 8y ago
    :) No worries. But don't mistake it for "erogenous" or "gangrenous"
Allar 9y ago
I think I'mma have to use this quote somewhere, if permitted. 'It's funny how small scale engineering problems are rooted in code and large scale engineering problems are rooted in people.'
- 9y ago

nudpiedo 9y ago

Since I've lived, almost to the letter, a few times many of the scenarios detailed here, I cannot upvote more this post. My question now is: is there any company mid-big size not guilty of those sins? Does it even exist or it is an utopia wished but not achievable? (just asking)