13 ms·

There’s an implicit story here which I find quite sad that not a single developer at Rockstar in the prior eight years had the autonomy or bandwidth to discover

There’s an implicit story here which I find quite sad that not a single developer at Rockstar in the prior eight years had the autonomy or bandwidth to discover and fix this themselves.

pinche_gazpacho 4y ago
nor the incentive
- deleted 4y ago
  [deleted]
azemetre 4y ago
I’m starting to slowly realize how terrible it is working in environments where people simply aren’t curious. It’s so hard for me as an individual contributor to decouple this way of thinking. Curiosity is how we find better ways of doing things or finding out hidden flows that can be better. Incurious people are a contagion, but I want to believe you can make people curious. It can’t be innate. Every company talks about wanting to have driven curious employees to solve complex issues or challenges but you just get stomped with a boot to your face if you do anything outside your lane or simply ask why. It’s also worse because I’m starting to realize how lucky I was to be raised as a curious individual from my parents who never went to college but at least were curious themselves about different things (pottery, soldering, gardening). The quote “Curiosity is insubordination in its purest form” suddenly has a different context when working for a company.
- TheAceOfHearts 4y ago
  Don't blame the people, blame the environment. People might start off curious but eventually they keep running into political roadblocks and other nonsense, which slowly beats the interest out of them. If those are the kinds of people that continue surviving then those are the kinds of people which your environment cultivates.
  - azemetre 4y ago
    That’s totally fair, I honestly meant to blame the environment and not people. It just sucks to see the pattern of curiosity get stamped out of people due to environmental changes that aren’t obvious.
    - deleted 4y ago
      [deleted]
  - miahi 4y ago
    Discussion from today: "Do you know if feature X works? A customer is asking about it" "No, it was disabled because of a bug a while back" "Hmm, found the ticket, it's from 4 years ago!" "Yes, basically it's a bug from a new version of a 3rd party software that made it unusable. At that point it was basically decided that we will not try to find a workaround, we just tell the customer that it's a 3rd party issue." The funny thing is, all the people who made that decision left the company/project and now we have the option to change it. Can't wait to see what happens next, if the new leadership changes that or not (my money is on "not"). The other funny thing is, I can think of at least two workarounds that only need a day of work and a couple for testing it.
- alach11 4y ago
  Sadly at Rockstar it's likely that employees are curious, but are under so much pressure for deadlines, and working such long hours, that they don't have time to follow their curiosity.
  - switchbak 4y ago
    When problems are this obvious, it's surprising that someone doesn't have a look, even if they're busy. Maybe I'm asking too much/projecting here, but I just don't understand how people put up with fairly obvious semi-breakage as much as they do. I have this at my current job, I seem to be the only one (not really, but seems like it sometimes) interested in finding out why things are slow/broken and investing the time to fix it. If you're "busy" all the time, but put up with things that slow you down, you're hamstringing yourself by not fixing the brokenness. When only a small number of people take the time to fix things (typically other's issues), that becomes very demotivating. That would be like being on an all-day ski tour, and feeling too rushed to bother getting rid of the slush on your skis. Yes, you feel rushed because you're moving slower that usual. Taking 4 minutes to kick the slush off will make you much faster on the overall trip. (Sharpening the saw analogy, etc). Sometimes "busy" is a state of mind, more than it is a reflection of reality. I know each situation is unique, but I've seen this enough times to understand that it's often a form of dysfunction rather than a reasonable reaction to the work situation. This is like when people used to say "we're too busy to write tests". Well, maybe you need to get better at communicating the trade-offs to the broader business, and be a bit more assertive about non-functional aspects that are still important to address.
    - 0xFACEFEED 4y ago
      > Maybe I'm asking too much/projecting here, but I just don't understand how people put up with fairly obvious semi-breakage as much as they do. > I have this at my current job, I seem to be the only one (not really, but seems like it sometimes) interested in finding out why things are slow/broken and investing the time to fix it. You will quickly shed yourself of this philosophy if you become a business owner. You can test those waters by releasing a non-trivial open source project that gains a non-trivial amount of users. There will always be at least 100 things that are broken in some way but not impacting the core business. You'll have to pick only 1 to work on. And, if you're unlucky, some frustrated customer may write a blog post about issue #65. An issue that you're fully aware of but haven't prioritized. Guess what you'll do as a business owner? You'll prioritize that issue since it's now very visible and maybe even throw $10,000 at the customer who took time to do their research; as is what happened here. > This is like when people used to say "we're too busy to write tests". Well, maybe you need to get better at communicating the trade-offs to the broader business, and be a bit more assertive about non-functional aspects that are still important to address. Or perhaps the need for tests was communicated and the business made the decision to not prioritize it. You may not like that decision. Mind that the decision to "not write tests" is never explicitly stated so simply. No one wants to look bad. Instead that decision is expressed through minimizing its level of priority compared to other things. Still don't like the decision? Well you have three options: 1) Ignore the stated goals of the company and do what you think is right instead. In which case you're acting against the interests of the company that's paying you. 2) Sacrifice your own free time in order to provide your idea of value to the company (that may actually be valuable! but that the company doesn't value enough to prioritize) 3) Do what the company asks with plausible deniability. Also known as kicking the can. Also known as not my problem. Sometimes doing #1 or #2 could greatly benefit you. Perhaps that thing you're doing (eg: writing tests) will somehow measurably improve the business in the future WITH PROOF so that you'll get get recognized. Or perhaps there's management shake-up and the new leadership perfectly aligns with your philosophy, etc, etc. But that's rare. More often than not your attempt at fixing the company's problems behind the scenes will go unnoticed and unrewarded. And if you do get rewarded, often times it's a pittance compared to the amount of time you invested. My personal philosophy is I simply do not work for companies that don't align with what I value. For example, to me, it's exhausting and unrewarding to work at a company that doesn't appreciate a GREAT user experience.
- jdwithit 4y ago
  +1 to the other replies here, I think it's very poor form to throw the engineers under the bus for not being curious. The most likely explanation is they were under insane time and resource constraints, especially in the games industry. They wrote something that worked, then were forced to move on to the other 5000 tasks in their backlog that had to get done before release. When you're already being told to crunch 80+ hour weeks for months or years at a time, you run out of gas. I guess it's possible that the engineers who were talented enough to create this extremely impressive game in the first place somehow threw up their hands and said "guess it can never take less than 7 minutes to load, best we can do". But it seems much more likely that they were well aware, but never allowed by poor management to go back and fix it. The game was released and printing money, why bother. They're on to the next title.
  - ALittleLight 4y ago
    The problem with this way of thinking is that it basically absolves everyone of blame. Bad stuff happens and nobody's at fault. You point to the managers, because the engineers were so busy, but the managers are going to claim they were busy too. Again, from the manager's perspective - the managers aren't even in a position to understand the problem or the reasons for it.
    - delusional 4y ago
      "the managers aren't even in a position to understand the problem or the reasons for it." Which is again the fault of management. It is the responsibility of management to facilitate proper management.
      - blowski 4y ago
        As a shareholder, I’d be rather happy with the management of GTA - it’s phenomenally profitable. That sounds like proper management to me.
    - mrandish 4y ago
      Having been on both sides of this (as a developer and later a senior manager with product/feature responsibility), I'd say it can be both. A really good manager/executive has enough knowledge and experience to ask the right questions and parse the answers even when they are technical. Sadly, many managers aren't that good (at least yet). On the other side, a really good developer has the knowledge and experience to frame the problem in an understandable business context when communicating the issue and options to management. Some of the best work experiences I've ever had were in orgs with such experienced developers and managers. Working together we were often able to solve apparently intractable problems with creative options which would never have been conceived absent the clarity of thought and communication on both sides.
- mrandish 4y ago
  > I want to believe you can make people curious In my experience it's the opposite. Most people are born innately curious. It takes intense systemic conditioning to suppress their natural curiosity. For many of us that conditioning begins in the traditional education system and is completed in the workplace.
  - jrlowe 4y ago
    >It takes intense systemic conditioning to suppress their natural curiosity. Yep, the public school system in America excels at squelching curiosity and any desire to learn.
    - sassy_quat 4y ago
      Just to be clear, you are saying, matter of factly, that the whole point of the American education system, is to suppress free thought. You better tell Congress, the last thing they want is for America to fall into totalitarianism.
      - NateEag 4y ago
        John Taylor Gatto was an award-winning public schoolteacher who more-or-less agreed with this sentiment. https://en.wikipedia.org/wiki/John_Taylor_Gatto https://en.wikipedia.org/wiki/John_Taylor_Gatto
        sassy_quat 4y ago
        How are you incapable of understand sarcasm? In fact you're arguing from Appeal to Authority now. I just, I don't understand. The weird thing is that your argument is extremely convincing at first glance even though the entirety of your argument is in a link to wikipedia?
        wizzwizz4 4y ago
        Sharing knowledge and understanding doesn't have to be an argument.
- khalladay 4y ago
  I'd be careful about painting the developers at Rockstar as "not curious," that feels like an incredibly unfair conclusion to draw from this. Those developers have other pressures, and deadlines to hit. There are a million reasons why this bug wasn't solved by the team there. When a user complains about a bug in software you wrote, is it because you just "weren't curious" enough? Or is the situation more complicated? Lots of incredibly talented people worked on GTA Online, it's worth trying to think through why a team of smart developers might not have prioritized this, rather than assuming something negative about them.
  - delusional 4y ago
    You might say that assuming this was due to inherent incuriousity amongst the developers is quite incurious.
  - azemetre 4y ago
    I wasn’t trying to blame engineers that wasn’t my intention, but someone needs to own establishing the environment that actively encourages or doesn’t encourage curiosity. Talented people are found everywhere, but hopefully someone with authority over them doesn’t slowly poison the drive out of them.
    - blowski 4y ago
      Why should developers necessarily be trying to cut loading times? Perhaps GTA is phenomenally successful because they have good BAs and POs who are able to identify and convince engineers to work on what matters to the bottom line.
      - dingleberry420 4y ago
        You've clearly never played it nor read any of the community discussions about the game. The #1 complaint is (or was) loading times. It's why I've personally stopped playing years ago.
        4y ago
- ajuc 4y ago
  In gamedev where crunch is the default and people are routinely working for 10, 12 or more hours a day, sometimes including weekends, sometimes for months on end - it's not very kind to assume it's curiosity that they lack.
  - azemetre 4y ago
    Yes but death marches are typically a sign of bad project management IME. As for lack of curiosity, the environment obviously plays a huge role in what is encouraged. I feel like history is repeating itself here (at least in regards to death marches): https://en.wikipedia.org/wiki/Erin_Hoffman#%22EA_Spouse%22_blog_post https://en.wikipedia.org/wiki/Erin_Hoffman#%22EA_Spouse%22_b...
- ramesh31 4y ago
  >Every company talks about wanting to have driven curious employees to solve complex issues or challenges but you just get stomped with a boot to your face if you do anything outside your lane or simply ask why. That's because "curiosity" translates to "more work" at #BigCo. And people work at #BigCo so that they can comfortably do as little work as necessary to maintain their lifestyles. This is a hard rule for any organization. So if you want what you're describing, it will only be found at small startups where everyone has skin in the game.
- rvba 4y ago
  It is not that developers arent curious. They probably are in their cages set by bad project managers who dont even play own game and dont notice it takes forever to load. The project managers could put load times as a priority but they genuinely dont see it as a problem or dont care at all. It happens all the time. There is a lot of unoptimized software. Perhaps the issue comes from the top: the project managers are supposed to only do things that increase revenue and all other things are not rewarded at all? Perhaps executives only care if they hit quartely results? (Short term gain traded for long term loss - the game is made for whales who are so invested that they will play even when it is technically bad - so milk them now with new content for whales and ignore everththing else, who cares that new players will be discouraged by bad load times). In my opinion if nobody plays own game / eats own dogfood, they dont even see its problems. Blizzard is at this stage now - some of their games barely work. You buy Starcraft 1 through the launcher... and it doesnt work(I think it works now, but it took them 1 month to notice it and another to fix it). Heroes of the storm - downloaded a 137mb patch every run - it took them half a year to notice it and do something with it. It is like nobody cares about the brand at all. Game company where your games dont work. IMHO a problem with people - that comes from the top. They want only story points for things that bring money. The project managers either dont care, or cannot deal with other problems at all. I bet they dont care. This happens in many companies - there were few people who tried, they got burned for that: terminated or quit -> so now nobody cares. "The game is good enough to still sell some skins to the whales, so why bother". Look at the results: they got an external comment what to fix. Did they launch their own internal project to optimize the game? Did they check for other things that make it load slow? It looka that someone else did a partial fix that is "good enough" so everything is back to usual and nobody cares about load times.
- rexpop 4y ago
  I'd never heard that Nabakov quote, but it's terrifically cynical and, I hope, false but, in my experience, not entirely untrue. Designers' unnecessary desire to delight users staves off the more Kafkaesque user experiences, and we all live in the gap of cops' unnecessary (occasional) sympathy for (some) nominal outlaws (we're all nominal outlaws).[0] When you mentioned curiosity I thought of a firm where colleagues scoffed at me as I rummaged through conference room drawers, saying, "you're not going to find anything interesting in there." I'm curious about cupboards as well as log traces -- an almost ubiquitously "eyes-glaze-over" boring artifact of modernity that, alongside popularly mind-numbing "code", I am equipped to find interesting. Is it any wonder, then, in this workplace where such innocent, cheap, curiosity were scoffed at, I was not too long after threatened with the extraordinarily archaic (in our at-will state, at least) legal notion of "insubordination"? It's a shame, because insofar as "all ambiguity is resolved by actions of practitioners at the sharp end of the system,"[1] the liberated curiosity of individual contributors is often the only "crack in everything" that's "how the light gets in."[2] 0. "With two lines of a man's handwriting, an accusation could be made against the most innocent." (Françoise Bertaut de Motteville) 1. https://how.complexsystems.fail/ https://how.complexsystems.fail/ 2. "Anthem" - Leonard Cohen
- deleted 4y ago
  [deleted]
- asdff 4y ago
  Curiosity has to come from the top and fall all the way down. As others and you have noted in other comments, its the environment for the workers that sets up whether or not they will even be able to pursue curious things, and that's on the managers to create that environent. However, the managers themselves have bossess too, so that has to trickle up, to their bosses bosses all the way up the chain. In short, every job needs to be able to have autonomy to enact on the ground solutions for what their immediate team needs. You shouldn't be penalized for going off the reservation and coming back with a creative solution. The problem is corporate structure for the past few decades has been about rooting out this autonomy and anything opaque about job responsibilities. There is a lot that's been done that would need undoing to change those environments, and imo you are better off finding another workplace than bringing about the monumental change required. A mature corporation is like a boulder tumbling down a mountainside; you can hardly change its course much less stop it short of where its heading.
- samstave 4y ago
  My brother is the polar opposite of me when it comes to curiosity. I ALWAYS ask questions like "How did this come to be, How much did this cost, who designed this and why" etc. My brother gets annoyed that A) I ask and question everything, and B) Laments he doesnt know as much as me. However, this is spurred by my ADHD... and I at times wish to be able to have a more precise control of my focus.
jenscow 4y ago
I strongly believe that R&D teams should have some time reserved to "do what you want" (limited to product improvement, obviously) Perhaps this might not be appropriate in the gaming industry... and who knows if Rockstar don't already have this. However, the GTA loading time is horrendous (and I had a spectrum sinclair when I was a kid) - surely there must have been a work item, ticket, bug reports, or whatever, for this.
boredemployee 4y ago
where I work at, it is completely discouraged to do anything out of what was asked you to do. so I don't judge the workers/slaves. It's not allowed to be creative/careful in most places if it's not harming the bosse's wallets.
- ReaLNero 4y ago
  In order for a worker to bother with this 1) It has to be alright to make miscellaneous changes like this, and 2) It has to contribute to getting promoted. The second point is IMO why random QoL changes like this are rare in large corporations.
  - switchbak 4y ago
    I've had this situation fluctuate at my workplace. When misc changes are supported, and those improvements are actually valued by management, I've seen the system improve dramatically. I've also seen a single-minded focus on pushing cards across a board, which seems to typically correlate with higher levels of a dogmatic SCRUM process. Pushing cards is great, but not when it prevents the various little things that have to happen to cultivate quality software. Especially not when there's a gatekeeper prioritizing these things into oblivion.
  - aidenn0 4y ago
    I don't know that #2 is universally true. I am at a point in my career now where I will be able to comfortably retire without ever being promoted again (i.,e. CoL raises only). Judging by the salary ranges I see on offer for jobs for people with 5+ years of experience, this can't be that rare. When I see a pain point that looks soluble, I take a crack at it. Sometimes it works, sometimes it doesn't. Sometimes I get recognized for it, sometimes I don't. One, relatively minor (to me) thing, ended up being a big business win and my manager made sure I got a fat bonus for it. Other things have seemed more like a win for me that nobody but 2 of my peers ended up caring about. #1 is true to the nth power though. I've seen places where if you do something even one millimeter outside of your silo, it will come up as a negative in your performance review. As in, if you write a 5 line perl script that makes you significantly more productive, do not share it as you will be asked time and time again why you wasted your time on something outside of your job description. I think something missing for the context of TFA is that the gaming industry tends to be much more deadline focused than other industries. In theory you might be free to "make miscellaneous changes like this" but only if things aren't already behind schedule, and things are always behind schedule.
jonwinstanley 4y ago
To be fair, the person that found the issue was persistent and did some serious digging to figure out what was going on. I imagine that even if you were on the GTA team, you'd most likely think the long loading time was unavoidable due to some complexity you were not aware of. Great work to solve it from outside of the company.
- shultays 4y ago
  Serious digging was needed because he does not have the source. Any devs could easily profile the game during loading and that sscanf or strlen would be at top Maybe they saw sscanf and assumed it is normal that it takes time, but more likely there were so much stuff to do and somehow no one checked
themerone 4y ago
I have a solid 2-3 years worth of work assigned to me in my issue tracker, with new issues rolling in every day. All I can do is prioritize and hope nobody gets too unhappy that their pet features is never going to happen.
xwolfi 4y ago
I work in a company where that can happen. In fact we had just a case today. For the last 20 years, our ever growing C# monster application used across the world to make millions, literally, had random crashes. Comes with volume spike, usage spike, we never quite knew: everything would display a red cross, the gdi handlers would reach 10k and bam. For 19 year we just looked at it in defeat. For the last year, bit on and off. Starting last monday it crashed every morning for a user sitting next to me, a big nasty german boss (Im French). I sat down, read his logs, saw it failed on ImageList creation again for some fucking reason, looked how many of these things we used and what they did. 2 hours later this 20 yo mystery bug dozens of devs had noped out of was fixed. We fully get it now. Wasnt even a matter of priority, it took 2 hours of looking at it. A lunch break basically. I dont have a sort of morale of the story or whatnot, but well, Rockstar would have figured it out eventually. Just need a nasty fucker next to you whining abt it everyday.
- LegitShady 4y ago
  2 hours lunch are you guys hiring?
  - xwolfi 4y ago
    Yes, desperately even, but if you join just for the lunch break (where we fix bugs as you saw lol, I haven't eaten during a lunch break for the last few years), it's a bit of a red flag :p But seriously, look at tier-2 investment banks: we're nicer and less stressed than the Goldmans of the world but still make more money than most. And problems are interesting I find. But you need to like handling money and not "changing the world" in any shape or form.
baby 4y ago
Shows that dogfooding is one of the best thing you can do for your product. I remember reading a story about Valve when they were working on Dota 2, I think Gabe said that every day after work they would play for hours and couldn't stop.
madrox 4y ago
This was discussed a bit in the original post [1] but I believe this comes down to the problem being sufficiently deep in the stack that there was an assumption that this wasn't something that could be improved on. I'm sure the loading code looked good. It took someone who didn't know what that code looked like being curious about what was actually happening. 1: https://news.ycombinator.com/item?id=26296339 https://news.ycombinator.com/item?id=26296339
havblue 4y ago
You would think that the more time people spend playing the game, the more money the game makes. I would think online loading metrics would be a similar problem for whomever is responsible for input latency.
tonmoy 4y ago
If I recall correctly the Dev team didn’t notice this as they were using a test JSON which was not very complicated. Only when the game was in production did the complex JSON cause the long loading times
greggman3 4y ago
There's a attitude, arguably backed by sales, that load times don't really matter. All of the Source games (HL2, Portal, Alyx, etc...) have arguably bad load times. They all sold great! GTA sold great even with it's previous load times. It's just not a priority for most teams. In Rockstar's case it happened to be easy to fix but in most cases it's not so easy because the engineering team didn't want to deal with it. AFAIK, Source, and maybe Unreal/Unity are built in such a way that by default, if you need to start the level over you have to re-load it entirely. They have mutable state spread all over the place and have no way to mark what's important (needs to be saved, what's not (can be zeroed), and/or they haven't bother to put all mutable state in a single places so either it can be restored from memory or loaded by just loading a tiny bit. There's zero reason HL:Alyx should take 15-30 seconds to restart a level you're already in when clearly less than 1k of state has changed since you started the level. But, fixing that would require a major re-design of the Source engine. Slow load times don't affect sales. So almost no one does it. Easy just to let each gameplay programmer do whatever they want and to restart just reload the level. I put up with it for some games (the ones mentioned above for example). But I've also quit a few recently. Games where you die a lot and to add to the frustration it takes 1-2 minutes to load again only to get killed in 5-20 seconds. That frustration means trial and erroring to figure out what to do gets a punch in the face and so I stop playing what might have been a good game otherwise.
viktorcode 4y ago
The most likely explanation is they never used this launcher themselves. That applies to QA department as well.
ChrisRR 4y ago
I'm not surprised really. Given how big a product like this is, how many people would actually have worked on this section of code itself.

There’s an implicit story here which I find quite sad that not a single developer at Rockstar in the prior eight years had the autonomy or bandwidth to discover and fix this themselves.

pinche_gazpacho 4y ago

nor the incentive

deleted 4y ago
[deleted]

azemetre 4y ago

I’m starting to slowly realize how terrible it is working in environments where people simply aren’t curious. It’s so hard for me as an individual contributor to decouple this way of thinking. Curiosity is how we find better ways of doing things or finding out hidden flows that can be better. Incurious people are a contagion, but I want to believe you can make people curious. It can’t be innate. Every company talks about wanting to have driven curious employees to solve complex issues or challenges but you just get stomped with a boot to your face if you do anything outside your lane or simply ask why. It’s also worse because I’m starting to realize how lucky I was to be raised as a curious individual from my parents who never went to college but at least were curious themselves about different things (pottery, soldering, gardening). The quote “Curiosity is insubordination in its purest form” suddenly has a different context when working for a company.

TheAceOfHearts 4y ago
Don't blame the people, blame the environment. People might start off curious but eventually they keep running into political roadblocks and other nonsense, which slowly beats the interest out of them. If those are the kinds of people that continue surviving then those are the kinds of people which your environment cultivates.
- azemetre 4y ago
  That’s totally fair, I honestly meant to blame the environment and not people. It just sucks to see the pattern of curiosity get stamped out of people due to environmental changes that aren’t obvious.
  - deleted 4y ago
    [deleted]
- miahi 4y ago
  Discussion from today: "Do you know if feature X works? A customer is asking about it" "No, it was disabled because of a bug a while back" "Hmm, found the ticket, it's from 4 years ago!" "Yes, basically it's a bug from a new version of a 3rd party software that made it unusable. At that point it was basically decided that we will not try to find a workaround, we just tell the customer that it's a 3rd party issue." The funny thing is, all the people who made that decision left the company/project and now we have the option to change it. Can't wait to see what happens next, if the new leadership changes that or not (my money is on "not"). The other funny thing is, I can think of at least two workarounds that only need a day of work and a couple for testing it.
alach11 4y ago
Sadly at Rockstar it's likely that employees are curious, but are under so much pressure for deadlines, and working such long hours, that they don't have time to follow their curiosity.
- switchbak 4y ago
  When problems are this obvious, it's surprising that someone doesn't have a look, even if they're busy. Maybe I'm asking too much/projecting here, but I just don't understand how people put up with fairly obvious semi-breakage as much as they do. I have this at my current job, I seem to be the only one (not really, but seems like it sometimes) interested in finding out why things are slow/broken and investing the time to fix it. If you're "busy" all the time, but put up with things that slow you down, you're hamstringing yourself by not fixing the brokenness. When only a small number of people take the time to fix things (typically other's issues), that becomes very demotivating. That would be like being on an all-day ski tour, and feeling too rushed to bother getting rid of the slush on your skis. Yes, you feel rushed because you're moving slower that usual. Taking 4 minutes to kick the slush off will make you much faster on the overall trip. (Sharpening the saw analogy, etc). Sometimes "busy" is a state of mind, more than it is a reflection of reality. I know each situation is unique, but I've seen this enough times to understand that it's often a form of dysfunction rather than a reasonable reaction to the work situation. This is like when people used to say "we're too busy to write tests". Well, maybe you need to get better at communicating the trade-offs to the broader business, and be a bit more assertive about non-functional aspects that are still important to address.
  - 0xFACEFEED 4y ago
    > Maybe I'm asking too much/projecting here, but I just don't understand how people put up with fairly obvious semi-breakage as much as they do. > I have this at my current job, I seem to be the only one (not really, but seems like it sometimes) interested in finding out why things are slow/broken and investing the time to fix it. You will quickly shed yourself of this philosophy if you become a business owner. You can test those waters by releasing a non-trivial open source project that gains a non-trivial amount of users. There will always be at least 100 things that are broken in some way but not impacting the core business. You'll have to pick only 1 to work on. And, if you're unlucky, some frustrated customer may write a blog post about issue #65. An issue that you're fully aware of but haven't prioritized. Guess what you'll do as a business owner? You'll prioritize that issue since it's now very visible and maybe even throw $10,000 at the customer who took time to do their research; as is what happened here. > This is like when people used to say "we're too busy to write tests". Well, maybe you need to get better at communicating the trade-offs to the broader business, and be a bit more assertive about non-functional aspects that are still important to address. Or perhaps the need for tests was communicated and the business made the decision to not prioritize it. You may not like that decision. Mind that the decision to "not write tests" is never explicitly stated so simply. No one wants to look bad. Instead that decision is expressed through minimizing its level of priority compared to other things. Still don't like the decision? Well you have three options: 1) Ignore the stated goals of the company and do what you think is right instead. In which case you're acting against the interests of the company that's paying you. 2) Sacrifice your own free time in order to provide your idea of value to the company (that may actually be valuable! but that the company doesn't value enough to prioritize) 3) Do what the company asks with plausible deniability. Also known as kicking the can. Also known as not my problem. Sometimes doing #1 or #2 could greatly benefit you. Perhaps that thing you're doing (eg: writing tests) will somehow measurably improve the business in the future WITH PROOF so that you'll get get recognized. Or perhaps there's management shake-up and the new leadership perfectly aligns with your philosophy, etc, etc. But that's rare. More often than not your attempt at fixing the company's problems behind the scenes will go unnoticed and unrewarded. And if you do get rewarded, often times it's a pittance compared to the amount of time you invested. My personal philosophy is I simply do not work for companies that don't align with what I value. For example, to me, it's exhausting and unrewarding to work at a company that doesn't appreciate a GREAT user experience.
jdwithit 4y ago
+1 to the other replies here, I think it's very poor form to throw the engineers under the bus for not being curious. The most likely explanation is they were under insane time and resource constraints, especially in the games industry. They wrote something that worked, then were forced to move on to the other 5000 tasks in their backlog that had to get done before release. When you're already being told to crunch 80+ hour weeks for months or years at a time, you run out of gas. I guess it's possible that the engineers who were talented enough to create this extremely impressive game in the first place somehow threw up their hands and said "guess it can never take less than 7 minutes to load, best we can do". But it seems much more likely that they were well aware, but never allowed by poor management to go back and fix it. The game was released and printing money, why bother. They're on to the next title.
- ALittleLight 4y ago
  The problem with this way of thinking is that it basically absolves everyone of blame. Bad stuff happens and nobody's at fault. You point to the managers, because the engineers were so busy, but the managers are going to claim they were busy too. Again, from the manager's perspective - the managers aren't even in a position to understand the problem or the reasons for it.
  - delusional 4y ago
    "the managers aren't even in a position to understand the problem or the reasons for it." Which is again the fault of management. It is the responsibility of management to facilitate proper management.
    - blowski 4y ago
      As a shareholder, I’d be rather happy with the management of GTA - it’s phenomenally profitable. That sounds like proper management to me.
  - mrandish 4y ago
    Having been on both sides of this (as a developer and later a senior manager with product/feature responsibility), I'd say it can be both. A really good manager/executive has enough knowledge and experience to ask the right questions and parse the answers even when they are technical. Sadly, many managers aren't that good (at least yet). On the other side, a really good developer has the knowledge and experience to frame the problem in an understandable business context when communicating the issue and options to management. Some of the best work experiences I've ever had were in orgs with such experienced developers and managers. Working together we were often able to solve apparently intractable problems with creative options which would never have been conceived absent the clarity of thought and communication on both sides.
mrandish 4y ago
> I want to believe you can make people curious In my experience it's the opposite. Most people are born innately curious. It takes intense systemic conditioning to suppress their natural curiosity. For many of us that conditioning begins in the traditional education system and is completed in the workplace.
- jrlowe 4y ago
  >It takes intense systemic conditioning to suppress their natural curiosity. Yep, the public school system in America excels at squelching curiosity and any desire to learn.
  - sassy_quat 4y ago
    Just to be clear, you are saying, matter of factly, that the whole point of the American education system, is to suppress free thought. You better tell Congress, the last thing they want is for America to fall into totalitarianism.
    - NateEag 4y ago
      John Taylor Gatto was an award-winning public schoolteacher who more-or-less agreed with this sentiment. https://en.wikipedia.org/wiki/John_Taylor_Gatto https://en.wikipedia.org/wiki/John_Taylor_Gatto
      - sassy_quat 4y ago
        How are you incapable of understand sarcasm? In fact you're arguing from Appeal to Authority now. I just, I don't understand. The weird thing is that your argument is extremely convincing at first glance even though the entirety of your argument is in a link to wikipedia?
        wizzwizz4 4y ago
        Sharing knowledge and understanding doesn't have to be an argument.
khalladay 4y ago
I'd be careful about painting the developers at Rockstar as "not curious," that feels like an incredibly unfair conclusion to draw from this. Those developers have other pressures, and deadlines to hit. There are a million reasons why this bug wasn't solved by the team there. When a user complains about a bug in software you wrote, is it because you just "weren't curious" enough? Or is the situation more complicated? Lots of incredibly talented people worked on GTA Online, it's worth trying to think through why a team of smart developers might not have prioritized this, rather than assuming something negative about them.
- delusional 4y ago
  You might say that assuming this was due to inherent incuriousity amongst the developers is quite incurious.
- azemetre 4y ago
  I wasn’t trying to blame engineers that wasn’t my intention, but someone needs to own establishing the environment that actively encourages or doesn’t encourage curiosity. Talented people are found everywhere, but hopefully someone with authority over them doesn’t slowly poison the drive out of them.
  - blowski 4y ago
    Why should developers necessarily be trying to cut loading times? Perhaps GTA is phenomenally successful because they have good BAs and POs who are able to identify and convince engineers to work on what matters to the bottom line.
    - dingleberry420 4y ago
      You've clearly never played it nor read any of the community discussions about the game. The #1 complaint is (or was) loading times. It's why I've personally stopped playing years ago.
      - 4y ago
ajuc 4y ago
In gamedev where crunch is the default and people are routinely working for 10, 12 or more hours a day, sometimes including weekends, sometimes for months on end - it's not very kind to assume it's curiosity that they lack.
- azemetre 4y ago
  Yes but death marches are typically a sign of bad project management IME. As for lack of curiosity, the environment obviously plays a huge role in what is encouraged. I feel like history is repeating itself here (at least in regards to death marches): https://en.wikipedia.org/wiki/Erin_Hoffman#%22EA_Spouse%22_blog_post https://en.wikipedia.org/wiki/Erin_Hoffman#%22EA_Spouse%22_b...
ramesh31 4y ago
>Every company talks about wanting to have driven curious employees to solve complex issues or challenges but you just get stomped with a boot to your face if you do anything outside your lane or simply ask why. That's because "curiosity" translates to "more work" at #BigCo. And people work at #BigCo so that they can comfortably do as little work as necessary to maintain their lifestyles. This is a hard rule for any organization. So if you want what you're describing, it will only be found at small startups where everyone has skin in the game.
rvba 4y ago
It is not that developers arent curious. They probably are in their cages set by bad project managers who dont even play own game and dont notice it takes forever to load. The project managers could put load times as a priority but they genuinely dont see it as a problem or dont care at all. It happens all the time. There is a lot of unoptimized software. Perhaps the issue comes from the top: the project managers are supposed to only do things that increase revenue and all other things are not rewarded at all? Perhaps executives only care if they hit quartely results? (Short term gain traded for long term loss - the game is made for whales who are so invested that they will play even when it is technically bad - so milk them now with new content for whales and ignore everththing else, who cares that new players will be discouraged by bad load times). In my opinion if nobody plays own game / eats own dogfood, they dont even see its problems. Blizzard is at this stage now - some of their games barely work. You buy Starcraft 1 through the launcher... and it doesnt work(I think it works now, but it took them 1 month to notice it and another to fix it). Heroes of the storm - downloaded a 137mb patch every run - it took them half a year to notice it and do something with it. It is like nobody cares about the brand at all. Game company where your games dont work. IMHO a problem with people - that comes from the top. They want only story points for things that bring money. The project managers either dont care, or cannot deal with other problems at all. I bet they dont care. This happens in many companies - there were few people who tried, they got burned for that: terminated or quit -> so now nobody cares. "The game is good enough to still sell some skins to the whales, so why bother". Look at the results: they got an external comment what to fix. Did they launch their own internal project to optimize the game? Did they check for other things that make it load slow? It looka that someone else did a partial fix that is "good enough" so everything is back to usual and nobody cares about load times.
rexpop 4y ago
I'd never heard that Nabakov quote, but it's terrifically cynical and, I hope, false but, in my experience, not entirely untrue. Designers' unnecessary desire to delight users staves off the more Kafkaesque user experiences, and we all live in the gap of cops' unnecessary (occasional) sympathy for (some) nominal outlaws (we're all nominal outlaws).[0] When you mentioned curiosity I thought of a firm where colleagues scoffed at me as I rummaged through conference room drawers, saying, "you're not going to find anything interesting in there." I'm curious about cupboards as well as log traces -- an almost ubiquitously "eyes-glaze-over" boring artifact of modernity that, alongside popularly mind-numbing "code", I am equipped to find interesting. Is it any wonder, then, in this workplace where such innocent, cheap, curiosity were scoffed at, I was not too long after threatened with the extraordinarily archaic (in our at-will state, at least) legal notion of "insubordination"? It's a shame, because insofar as "all ambiguity is resolved by actions of practitioners at the sharp end of the system,"[1] the liberated curiosity of individual contributors is often the only "crack in everything" that's "how the light gets in."[2] 0. "With two lines of a man's handwriting, an accusation could be made against the most innocent." (Françoise Bertaut de Motteville) 1. https://how.complexsystems.fail/ https://how.complexsystems.fail/ 2. "Anthem" - Leonard Cohen
deleted 4y ago
[deleted]
asdff 4y ago
Curiosity has to come from the top and fall all the way down. As others and you have noted in other comments, its the environment for the workers that sets up whether or not they will even be able to pursue curious things, and that's on the managers to create that environent. However, the managers themselves have bossess too, so that has to trickle up, to their bosses bosses all the way up the chain. In short, every job needs to be able to have autonomy to enact on the ground solutions for what their immediate team needs. You shouldn't be penalized for going off the reservation and coming back with a creative solution. The problem is corporate structure for the past few decades has been about rooting out this autonomy and anything opaque about job responsibilities. There is a lot that's been done that would need undoing to change those environments, and imo you are better off finding another workplace than bringing about the monumental change required. A mature corporation is like a boulder tumbling down a mountainside; you can hardly change its course much less stop it short of where its heading.
samstave 4y ago
My brother is the polar opposite of me when it comes to curiosity. I ALWAYS ask questions like "How did this come to be, How much did this cost, who designed this and why" etc. My brother gets annoyed that A) I ask and question everything, and B) Laments he doesnt know as much as me. However, this is spurred by my ADHD... and I at times wish to be able to have a more precise control of my focus.

jenscow 4y ago

I strongly believe that R&D teams should have some time reserved to "do what you want" (limited to product improvement, obviously) Perhaps this might not be appropriate in the gaming industry... and who knows if Rockstar don't already have this. However, the GTA loading time is horrendous (and I had a spectrum sinclair when I was a kid) - surely there must have been a work item, ticket, bug reports, or whatever, for this.

boredemployee 4y ago

where I work at, it is completely discouraged to do anything out of what was asked you to do. so I don't judge the workers/slaves. It's not allowed to be creative/careful in most places if it's not harming the bosse's wallets.

ReaLNero 4y ago
In order for a worker to bother with this 1) It has to be alright to make miscellaneous changes like this, and 2) It has to contribute to getting promoted. The second point is IMO why random QoL changes like this are rare in large corporations.
- switchbak 4y ago
  I've had this situation fluctuate at my workplace. When misc changes are supported, and those improvements are actually valued by management, I've seen the system improve dramatically. I've also seen a single-minded focus on pushing cards across a board, which seems to typically correlate with higher levels of a dogmatic SCRUM process. Pushing cards is great, but not when it prevents the various little things that have to happen to cultivate quality software. Especially not when there's a gatekeeper prioritizing these things into oblivion.
- aidenn0 4y ago
  I don't know that #2 is universally true. I am at a point in my career now where I will be able to comfortably retire without ever being promoted again (i.,e. CoL raises only). Judging by the salary ranges I see on offer for jobs for people with 5+ years of experience, this can't be that rare. When I see a pain point that looks soluble, I take a crack at it. Sometimes it works, sometimes it doesn't. Sometimes I get recognized for it, sometimes I don't. One, relatively minor (to me) thing, ended up being a big business win and my manager made sure I got a fat bonus for it. Other things have seemed more like a win for me that nobody but 2 of my peers ended up caring about. #1 is true to the nth power though. I've seen places where if you do something even one millimeter outside of your silo, it will come up as a negative in your performance review. As in, if you write a 5 line perl script that makes you significantly more productive, do not share it as you will be asked time and time again why you wasted your time on something outside of your job description. I think something missing for the context of TFA is that the gaming industry tends to be much more deadline focused than other industries. In theory you might be free to "make miscellaneous changes like this" but only if things aren't already behind schedule, and things are always behind schedule.

jonwinstanley 4y ago

To be fair, the person that found the issue was persistent and did some serious digging to figure out what was going on. I imagine that even if you were on the GTA team, you'd most likely think the long loading time was unavoidable due to some complexity you were not aware of. Great work to solve it from outside of the company.

shultays 4y ago
Serious digging was needed because he does not have the source. Any devs could easily profile the game during loading and that sscanf or strlen would be at top Maybe they saw sscanf and assumed it is normal that it takes time, but more likely there were so much stuff to do and somehow no one checked

themerone 4y ago

I have a solid 2-3 years worth of work assigned to me in my issue tracker, with new issues rolling in every day. All I can do is prioritize and hope nobody gets too unhappy that their pet features is never going to happen.

xwolfi 4y ago

I work in a company where that can happen. In fact we had just a case today. For the last 20 years, our ever growing C# monster application used across the world to make millions, literally, had random crashes. Comes with volume spike, usage spike, we never quite knew: everything would display a red cross, the gdi handlers would reach 10k and bam. For 19 year we just looked at it in defeat. For the last year, bit on and off. Starting last monday it crashed every morning for a user sitting next to me, a big nasty german boss (Im French). I sat down, read his logs, saw it failed on ImageList creation again for some fucking reason, looked how many of these things we used and what they did. 2 hours later this 20 yo mystery bug dozens of devs had noped out of was fixed. We fully get it now. Wasnt even a matter of priority, it took 2 hours of looking at it. A lunch break basically. I dont have a sort of morale of the story or whatnot, but well, Rockstar would have figured it out eventually. Just need a nasty fucker next to you whining abt it everyday.

LegitShady 4y ago
2 hours lunch are you guys hiring?
- xwolfi 4y ago
  Yes, desperately even, but if you join just for the lunch break (where we fix bugs as you saw lol, I haven't eaten during a lunch break for the last few years), it's a bit of a red flag :p But seriously, look at tier-2 investment banks: we're nicer and less stressed than the Goldmans of the world but still make more money than most. And problems are interesting I find. But you need to like handling money and not "changing the world" in any shape or form.

baby 4y ago

Shows that dogfooding is one of the best thing you can do for your product. I remember reading a story about Valve when they were working on Dota 2, I think Gabe said that every day after work they would play for hours and couldn't stop.

madrox 4y ago

This was discussed a bit in the original post [1] but I believe this comes down to the problem being sufficiently deep in the stack that there was an assumption that this wasn't something that could be improved on. I'm sure the loading code looked good. It took someone who didn't know what that code looked like being curious about what was actually happening. 1: https://news.ycombinator.com/item?id=26296339 https://news.ycombinator.com/item?id=26296339

havblue 4y ago

You would think that the more time people spend playing the game, the more money the game makes. I would think online loading metrics would be a similar problem for whomever is responsible for input latency.

tonmoy 4y ago

If I recall correctly the Dev team didn’t notice this as they were using a test JSON which was not very complicated. Only when the game was in production did the complex JSON cause the long loading times

greggman3 4y ago

There's a attitude, arguably backed by sales, that load times don't really matter. All of the Source games (HL2, Portal, Alyx, etc...) have arguably bad load times. They all sold great! GTA sold great even with it's previous load times. It's just not a priority for most teams. In Rockstar's case it happened to be easy to fix but in most cases it's not so easy because the engineering team didn't want to deal with it. AFAIK, Source, and maybe Unreal/Unity are built in such a way that by default, if you need to start the level over you have to re-load it entirely. They have mutable state spread all over the place and have no way to mark what's important (needs to be saved, what's not (can be zeroed), and/or they haven't bother to put all mutable state in a single places so either it can be restored from memory or loaded by just loading a tiny bit. There's zero reason HL:Alyx should take 15-30 seconds to restart a level you're already in when clearly less than 1k of state has changed since you started the level. But, fixing that would require a major re-design of the Source engine. Slow load times don't affect sales. So almost no one does it. Easy just to let each gameplay programmer do whatever they want and to restart just reload the level. I put up with it for some games (the ones mentioned above for example). But I've also quit a few recently. Games where you die a lot and to add to the frustration it takes 1-2 minutes to load again only to get killed in 5-20 seconds. That frustration means trial and erroring to figure out what to do gets a punch in the face and so I stop playing what might have been a good game otherwise.

viktorcode 4y ago

The most likely explanation is they never used this launcher themselves. That applies to QA department as well.

ChrisRR 4y ago

I'm not surprised really. Given how big a product like this is, how many people would actually have worked on this section of code itself.