12 ms·

I fixed a bug the other day

charcircuit 3y ago
This sounds like a monitoring problem. If it's happening in production there isn't a requirement to figure out how to reproduce it locally.
- sosodev 3y ago
  You need to reproduce the bug to know if your fix works. Not always but that’s how most teams operate. Particularly because you typically won’t pass review without some tangible improvement.
  - javier123454321 3y ago
    Yeah, we actually needed to 'test' in production. I wrote tests for how it was supposed to work, and made the best effort to reduce the risk, but the only way to get a reproducible environment for this particular issue was to deploy the change. That definitely contributed, though was not exclusively the cause of, the risk aversion.
mypastself 3y ago
Did the author go behind his manager’s back to pick up the task, or was it assigned? Because if it’s the former, it’s possible that the issue was not addressed because it wasn’t deemed high-priority compared to other tasks, regardless of the number of user complaints. If it’s the latter, well, I don’t know. Congratulations on completing an assigned task at your job?
- withinboredom 3y ago
  Seemed like it was a case of "cannot reproduce" and any attempt at reproducing it failed. Since it couldn't be reproduced, it was never prioritized because it never made it past triage. At least that's the workflow I imagine when reading this and from working at similar places. Also, I advise leaving a place that requires your manager to dictate which tickets you work on. Sounds like a crappy place to work.
  - LambdaComplex 3y ago
    > I advise leaving a place that requires your manager to dictate which tickets you work on As someone who recently left a place like that: I agree. I'm a highly-paid professional; it was really frustrating to feel like my manager didn't trust me enough to figure out what I needed to work on.
    - bryanrasmussen 3y ago
      It really depends on what you are shipping: 1. does what you are shipping have lots of international regulations and compliance requirements per country it will be available in with new countries being rolled out on a planned basis -> your manager will determine what you should work on 2. Are there things being released on a set date due to some sort of legal requirement, business deal? -> your manager will determine what you should work on 3. Is it basically the same product in all markets / countries other than internationalized text with not pending contracts or business deals requiring specific functionalities impacting the company? You should be able to determine what you should be working on.
      - withinboredom 3y ago
        I disagree. Though (1) and (2) are good points, but they shouldn't be telling you what to work on at the individual ticket level. Even then, they should prioritize and set a healthy deadline (if appropriate); and that is it. Implementing a high priority ticket may still require addressing technical debt to implement, or dealing with other lower priority tickets first -- because they block the new, higher priority ticket. If your manager needs to know all these details and can tell you "no, you can't fix X to build Y" then I really think that is a crappy place to work; not just due to the politics involved, but also the code must be absolute crap.
    - maccard 3y ago
      Manager/Tech Lead here who had someone who would choose their tickets themselves. It's a giant pain and makes my job 10x more difficult when I tell another team that something that's low priority to us but high priority to them that something will be done, and instead find it sitting in in-progress with a tech debt bugbear of theirs in review. The priority on the ticket was set for a reason, it was assigned and scheduled for a reason. There's only so many hours in the day to explain why some things are prioritised. If you think that's wrong, be a professional and talk to me about it. I promise you, I want to have the conversation that you need to actually work with your team far less than you want to fix that JSON output.
- LambdaComplex 3y ago
  The author says "I had never looked at this code, but was told to take a look," so I assume they weren't doing this behind their manager's back.
- javier123454321 3y ago
  The manager told me something along the lines of "hey this has been causing issues, can you take a look? Others haven't been able to figure this out." It's the latter part of the statement that I was musing over in the article. I'll take the sarcastic congrats, though, better than internet name calling.
  - mypastself 3y ago
    Must have missed that part. But my intention wasn’t to offend. It felt like the part about solving a problem other developers couldn’t was the main crux of the article, and the insight about inertia in big companies was tacked on. But I’m sure others will disagree.
- cratermoon 3y ago
  I read it as saying everybody else gave excuses to get out of dealing with it while the author went ahead and did it when assigned, even though all the previous people had said it couldn't be fixed.
- dontupvoteme 3y ago
  I don't work in a software company so I found it interesting to see how things can become dysfunctional there.
devjab 3y ago
I’ve worked quite a bit in enterprise software development, and later in teams that are transitioning from startups/mid-size to enterprise and in my very anecdotal experience it’s always down to the internal processes. Software developers like solving issues. Most people, myself included prefer building new things, but a lot of us actually like solving bugs as well. At least that is my experience. What I’ve seen get in the way is when the processes around reporting, examining, describing and prioritising the “tasks” get so convoluted that that developers get either overwhelmed or detached from the issue by too much bureaucracy. Which happens very often in non-tech organisations in my experience. Maybe it also happens on tech organisations but I’ve never worked in one, but from buying software from around two hundred it didn’t seem like it was a problem in quite the same way it can in regular enterprise. I realise this may be a little more clear in my head than for you as the reader. So I’ll try to explain by an example I encountered recently with a parking app. My wife and I were visiting a midwife for a comfort scan, and they have you register your license plate in an app so that their patrolmen/parking-guards/whatever they are called don’t give you a ticket when they scan your car. Anyway, the app wasn’t working and the midwife was like: “oh, that happens from time to time, don’t worry that means they can’t check if parking are legal or not”. As though it was the most normal thing in the world. Because it is… IT failing is the most normal thing in the world to a lot of people, but think about it. That app being down, at least on a city scale maybe even global meant that all those parking-guards were paid to do nothing for however long it took to fix it. It also meant anyone could essentially park for free… and that was a semi-regular thing? When you work in enterprise organisations. They often become accustomed to IT not working. Part of this is because the process people they put in between IT professionals and their organisation don’t actually know IT. They are very good at defining processes, making things lean and what not, but that’s not how you actually make things work in my experience. I’m guessing this is true outside of IT as well, but once you have more process managers, scrum masters, product owners and what not than actual technicians, then you’ve probably ducked it up. This is because programmers are often the best people at determining what’s important, not always, but often, and despite an entire industry’s attempts at “streamlining” the development process I suspect it’s also why sometimes a team of 4 developers can do more quality work in a shorter amount of time than 10 teams of 10 people combined. It sounds like the author has encountered some of this. They have an important issue. It loses some of its importance on the processes and then once it finally gets to the technicians it can’t be reproduced… Of course it doesn’t get solved. Especially if the organisation has set up systems to reward not solving seemingly unimportant reproducible issues, which many enterprise organisations end up doing. Corporate culture is hard.
samhuk 3y ago
> Deny that there is a problem. > Deny that it is your problem. > Ask for more information. > Complain. Oh my does this ever bring traumatic memories! This entire article reminds me of a previous company I used to work for. There was a sufficiently large number of engineers that had a similar, if not more elaborate, deliberate, and sinister way of responding to bugs: * Deny {bug} is a problem - "I don't see how this affects you/anyone?" * Deny {bug} can even be fixed - "Ok, fair, but I don't see how this can be fixed?" * Deny {bug} is their problem - "Ok, fair, but have you tried asking {other employee?}, they are responsible for this." (they are obviously not) * Be rude - "If you think this is a problem, then go fix it" All this was just obvious defensive walls of lies to protect their reputation and hide their lack of skill and their laziness. Around 5% of the engineers were like this and I've always analogized it to the proportion of U238 in a sample of Uranium - just enough enriched dogshit to cause a critical mass of misery. I left (for greener pastures) shortly after I had a couple of tasks requiring interaction with these "engineers", which made working around them impossible, and then realizing that leadership had totally lost control of the zoo and didn't care one iota.
- withinboredom 3y ago
  I very recently had to push back on a bug that wasn’t a bug in my part of the stack. If they wanted it “fixed” in my part, it would be a feature. (Enum values from an API being localized — these were hardcoded on the backend). IMHO, these should be localized in the client, which has the current locale (I do not have the current locale and no strings are translated from this API by this API). That was pretty fun, but I don’t think generalizing “denying a bug” as saying we are being “lazy” is a correct assumption.
  - samhuk 3y ago
    I agree, "denying a bug" does not equal "lazy", but rather "denying a bug one has created" does at best equal "lazy", at worst malicious (e.g. rep protection).
  - dkarl 3y ago
    I had a similar experience adding new functionality to a web application. It was a simple enough feature that I was able to supply the English myself, but I knew enough not to provide the international copy, even though the only other language we supported was a language I studied in school. I asked the product manager a few times, then created a ticket assigned to them and sent them the link. Meanwhile, my code got reviewed and deployed, behind a feature flag of course. As soon as they found out the code was deployed, the product manager asked if we could turn on the feature. I reminded them that the reason we supported the other language was that it was a legal requirement for our customers in a certain country. They said turn in on, nobody was going to notice except a big US customer who desperately needed it. So I did. Within a few hours we had an enraged customer on the phone. Credit to the product manager, they took full responsibility, but the initial response from the founder/head of customer success who took the call was to walk straight to my desk and ask me what the hell I did.
altacc 3y ago
> What I did is come in as an outsider [and] let my naivety guide my investigation... I call this being "a useful idiot", which I sometimes use to describe my role. A large part of my job is floating between teams & applications, identifying gaps and problems and knowing just enough to ask meaningful questions whilst not being deeply invested in the existing code & processes. I can see people asking themselves "why are they wasting my time even asking these questions?!" but in explaining the answer they get to review the process and sometimes work out a new solution. We all get comfortable with how things are and someone prodding us occasionally helps creativity.
- mbork_pl 3y ago
  Nice, but nit: this is not what a "useful idiot" usually means.
  - vxNsr 3y ago
    It’s an ironic use of the term.
  - altacc 3y ago
    I live in a country where English is not the native language and the idiom is not in common usage, so the literal translation works. Sometimes I often wonder if it would be useful for more internet forums to allow users to add a flair to denote they are non US, as the assumption on so many sites is that everyone is US and living on the east or west coast.
    - deleted 3y ago
      [deleted]
    - mbork_pl 3y ago
      I think I agree in general, but in this case he "idiom" (in fact, more of a metaphor, I guess) is actually Russian and just translated literally into English.
roydivision 3y ago
The satisfaction of finding the root cause of something like this and then fixing it is great, and one of the reasons I’m still in this industry. The great write up reminds me of a few similar cases I’ve dealt with over my career, I still think back to them and smile to myself.
motbus3 3y ago
In my company we have a bug sitting in the to do list for about a year. The root cause was a previous change in requirements that was not well defined and explored a couple of years ago. None of the original developers are working for the company anymore. We offered ourselves to fix that bug, but it would take at least a couple of weeks. This bug prevents us to roll out other deemed important features. But, it is just a bug. No product owner wants it to say to his boss he spent 2w of an engineer solving an issue instead of building another feature. The boss of the boss called. He says that this bug is not tolerable. He ask us to fix it. We say we agree and we already have screened and have a solution for it, but it will take 2 weeks. He says he wants that earlier and we say no because no one wants to rush it as we don't want to cause a possible side effect if we fail in something. The solution according to the is to not fail. Everybody goes back to the planning and no one picks that issue because the condition is to do it in half of time without fail while working on a new feature. Go back to line 1.
- Etheryte 3y ago
  Ah yes, the classic low-skill management maneuver of setting an unrealistic deadline in the hopes that people will then work faster. Naturally, Joel has written about this [0], see point 12. [0] https://www.joelonsoftware.com/2000/03/29/painless-software-schedules/ https://www.joelonsoftware.com/2000/03/29/painless-software-...
  - gberger 3y ago
    > 8) Put in line items for Vacations, Holidays, etc. If your schedule is going to take about a year, each programmer will probably take 10 to 15 days of vacation. That's so little vacation. America is wild.
    - Ryoung27 3y ago
      Wild is a nice way of saying crushing my soul. I've worked at companies that give you 10 days, that they choose, for your first five years. I have no clue why we do this to ourselves other than the person in charge is not bound by it, and one day we all hope to be in charge.
alentred 3y ago
Congrats for the bug fixed! I know how this kind of bug can be really hard. For a company. Less so for "an outsider", just like the author suggests. I can easily see this happening in literally any company. We humans are very adaptable creatures, we get accustomed to things and just accept them as they are. In my company I preach for razing the flags and looking for things that just don't look right, and we generally do so indeed, but I am pretty sure that for an outsider some acceptable "features" will look very questionable. Rule of thumb I use sometimes: does it feel awkward to answer some of the newcomers’ questions? Independently of their experience, newcomers always ask very good questions. -- Serenity Prayer comes to mind: > God, give me the serenity to accept the things I cannot change, > Courage to change the things I can, > and Wisdom to know the difference.
- Loughla 3y ago
  I have the full version of the serenity prayer as a cross stitch framed in my office. >God, grant me the serenity to accept the things I cannot change, >The courage to change the things I am able, >The wisdom to know the difference, >And the ability to destroy all of the evidence of the inevitable murder, should the first three fail.
- jimmaswell 3y ago
  > I know how this kind of bug can be really hard. It sounds more like it was embarrassingly easy, but the author is surrounded by 0.1x engineers.
perlgeek 3y ago
So, people have blamed the vendor's API... but have they opened a ticket with them, raised the issue, escalated it? I can emphasize with not wanting to work around the bugs of an upstream provider, but just sitting on it when it causes real pain to customers is also shitty. > That's right, for any developer, the answer here is simple: use the coupons that we are ALREADY receiving on the frontend to populate the fields that users expect so they can get their discounts. Hard to tell without knowing more details, but that might run into the risk that people can inject made-up discounts in the browser and send them to the backend, which then charges less.
- beezlewax 3y ago
  Sounds like that to me too. I'm sure there is more going on there though
- doix 3y ago
  > Hard to tell without knowing more details, but that might run into the risk that people can inject made-up discounts in the browser and send them to the backend, which then charges less. That was my first thought, but thinking about it more, it probably makes no difference. If it's susceptible to injected made-up discounts now, it would have been susceptible before as well. It doesn't matter if the coupon code comes from the vendor API or their own API if you're going to inject fake data anyway.
- javier123454321 3y ago
  I'm not privy to the extent of the conversations with the vendor, I came in with fresh eyes and found a solution. The writeup was just musing on the process that let it get to that point from my perspective. Your final issue is not a risk in this case, because we still are validating the coupons. This was a frontend fix, and we obviously don't trust any data coming from the frontend.
- Nemi 3y ago
  Here’s the deal, at two years and for a problem that is by all accounts serious, any solution is better than no solution. Now, if this is code that runs ventilators for sick people, runs the movement of money for a hedge fund, or balances the US budget, then no, you have to be absolutely, positively sure that you are not causing another problem. But when you have a real problem that is affecting people everyday for years, then a little creativity is certainly called for. Even at the expense of hypothetically causing another problem.
eliac 3y ago
> Harvard and others have written about it Why Big Companies Can't Innovate. I am interested in this, because I am interested in doing work, at scale, within a company that has a culture of solving problems, not of being a behemoth with enough inertia to not need to correct when there is a need. You might like "Loonshots" by Safi Bahcall. The book proposes to solve this by nurturing an innovation unit inside the behemoth, and carefully maintaining the balance between the two.
deleted 3y ago
[deleted]
globalise83 3y ago
Escalating this issue to a senior director and have them call up the CEO of the marketing partner and threaten to switch providers if they don't resolve the coupon bug within 48 hours would have solved this problem a long time ago.
duckmysick 3y ago
I don't understand the problem. Or rather the design of their system. > The issue was that our system for weekly ad was having an issue where the ads needed to 'clip' a coupon (read hit an api) for the discounted price to reach our cart calculations. However, the api that we used for managing the weekly ads was managed by an external vendor, and often when we queried their endpoint, the coupon field was returning empty when it shouldn't. What's the authoritative source if I want to know that a product has a discount? The external vendor? If the coupon field is empty, doesn't it mean there's no discount? Unless it means there are two different fields: one for is_there_a_discount and another for coupon_id. Would there ever be an expected situation where there is a discount but not a coupon? Or maybe the source is the internal system. In which case, can't the coupon be generated there? Surely the source is not the frontend, right? > Products already had the associated coupons. From where? The source aside, does that mean the coupon_id never change or expire? If the same product has a different discount a few months from now, will it still use the same coupon? Also, the apparent solution was hardcoding: > so I completely hacked it, and replaced the api call service with a dumb function that returned hardcoded data copied from a production. but a couple paragraphs before, the author mentioned it changes often: > Because this data was missing a field, it required manual data entry to update it. However, this data changed in real time, so it was OFTEN out of date. I don't understand how this was a showstopper before but isn't anymore. Or maybe these paragraphs talk about different things. I don't question the bug fix or the author. I just can't grok it.
- KnobbleMcKnees 3y ago
  I believe the part about hard coding production data was referring to recreation of the bug, not implementing the solution which the author describes subsequently as "a couple of array functions". There's a fair bit of hand waving going on with technical details throughout but I don't think it's badly intended, more aimed at keeping the focus on the organisational issues described.
- javier123454321 3y ago
  Hey, Author here. The source was our internal system, but the vendor was taking our id, converting it to their own id, and then serving us an ad with the coupon labeled formatted to their system. We were then using the vendor id to translate it back to ours. The coupons associated with the products were already from our system, so I could skip the entire translation back to our id. The manual overrides bit can be omitted, it's just how people in product were fighting a loosing battle of manually overriding the bad data. The hardcoding was in order to replicate the behavior, as it wasn't showing up in lower envs. It wasn't the fix, the fix was to just render data we already had. I didn't go too much into details because 1. I can't due to company policy, and 2. it's actually not the interesting bit. I was more making observation of the process.
foobarbecue 3y ago
Oh my god, I absolutely hate the torture system this person works for. Pretty sure they are talking about the thing at Ralph's where you have to "clip" a coupon by scanning a QR code in the store, except: - the QR code is often deep in a refrigerated display case, so you're holding the door open with one hand while you stick your phone in there and the camera fogs up - there is rarely good enough cell signal in the store, even more rarely in the fridge - the web app for clipping is buggy as hell (yesterday I used it and clipping via the QR code failed on 4 out 4 coupons, and it was the worst kind of failure - "clipping" animation just plays forever while I was standing there blocking the supermarket aisle) This damn thing has been around for a few years and has changed shopping from fun and easy to frustrating. I might just go back to Amazon fresh or whatever. Better, I think I'll just write a script to hit all the coupon endpoints every day before I go shopping. Coupons were always a stupid, user-hostile game, but this system is taking it several steps too far. Also, it's discriminatory against seniors and people who don't have smartphones (or don't want to have to have them out the whole time while shopping).
hardware2win 3y ago
A lot of fancy links referenced, yet as usual it is really simple Desire to get shit done and motivation is more important than anything else
- javier123454321 3y ago
  Sure, but the issue was entrenched in the system. Encouraging autonomy and ownership seems a difficult task in a large scale org.
penjelly 3y ago
> Ask for more information. very often this is part of the solution, asking for info isnt deflection in most cases. Usually the worst bugs are a result of some unknown unknown or excessive complexity in code. Discussion is often the best way out imo
grumpy_coder 3y ago
At a meta level in most companies it is important to not become the quick 'bug fixing' specialist. Don't be the developer product managers use/abuse to get around processes and fix what they want fixing today. Work with your manager to plan your career path. 'I fixed a bug the other day' is good once a year as a bullet point on 'why you should get that promotion' nothing more. 'I built and led a team to fix long standing bugs' is more what you want to be doing.
- _justinfunk 3y ago
  I don't think I agree. In my experience, being the guy or gal that managers know they can turn to in order to quickly solve problems is a great way to get noticed and get involved in projects that normally you wouldn't be able to.
ot1138 3y ago
Just happened to me with MongoDB. I don't mean the open source project, I mean the public company that sells premium SaaS hosted MongoDB (Atlas) instances along with their charts product. I found a very serious bug that was trivially reproducible in my charts account (valid Mongo data being corrupted and displaying non-sensical data in charts). No way to report a bug, so I reached out through chat. Customer Support started playing the telephone game, asking me for various things every 2-3 weeks or so. If I didn't get back to them in 24 hours, I then got a warning that the bug would be closed out in another day. This went on for several months. I sent them screenshot after screenshot along with the original mongoDB documents which showed very clearly they had a bug. Perhaps the fourth or fifth time that they asked me for some heavy lifting type deliverables, I didn't have time. They simply just closed the ticket. Oh - near the end of that time, someone on their sales team cold-called me to try and upsell me on annual support contract. Yeah, right. Anyway, the story has a happy ending. After getting the runaround for a few weeks, I created a Jupyter notebook that displayed the necessary chart I needed. It ended up being a lot more useful than the generic graphs I could pull from Atlas anyway.
- pdonis 3y ago
  > near the end of that time, someone on their sales team cold-called me to try and upsell me on annual support contract. Yeah, right. Did you tell the sales person why you were not interested in the support contract?
- chris_wot 3y ago
  It’s not just bug reports, companies won’t even tell you basic information about their services. Yesterday I hopped into online chat because I needed to find out the servers to add to our firewall exception list in order to prevent Spotify from playing. (We saw at least one web socket being blocked, but we needed to know what servers to unblock, we aren’t going to play whack-a-mole constantly) They wouldn’t tell me. Eventually I asked Spotify to escalate to a supervisor, at least I did when they told me the only solution was to turn off our firewall! The supervisor started asking what firewall we were using, whether we checked the firewall vendor’s community forums, etc. I eventually said “look, to me this is not great. I work at a school with over 1,100 students, many of whom are paying for a Spotify subscription. We are happy to allow them to use your service and we are happy to add Spotify servers to our exception list, we just need to know what they are. If you decide not to tell us, we don’t mind - from my POV I’ve done my best to reach out to you and you have decided not to give me the information we need to allow your service. I will therefore just tell all 1,100+ students that we reached out to Spotify but you guys refuse to tell us how to unblock your service. I’ll even show them this chat log. We aren’t going to disable our firewall and so I will just advise students to use a different paid service and that - until Spotify give us the basic info we need - Spotify won’t work.” If a company like Spotify won’t even tell you what servers they use so we can allow access to them, then it’s no wonder stupid bugs like the one OP describes come about for years. I mean, Spotify is about to likely lose hundreds of subscriptions soon, even though we would prefer they wouldn’t, all because of some stupid policy that say that the servers they use to serve content to end users can’t be provided to network admins who want to allow their end users unfettered access to a service they pay for! Truly, big companies are stupid.
w10-1 3y ago
This isn't about fixing a bug, but about working for the customer when the organization can't. For theoretical background on why companies grow in this manner, read the transaction cost economics literature on bureaucratic cost. That's what drove the auto industry to abandon the hierarchy for the M-form, with some internal contracting replacing hierarchy. It's why US companies are relatively flat. In this case, everyone actually wanted the bug fixed. Problems really arise when some do and some don't, and when it's in the customer's interest but not the company's. The solution here is not to fix the bug but fix the problem with test infrastructure. It's causing a whole class of hard-to-reproduce bugs to be ignored. In a bureaucracy, the bug will get fixed but the problem will persist. In a healthy company, middle management will work together to fix the problem.
gdss 3y ago
on the cybernetic systems thing: take a look at a book named "brain of the firm" by stafford beer
- javier123454321 3y ago
  This is great, though seemingly difficult to come by (most books online seem to be anywhere from ($100 to $400 usd). Added Stafford Beer to my people to learn from list, thank you.
cratermoon 3y ago
The article discusses how, when companies reach a certain size, tension arises "between owning your work and standing out, vs just doing the minimum". Plenty of folks on HN abhor doing the minimum and rightly so, but rarely is there discussion of the forces pushing programmers towards doing the minimum when they'd prefer to own their work. At one place I consulted, there was an engineering manager who firmly pushed team members to own their work, at least verbally. Conflictingly, her idea of owning the work was to do it exactly as the organization and leadership directed, without complaint, and never raising any issues about how and what was being done. How does a programmer own the work and stand out when the incentives reward conformity but punish going outside the lines?
arein3 3y ago
Crazy that other engineers are so ignorant/bad.

This sounds like a monitoring problem. If it's happening in production there isn't a requirement to figure out how to reproduce it locally.

sosodev 3y ago
You need to reproduce the bug to know if your fix works. Not always but that’s how most teams operate. Particularly because you typically won’t pass review without some tangible improvement.
- javier123454321 3y ago
  Yeah, we actually needed to 'test' in production. I wrote tests for how it was supposed to work, and made the best effort to reduce the risk, but the only way to get a reproducible environment for this particular issue was to deploy the change. That definitely contributed, though was not exclusively the cause of, the risk aversion.

mypastself 3y ago

Did the author go behind his manager’s back to pick up the task, or was it assigned? Because if it’s the former, it’s possible that the issue was not addressed because it wasn’t deemed high-priority compared to other tasks, regardless of the number of user complaints. If it’s the latter, well, I don’t know. Congratulations on completing an assigned task at your job?

withinboredom 3y ago
Seemed like it was a case of "cannot reproduce" and any attempt at reproducing it failed. Since it couldn't be reproduced, it was never prioritized because it never made it past triage. At least that's the workflow I imagine when reading this and from working at similar places. Also, I advise leaving a place that requires your manager to dictate which tickets you work on. Sounds like a crappy place to work.
- LambdaComplex 3y ago
  > I advise leaving a place that requires your manager to dictate which tickets you work on As someone who recently left a place like that: I agree. I'm a highly-paid professional; it was really frustrating to feel like my manager didn't trust me enough to figure out what I needed to work on.
  - bryanrasmussen 3y ago
    It really depends on what you are shipping: 1. does what you are shipping have lots of international regulations and compliance requirements per country it will be available in with new countries being rolled out on a planned basis -> your manager will determine what you should work on 2. Are there things being released on a set date due to some sort of legal requirement, business deal? -> your manager will determine what you should work on 3. Is it basically the same product in all markets / countries other than internationalized text with not pending contracts or business deals requiring specific functionalities impacting the company? You should be able to determine what you should be working on.
    - withinboredom 3y ago
      I disagree. Though (1) and (2) are good points, but they shouldn't be telling you what to work on at the individual ticket level. Even then, they should prioritize and set a healthy deadline (if appropriate); and that is it. Implementing a high priority ticket may still require addressing technical debt to implement, or dealing with other lower priority tickets first -- because they block the new, higher priority ticket. If your manager needs to know all these details and can tell you "no, you can't fix X to build Y" then I really think that is a crappy place to work; not just due to the politics involved, but also the code must be absolute crap.
  - maccard 3y ago
    Manager/Tech Lead here who had someone who would choose their tickets themselves. It's a giant pain and makes my job 10x more difficult when I tell another team that something that's low priority to us but high priority to them that something will be done, and instead find it sitting in in-progress with a tech debt bugbear of theirs in review. The priority on the ticket was set for a reason, it was assigned and scheduled for a reason. There's only so many hours in the day to explain why some things are prioritised. If you think that's wrong, be a professional and talk to me about it. I promise you, I want to have the conversation that you need to actually work with your team far less than you want to fix that JSON output.
LambdaComplex 3y ago
The author says "I had never looked at this code, but was told to take a look," so I assume they weren't doing this behind their manager's back.
javier123454321 3y ago
The manager told me something along the lines of "hey this has been causing issues, can you take a look? Others haven't been able to figure this out." It's the latter part of the statement that I was musing over in the article. I'll take the sarcastic congrats, though, better than internet name calling.
- mypastself 3y ago
  Must have missed that part. But my intention wasn’t to offend. It felt like the part about solving a problem other developers couldn’t was the main crux of the article, and the insight about inertia in big companies was tacked on. But I’m sure others will disagree.
cratermoon 3y ago
I read it as saying everybody else gave excuses to get out of dealing with it while the author went ahead and did it when assigned, even though all the previous people had said it couldn't be fixed.
dontupvoteme 3y ago
I don't work in a software company so I found it interesting to see how things can become dysfunctional there.

devjab 3y ago

I’ve worked quite a bit in enterprise software development, and later in teams that are transitioning from startups/mid-size to enterprise and in my very anecdotal experience it’s always down to the internal processes. Software developers like solving issues. Most people, myself included prefer building new things, but a lot of us actually like solving bugs as well. At least that is my experience. What I’ve seen get in the way is when the processes around reporting, examining, describing and prioritising the “tasks” get so convoluted that that developers get either overwhelmed or detached from the issue by too much bureaucracy. Which happens very often in non-tech organisations in my experience. Maybe it also happens on tech organisations but I’ve never worked in one, but from buying software from around two hundred it didn’t seem like it was a problem in quite the same way it can in regular enterprise. I realise this may be a little more clear in my head than for you as the reader. So I’ll try to explain by an example I encountered recently with a parking app. My wife and I were visiting a midwife for a comfort scan, and they have you register your license plate in an app so that their patrolmen/parking-guards/whatever they are called don’t give you a ticket when they scan your car. Anyway, the app wasn’t working and the midwife was like: “oh, that happens from time to time, don’t worry that means they can’t check if parking are legal or not”. As though it was the most normal thing in the world. Because it is… IT failing is the most normal thing in the world to a lot of people, but think about it. That app being down, at least on a city scale maybe even global meant that all those parking-guards were paid to do nothing for however long it took to fix it. It also meant anyone could essentially park for free… and that was a semi-regular thing? When you work in enterprise organisations. They often become accustomed to IT not working. Part of this is because the process people they put in between IT professionals and their organisation don’t actually know IT. They are very good at defining processes, making things lean and what not, but that’s not how you actually make things work in my experience. I’m guessing this is true outside of IT as well, but once you have more process managers, scrum masters, product owners and what not than actual technicians, then you’ve probably ducked it up. This is because programmers are often the best people at determining what’s important, not always, but often, and despite an entire industry’s attempts at “streamlining” the development process I suspect it’s also why sometimes a team of 4 developers can do more quality work in a shorter amount of time than 10 teams of 10 people combined. It sounds like the author has encountered some of this. They have an important issue. It loses some of its importance on the processes and then once it finally gets to the technicians it can’t be reproduced… Of course it doesn’t get solved. Especially if the organisation has set up systems to reward not solving seemingly unimportant reproducible issues, which many enterprise organisations end up doing. Corporate culture is hard.

samhuk 3y ago

> Deny that there is a problem. > Deny that it is your problem. > Ask for more information. > Complain. Oh my does this ever bring traumatic memories! This entire article reminds me of a previous company I used to work for. There was a sufficiently large number of engineers that had a similar, if not more elaborate, deliberate, and sinister way of responding to bugs: * Deny {bug} is a problem - "I don't see how this affects you/anyone?" * Deny {bug} can even be fixed - "Ok, fair, but I don't see how this can be fixed?" * Deny {bug} is their problem - "Ok, fair, but have you tried asking {other employee?}, they are responsible for this." (they are obviously not) * Be rude - "If you think this is a problem, then go fix it" All this was just obvious defensive walls of lies to protect their reputation and hide their lack of skill and their laziness. Around 5% of the engineers were like this and I've always analogized it to the proportion of U238 in a sample of Uranium - just enough enriched dogshit to cause a critical mass of misery. I left (for greener pastures) shortly after I had a couple of tasks requiring interaction with these "engineers", which made working around them impossible, and then realizing that leadership had totally lost control of the zoo and didn't care one iota.

withinboredom 3y ago
I very recently had to push back on a bug that wasn’t a bug in my part of the stack. If they wanted it “fixed” in my part, it would be a feature. (Enum values from an API being localized — these were hardcoded on the backend). IMHO, these should be localized in the client, which has the current locale (I do not have the current locale and no strings are translated from this API by this API). That was pretty fun, but I don’t think generalizing “denying a bug” as saying we are being “lazy” is a correct assumption.
- samhuk 3y ago
  I agree, "denying a bug" does not equal "lazy", but rather "denying a bug one has created" does at best equal "lazy", at worst malicious (e.g. rep protection).
- dkarl 3y ago
  I had a similar experience adding new functionality to a web application. It was a simple enough feature that I was able to supply the English myself, but I knew enough not to provide the international copy, even though the only other language we supported was a language I studied in school. I asked the product manager a few times, then created a ticket assigned to them and sent them the link. Meanwhile, my code got reviewed and deployed, behind a feature flag of course. As soon as they found out the code was deployed, the product manager asked if we could turn on the feature. I reminded them that the reason we supported the other language was that it was a legal requirement for our customers in a certain country. They said turn in on, nobody was going to notice except a big US customer who desperately needed it. So I did. Within a few hours we had an enraged customer on the phone. Credit to the product manager, they took full responsibility, but the initial response from the founder/head of customer success who took the call was to walk straight to my desk and ask me what the hell I did.

altacc 3y ago

> What I did is come in as an outsider [and] let my naivety guide my investigation... I call this being "a useful idiot", which I sometimes use to describe my role. A large part of my job is floating between teams & applications, identifying gaps and problems and knowing just enough to ask meaningful questions whilst not being deeply invested in the existing code & processes. I can see people asking themselves "why are they wasting my time even asking these questions?!" but in explaining the answer they get to review the process and sometimes work out a new solution. We all get comfortable with how things are and someone prodding us occasionally helps creativity.

mbork_pl 3y ago
Nice, but nit: this is not what a "useful idiot" usually means.
- vxNsr 3y ago
  It’s an ironic use of the term.
- altacc 3y ago
  I live in a country where English is not the native language and the idiom is not in common usage, so the literal translation works. Sometimes I often wonder if it would be useful for more internet forums to allow users to add a flair to denote they are non US, as the assumption on so many sites is that everyone is US and living on the east or west coast.
  - deleted 3y ago
    [deleted]
  - mbork_pl 3y ago
    I think I agree in general, but in this case he "idiom" (in fact, more of a metaphor, I guess) is actually Russian and just translated literally into English.

roydivision 3y ago

The satisfaction of finding the root cause of something like this and then fixing it is great, and one of the reasons I’m still in this industry. The great write up reminds me of a few similar cases I’ve dealt with over my career, I still think back to them and smile to myself.

motbus3 3y ago

In my company we have a bug sitting in the to do list for about a year. The root cause was a previous change in requirements that was not well defined and explored a couple of years ago. None of the original developers are working for the company anymore. We offered ourselves to fix that bug, but it would take at least a couple of weeks. This bug prevents us to roll out other deemed important features. But, it is just a bug. No product owner wants it to say to his boss he spent 2w of an engineer solving an issue instead of building another feature. The boss of the boss called. He says that this bug is not tolerable. He ask us to fix it. We say we agree and we already have screened and have a solution for it, but it will take 2 weeks. He says he wants that earlier and we say no because no one wants to rush it as we don't want to cause a possible side effect if we fail in something. The solution according to the is to not fail. Everybody goes back to the planning and no one picks that issue because the condition is to do it in half of time without fail while working on a new feature. Go back to line 1.

Etheryte 3y ago
Ah yes, the classic low-skill management maneuver of setting an unrealistic deadline in the hopes that people will then work faster. Naturally, Joel has written about this [0], see point 12. [0] https://www.joelonsoftware.com/2000/03/29/painless-software-schedules/ https://www.joelonsoftware.com/2000/03/29/painless-software-...
- gberger 3y ago
  > 8) Put in line items for Vacations, Holidays, etc. If your schedule is going to take about a year, each programmer will probably take 10 to 15 days of vacation. That's so little vacation. America is wild.
  - Ryoung27 3y ago
    Wild is a nice way of saying crushing my soul. I've worked at companies that give you 10 days, that they choose, for your first five years. I have no clue why we do this to ourselves other than the person in charge is not bound by it, and one day we all hope to be in charge.

alentred 3y ago

Congrats for the bug fixed! I know how this kind of bug can be really hard. For a company. Less so for "an outsider", just like the author suggests. I can easily see this happening in literally any company. We humans are very adaptable creatures, we get accustomed to things and just accept them as they are. In my company I preach for razing the flags and looking for things that just don't look right, and we generally do so indeed, but I am pretty sure that for an outsider some acceptable "features" will look very questionable. Rule of thumb I use sometimes: does it feel awkward to answer some of the newcomers’ questions? Independently of their experience, newcomers always ask very good questions. -- Serenity Prayer comes to mind: > God, give me the serenity to accept the things I cannot change, > Courage to change the things I can, > and Wisdom to know the difference.

Loughla 3y ago
I have the full version of the serenity prayer as a cross stitch framed in my office. >God, grant me the serenity to accept the things I cannot change, >The courage to change the things I am able, >The wisdom to know the difference, >And the ability to destroy all of the evidence of the inevitable murder, should the first three fail.
jimmaswell 3y ago
> I know how this kind of bug can be really hard. It sounds more like it was embarrassingly easy, but the author is surrounded by 0.1x engineers.

perlgeek 3y ago

So, people have blamed the vendor's API... but have they opened a ticket with them, raised the issue, escalated it? I can emphasize with not wanting to work around the bugs of an upstream provider, but just sitting on it when it causes real pain to customers is also shitty. > That's right, for any developer, the answer here is simple: use the coupons that we are ALREADY receiving on the frontend to populate the fields that users expect so they can get their discounts. Hard to tell without knowing more details, but that might run into the risk that people can inject made-up discounts in the browser and send them to the backend, which then charges less.

beezlewax 3y ago
Sounds like that to me too. I'm sure there is more going on there though
doix 3y ago
> Hard to tell without knowing more details, but that might run into the risk that people can inject made-up discounts in the browser and send them to the backend, which then charges less. That was my first thought, but thinking about it more, it probably makes no difference. If it's susceptible to injected made-up discounts now, it would have been susceptible before as well. It doesn't matter if the coupon code comes from the vendor API or their own API if you're going to inject fake data anyway.
javier123454321 3y ago
I'm not privy to the extent of the conversations with the vendor, I came in with fresh eyes and found a solution. The writeup was just musing on the process that let it get to that point from my perspective. Your final issue is not a risk in this case, because we still are validating the coupons. This was a frontend fix, and we obviously don't trust any data coming from the frontend.
Nemi 3y ago
Here’s the deal, at two years and for a problem that is by all accounts serious, any solution is better than no solution. Now, if this is code that runs ventilators for sick people, runs the movement of money for a hedge fund, or balances the US budget, then no, you have to be absolutely, positively sure that you are not causing another problem. But when you have a real problem that is affecting people everyday for years, then a little creativity is certainly called for. Even at the expense of hypothetically causing another problem.

eliac 3y ago

> Harvard and others have written about it Why Big Companies Can't Innovate. I am interested in this, because I am interested in doing work, at scale, within a company that has a culture of solving problems, not of being a behemoth with enough inertia to not need to correct when there is a need. You might like "Loonshots" by Safi Bahcall. The book proposes to solve this by nurturing an innovation unit inside the behemoth, and carefully maintaining the balance between the two.

deleted 3y ago

[deleted]

globalise83 3y ago

Escalating this issue to a senior director and have them call up the CEO of the marketing partner and threaten to switch providers if they don't resolve the coupon bug within 48 hours would have solved this problem a long time ago.

duckmysick 3y ago

I don't understand the problem. Or rather the design of their system. > The issue was that our system for weekly ad was having an issue where the ads needed to 'clip' a coupon (read hit an api) for the discounted price to reach our cart calculations. However, the api that we used for managing the weekly ads was managed by an external vendor, and often when we queried their endpoint, the coupon field was returning empty when it shouldn't. What's the authoritative source if I want to know that a product has a discount? The external vendor? If the coupon field is empty, doesn't it mean there's no discount? Unless it means there are two different fields: one for is_there_a_discount and another for coupon_id. Would there ever be an expected situation where there is a discount but not a coupon? Or maybe the source is the internal system. In which case, can't the coupon be generated there? Surely the source is not the frontend, right? > Products already had the associated coupons. From where? The source aside, does that mean the coupon_id never change or expire? If the same product has a different discount a few months from now, will it still use the same coupon? Also, the apparent solution was hardcoding: > so I completely hacked it, and replaced the api call service with a dumb function that returned hardcoded data copied from a production. but a couple paragraphs before, the author mentioned it changes often: > Because this data was missing a field, it required manual data entry to update it. However, this data changed in real time, so it was OFTEN out of date. I don't understand how this was a showstopper before but isn't anymore. Or maybe these paragraphs talk about different things. I don't question the bug fix or the author. I just can't grok it.

KnobbleMcKnees 3y ago
I believe the part about hard coding production data was referring to recreation of the bug, not implementing the solution which the author describes subsequently as "a couple of array functions". There's a fair bit of hand waving going on with technical details throughout but I don't think it's badly intended, more aimed at keeping the focus on the organisational issues described.
javier123454321 3y ago
Hey, Author here. The source was our internal system, but the vendor was taking our id, converting it to their own id, and then serving us an ad with the coupon labeled formatted to their system. We were then using the vendor id to translate it back to ours. The coupons associated with the products were already from our system, so I could skip the entire translation back to our id. The manual overrides bit can be omitted, it's just how people in product were fighting a loosing battle of manually overriding the bad data. The hardcoding was in order to replicate the behavior, as it wasn't showing up in lower envs. It wasn't the fix, the fix was to just render data we already had. I didn't go too much into details because 1. I can't due to company policy, and 2. it's actually not the interesting bit. I was more making observation of the process.

foobarbecue 3y ago

Oh my god, I absolutely hate the torture system this person works for. Pretty sure they are talking about the thing at Ralph's where you have to "clip" a coupon by scanning a QR code in the store, except: - the QR code is often deep in a refrigerated display case, so you're holding the door open with one hand while you stick your phone in there and the camera fogs up - there is rarely good enough cell signal in the store, even more rarely in the fridge - the web app for clipping is buggy as hell (yesterday I used it and clipping via the QR code failed on 4 out 4 coupons, and it was the worst kind of failure - "clipping" animation just plays forever while I was standing there blocking the supermarket aisle) This damn thing has been around for a few years and has changed shopping from fun and easy to frustrating. I might just go back to Amazon fresh or whatever. Better, I think I'll just write a script to hit all the coupon endpoints every day before I go shopping. Coupons were always a stupid, user-hostile game, but this system is taking it several steps too far. Also, it's discriminatory against seniors and people who don't have smartphones (or don't want to have to have them out the whole time while shopping).

hardware2win 3y ago

A lot of fancy links referenced, yet as usual it is really simple Desire to get shit done and motivation is more important than anything else

javier123454321 3y ago
Sure, but the issue was entrenched in the system. Encouraging autonomy and ownership seems a difficult task in a large scale org.

penjelly 3y ago

> Ask for more information. very often this is part of the solution, asking for info isnt deflection in most cases. Usually the worst bugs are a result of some unknown unknown or excessive complexity in code. Discussion is often the best way out imo

grumpy_coder 3y ago

At a meta level in most companies it is important to not become the quick 'bug fixing' specialist. Don't be the developer product managers use/abuse to get around processes and fix what they want fixing today. Work with your manager to plan your career path. 'I fixed a bug the other day' is good once a year as a bullet point on 'why you should get that promotion' nothing more. 'I built and led a team to fix long standing bugs' is more what you want to be doing.

_justinfunk 3y ago
I don't think I agree. In my experience, being the guy or gal that managers know they can turn to in order to quickly solve problems is a great way to get noticed and get involved in projects that normally you wouldn't be able to.

ot1138 3y ago

Just happened to me with MongoDB. I don't mean the open source project, I mean the public company that sells premium SaaS hosted MongoDB (Atlas) instances along with their charts product. I found a very serious bug that was trivially reproducible in my charts account (valid Mongo data being corrupted and displaying non-sensical data in charts). No way to report a bug, so I reached out through chat. Customer Support started playing the telephone game, asking me for various things every 2-3 weeks or so. If I didn't get back to them in 24 hours, I then got a warning that the bug would be closed out in another day. This went on for several months. I sent them screenshot after screenshot along with the original mongoDB documents which showed very clearly they had a bug. Perhaps the fourth or fifth time that they asked me for some heavy lifting type deliverables, I didn't have time. They simply just closed the ticket. Oh - near the end of that time, someone on their sales team cold-called me to try and upsell me on annual support contract. Yeah, right. Anyway, the story has a happy ending. After getting the runaround for a few weeks, I created a Jupyter notebook that displayed the necessary chart I needed. It ended up being a lot more useful than the generic graphs I could pull from Atlas anyway.

pdonis 3y ago
> near the end of that time, someone on their sales team cold-called me to try and upsell me on annual support contract. Yeah, right. Did you tell the sales person why you were not interested in the support contract?
chris_wot 3y ago
It’s not just bug reports, companies won’t even tell you basic information about their services. Yesterday I hopped into online chat because I needed to find out the servers to add to our firewall exception list in order to prevent Spotify from playing. (We saw at least one web socket being blocked, but we needed to know what servers to unblock, we aren’t going to play whack-a-mole constantly) They wouldn’t tell me. Eventually I asked Spotify to escalate to a supervisor, at least I did when they told me the only solution was to turn off our firewall! The supervisor started asking what firewall we were using, whether we checked the firewall vendor’s community forums, etc. I eventually said “look, to me this is not great. I work at a school with over 1,100 students, many of whom are paying for a Spotify subscription. We are happy to allow them to use your service and we are happy to add Spotify servers to our exception list, we just need to know what they are. If you decide not to tell us, we don’t mind - from my POV I’ve done my best to reach out to you and you have decided not to give me the information we need to allow your service. I will therefore just tell all 1,100+ students that we reached out to Spotify but you guys refuse to tell us how to unblock your service. I’ll even show them this chat log. We aren’t going to disable our firewall and so I will just advise students to use a different paid service and that - until Spotify give us the basic info we need - Spotify won’t work.” If a company like Spotify won’t even tell you what servers they use so we can allow access to them, then it’s no wonder stupid bugs like the one OP describes come about for years. I mean, Spotify is about to likely lose hundreds of subscriptions soon, even though we would prefer they wouldn’t, all because of some stupid policy that say that the servers they use to serve content to end users can’t be provided to network admins who want to allow their end users unfettered access to a service they pay for! Truly, big companies are stupid.

w10-1 3y ago

This isn't about fixing a bug, but about working for the customer when the organization can't. For theoretical background on why companies grow in this manner, read the transaction cost economics literature on bureaucratic cost. That's what drove the auto industry to abandon the hierarchy for the M-form, with some internal contracting replacing hierarchy. It's why US companies are relatively flat. In this case, everyone actually wanted the bug fixed. Problems really arise when some do and some don't, and when it's in the customer's interest but not the company's. The solution here is not to fix the bug but fix the problem with test infrastructure. It's causing a whole class of hard-to-reproduce bugs to be ignored. In a bureaucracy, the bug will get fixed but the problem will persist. In a healthy company, middle management will work together to fix the problem.

gdss 3y ago

on the cybernetic systems thing: take a look at a book named "brain of the firm" by stafford beer

javier123454321 3y ago
This is great, though seemingly difficult to come by (most books online seem to be anywhere from ($100 to $400 usd). Added Stafford Beer to my people to learn from list, thank you.

cratermoon 3y ago

The article discusses how, when companies reach a certain size, tension arises "between owning your work and standing out, vs just doing the minimum". Plenty of folks on HN abhor doing the minimum and rightly so, but rarely is there discussion of the forces pushing programmers towards doing the minimum when they'd prefer to own their work. At one place I consulted, there was an engineering manager who firmly pushed team members to own their work, at least verbally. Conflictingly, her idea of owning the work was to do it exactly as the organization and leadership directed, without complaint, and never raising any issues about how and what was being done. How does a programmer own the work and stand out when the incentives reward conformity but punish going outside the lines?

arein3 3y ago

Crazy that other engineers are so ignorant/bad.