36 ms·

I accidentally saved my company half a million dollars

• j0hnyl 3y ago
  Once upon a time I uncovered a bug that recovered $4MM/year in revenue. It was swept under the rug to protect the team and executives that let the blunder continue for as long as it did. I didn't get a raise, but I made some allies and got to coast for a while.
  • chasd00 3y ago
    I did something similar early in my career but it very much went noticed. The system I wrote to correct the bug was basically a MITM attack against pharmacy rx transmissions, it re-priced prescriptions just before they went to the insurance carrier's system because the pharmacies would never apply price updates (i worked at a small independent pharmacy chain, they didn't have a central dispensing system). In my infinite genius I named the system after my current crush. Corporate liked it so much they made an annual award named after my system... so therefore after my crush but i was too embarrassed to tell them the real origin of the name. That was 25 years ago but, to this day, every year a trophy gets made with the first name of my old crush printed on it and handed out. heh if she ever knew she'd be mortified.
    • WD40forRust 3y ago
      You must tell her then update us on her reaction. Bonus if you record it on video. :^)
  • busterarm 3y ago
    Very early in my career I discovered some mission-critical network devices at the hedge fund I worked at that would have caused a network loop if either of the two machines were ever rebooted. The two servers were related to some feed data and I had noticed that they were cabled together oddly based on how their function was described to me. The estimated cost of downtime for these systems was something like $7 million per minute. I had raised the issue to a couple of the staff responsible for the machines and to the networking team but was completely dismissed because "there is no way we would have hooked them up that way" and because I was the FNG. I then raised the issue again at the weekly group meeting because it seemed important -- somebody was dispatched to check visually and came back to confirm what I said. It was a big deal -- the networking team had about 2 weeks of emergency work to do to resolve the issue cleanly. EVERYONE was angry at me. Even though I had just averted a catastrophe for the company, I made everyone look bad by doing it and particularly because of my status/position on the team. It was an important lesson learned.
    • tehlike 3y ago
      Keep doing the right thing.
      • busterarm 3y ago
        It's 20 years later and I've made my career off of pointing out fundamental mistakes made by very smart people. ;) (just with more skill and tact) 90% of what I do is ask dumb questions as if I'm completely clueless.
        • tehlike 3y ago
          I know the feeling. I'm similar in that regard. Debugging complex things by asking dumb questions is extremely fruitful. Also asking dumb questions to newer engs teaches them to think different aspects of a problem themselves. One of the things I'm hoping I could keep alive in my child
    • phkahler 3y ago
      >> EVERYONE was angry at me. Even though I had just averted a catastrophe for the company, I made everyone look bad by doing it and particularly because of my status/position on the team. People can be excessive in both taking credit and placing blame. An appropriate and helpful way to frame this is "the system was configured incorrectly but nobody noticed because the problem never actually happened. It's a good thing the new guy had time to go through things and spotted the problem before it ever happened." No need to crucify the team or exaggerate the value of the new guy.
    • mablopoule 3y ago
      On his book "The Secrets of Consulting", Gerald Weinberg advise against improving more than 10% of performance, and if so, of hoping to have any credit. Just like the article, his reasoning is that if you improve performance too much, it makes management/the team look bad for not doing it before, while a smaller improvement in performance make management looks good.
      • 3y ago
• pavel_lishin 3y ago
  This person is a good writer, I'm looking forward to seeing what else they've written.
  • bcjordan 3y ago
    It is really what you want out of a hacker-stuck-in-corporate story. There was a tale maybe 10+ years ago about someone who automated their job with a script or Excel sheet or macro and didn't tell anyone about it. Having a hard time tracking it down again, anyone remember what that was?
    • pavel_lishin 3y ago
      Google reveals two things, followed by a bunch of shitty blogspam: - Last year, there was a post on Reddit, with an HN discussion as well: https://news.ycombinator.com/item?id=29994776 https://news.ycombinator.com/item?id=29994776 - In 2017, there was a question about whether someone should tell their employer that they've automated their job: https://workplace.stackexchange.com/questions/93696/is-it-unethical-for-me-to-not-tell-my-employer-i-ve-automated-my-job https://workplace.stackexchange.com/questions/93696/is-it-un...
    • orzig 3y ago
      I’ve found LLMs to be good for those sorts of vague “remember that thing like this?” searches. Bing (with GPT) or even Bard with potentially better search integration are both worth a try.
• davedx 3y ago
  Incredible. Meanwhile big tech thinks the way to reduce costs is to wholesale fire 1/2 their company.
  • pavel_lishin 3y ago
    For what it's worth, OP also recommended that approach.
    • mcv 3y ago
      Yeah, but corporations rarely fire only the incompetent people. Quite often it's the best paid engineers who get fired first because that saves the most money. And when there's a voluntary get-out program, it's the unappreciated competent people who are the most eager to leave.
  • goalieca 3y ago
    The two are not exclusive. Companies can run hundreds of programs many of which are not productive and actually hurt and frustrate engineering teams. Just in the same way government can hurt the environment by throwing up a lot of red tape on green initiatives, companies can hurt quality and productivity by spending thousands of hours on things like employee annual reviews.
• alex7734 3y ago
  In a sane world this would have to be satire, but unfortunately I believe every single word of this
  • tgv 3y ago
    A writer cannot make up so much shit. You need a board of managers for that. Effing hell.
• vwoolf 3y ago
  This reminds me of some of Dan Luu's stories, https://danluu.com/nothing-works/ https://danluu.com/nothing-works/ Likewise with chip software tooling; despite it being standard to outsource tooling to large EDA vendors, we got a lot of mileage out using our own custom tools, generally created or maintained by one person, e.g., while I was there, most simulator cycles were run on a custom simulator that was maintained by one person, which saved millions a year in simulator costs (standard pricing for a simulator at the time was a few thousand dollars per license per year and we had a farm of about a thousand simulation machines). You might think that, if a single person can create or maintain a tool that's worth millions of dollars a year to the company, our competitors would do the same thing, just like you might think that if you can ship faster and at a lower cost by hiring a person who knows how to crack a wafer open, our competitors would do that, but they mostly didn't.
  • TehShrike 3y ago
    I think about that, and his culture post ( https://danluu.com/culture/ https://danluu.com/culture/ ) a lot. There are some companies that represent the opposite of what OP describes. Once you work with one of them, I've found it impossible to settle for one of the bad places.
  • pjc50 3y ago
    Have worked in EDA and .. yeah, the software is bad but the buyers are conservative, precisely because it's a risky and expensive business. He talks about a "cocktail party version of the EMH". The credentialed econ version of "nothing works" is https://en.wikipedia.org/wiki/The_Market_for_Lemons https://en.wikipedia.org/wiki/The_Market_for_Lemons , which discusses the role of information and its asymmetry in markets. The EMH fails precisely because total knowledge is impossible and adverse selection is real.
• cybernoodles 3y ago
  I saved Amazon $10MM as an intern back in 2012. If only I could have seen 1% of that.
  • shepherdjerred 3y ago
    I also saved Amazon a ton of money and saw nothing from it!
    • gottorf 3y ago
      On the other hand, employees aren't liable for the company's losses and debts, so it works out in the end.
      • TuringNYC 3y ago
        In trading/PM roles in finance, before clawbacks became more popular, employees would regularly get the upside and avoid the downside.
        • searchableguy 3y ago
          Are clawbacks common at trading companies?
      • foxyv 3y ago
        > employees aren't liable for the company's losses and debts Never been laid off during a recession or had your pay frozen and bonuses cancelled during a hard time? Employees risk a lot more than most stock holders by working for a company. On average, stock holders are way more diversified.
        • throwaway98797 3y ago
          losing your job is not the same as losing capital
          • 3y ago
• rybosworld 3y ago
  TLDR: Bureaucracy is everywhere.
• jack_riminton 3y ago
  "this whole department, like many departments, is some sort of weird political PsyOp to get executives promoted. It's cosplaying as a real business and the board thinks the costume is convincing." Came for the engineering, stayed for the blisteringly on-point observations of corporate life
  • andrewstuart2 3y ago
    At my previous $JOB I started calling this Promotion-Driven Development because it's everywhere. Take the easy problems that look good and you can finish quickly, get promoted, hand off the facade to team that has to actually solve the real problem, and repeat.
    • dmitrygr 3y ago
      This is very much how Google is on the inside. And that’s not surprising. People respond to the incentives presented. If you spend years fixing hard problems that actually affect users, and don’t get promoted, you’ll almost certainly become jaded, and start doing things that are pointless but do get you promoted. Because that’s how you make more money. And that’s the actual reason to be working.
      • andrewstuart2 3y ago
        And yeah, it shows via ye olde Google graveyard. Honestly I do think it's just a hard problem not to have, because the world tends to work on visibility combined with innovation. You can't solve hard problems and make money if nobody uses your solution because they don't know it exists. But I struggle with the balance because yeah, I'm a bit jaded.
    • syntaxing 3y ago
      I think the crappy part is that it works most of the time.
    • shalmanese 3y ago
      The purpose of a system is what it does. All development is promotion driven development, different people can either choose to be clear eyed about this or not and, subsequently, good at it or not. Your job is the venn diagram intersection of an organized system of rewards and punishments from both inside and outside of the firm intersected with your own goals and aspirations over time. Anything not in this intersection is not a job, it's something else we like to delude ourselves into thinking is a job to preserve our egos. More specifically, your worry about how distorted incentives lead to poor quality software is not your job unless it is your job to worry about said incentives. People keep hoping to go work for one of the magically sane company where all the incentives are correct and everyone is doing the quote-unquote "right" thing at all times. Such a company does not exist because the very concept is incoherent, every system of possible incentives will come with a different set of tradeoffs and the art of operating a firm is to pick from amongst a bunch of shitty options for the least shitty one. People who do not accept this will be perpetually unhappy and bitter about how others obtain undeserved success because they simply did the unsporting thing of playing the game they were asked to play. You can either be one of those people or you can receive the radical acceptance of what a job is.
• neilv 3y ago
  > The entire thing is stitched together by spreadsheets that are parsed by Python, dropped into S3, parsed by Lambdas into more S3, the S3 files are picked up by MongoDB, then MongoDB records are passed by another Lambda into S3, the S3 files are pulled into Snowflake via Snowpipe, the new Snowflake data is pivoted by a Javascript stored procedure into a relational format... and that's how you edit someone's database access. That whole process is to upload like a 2KB CSV to a database that has people's database roles in it. Sometimes it's hard to distinguish resume-driven development from iterative-StackOverflow-driven development.
  • syntaxing 3y ago
    I’m genuinely curious what a unit test for something would look like.
    • jacquesm 3y ago
      Checksum on the resulting csv with a parallel implementation of the whole pipeline ;)
      • syntaxing 3y ago
        I don’t work with large databases so pardon my ignorance. Is there typically a “unit test” bucket you run it on or do you just put in test entries on a production bucket?
        • hightrix 3y ago
          We tend to only test what is being tested. So, most DB calls are mocked in our unit tests. For stored procs or other tests that need to be run on a DB, we use a test DB that is setup to mirror production. I'd bet there are a 100 different answers to your question though. This is the way we handle it.
        • jacquesm 3y ago
          Normally you'd fire up a separate environment, mock the process and see if it produces the expected results. By the time you put 'test entries in a production bucket' there are so many lines crossed that it likely won't end well even if the tests do pass.
• trealira 3y ago
  > I'm not sure what the original estimate was, but I think it was intended to cost something like 200K for a year of operations, but we were now close to a million dollars. > ... > I return to work the following Monday. I suspected that this would save a bunch of money, and guess what, our projected bill dropped from a million to half a million dollars, and everyone is losing their fucking minds. Wow, so they're still over budget by 300k dollars. This is a funny story, but the company sounds incompetent.
  • NoMoreNicksLeft 3y ago
    > This is a funny story, but the company sounds incompetent. I've seen "typical" spelled this way several times on Hacker News. Is it a British thing or something like that?
    • pinkmuffinere 3y ago
      lol I know HN isn't the most 'jokey' crowd, but I thought this was pretty funny.
      • trealira 3y ago
        Yeah, I thought it was funny too, and upvoted it, but I guess more people disliked it than liked it.
  • plagiarist 3y ago
    The author seems to agree with you, they are still wildly incompetent with their response to this.
• digging 3y ago
  This is such a good article. My skin was crawling. What really scared me is that I couldn't identify any of the issues raised in my own organization, even though we often run into similar, smaller-magnitude problems caused by a blindness to obvious mistakes. It makes me fear I, too, am blind to massive bleeding wounds. Here's hoping they actually don't exist.
• runamuck 3y ago
  Thank you for writing a hilarious and painfully true slice of office life.
• datadrivenangel 3y ago
  Why are corporations so allergic to competence?
  • yetihehe 3y ago
    Competent people in corporation could show how uncompetent the rest of corporation is, so they are found and eliminated before they can do too much damage to management.
    • SillyUsername 3y ago
      That last line isn't a joke. I've been on the receiving end of middle management because I've been able to fix things in the business that have always been broken but never got fixed until I worked on them. Management will claim it's their work, will give you just lip service, will not use your name higher up the hierarchy and will actively down play their own mistakes whilst blaming the rest of the department or developers (e.g. like building a project with wrong requirements they actually provided, then missing the deadlines because stakeholders demand changes). Where that has happened I quit and then they just go back to the status quo.
      • ikekkdcjkfke 3y ago
        Hominids in suits
    • prmoustache 3y ago
      They are not eliminated, they are assimilated. A company is like a chain, as strong as its weakest link. There is not much gain to have some links much stronger than the others. The famous "I've learnt my lesson" which means you really reached the same low level of incompetence as everyone but secretly thinking you are more competent than all the others. The funny thing is that among all those incompetent peoples / idiots mentionned there are probably smart people who just learnt their lesson years/months and adjusted to the weakest links in the chain.
      • 3y ago
• tehlike 3y ago
  The whole post is gold. - managers asked how it was possible we saved that much without help from them - asked to prepare slides - asked many times on how it happened - had to roll it out slowly to make it look like they did it over time incrementally vs one small toggle - asked for a raise due to impact and did not happen. Sir, for your sake, apply to a FAANG or something, you'll be at least taken care of better. Also please implement Twitter card metadata in your blog so it looks better on twitter :)
  • pavlov 3y ago
    > “please implement Twitter card metadata in your blog so it looks better on twitter” Didn’t Xitter recently remove the display of all external site metadata except the image?
    • tehlike 3y ago
      Yeah people use image interestingly these days though See, for example: https://twitter.com/simonw/status/1717768637799706922 https://twitter.com/simonw/status/1717768637799706922
  • herpdyderp 3y ago
    Maybe my friends are all just bad devs but I don't hear of people getting treated well at FAANG anymore. (Though I haven't heard anything about Netlifx recently.)
    • tehlike 3y ago
      Maybe they are busy working - after while layoffs everyone is working a bit more to say the truth
      • y-c-o-m-b 3y ago
        I'm at a FAANG, still getting treated like shit. The problems outlined in the article are 1000x worse in a FAANG and you'll have the same 99% of the people that have drank the proverbial kool-aid telling you everything is great and it's supposed to function like this. At least you get paid more for it though.
• world2vec 3y ago
  Like staring at a mirror, this post made me cringe with how relatable it was. Guess I have a whole blog to read now.
• ChrisMarshallNY 3y ago
  Too bad Scott Adams self-immolated. This is Dilbertland, at its finest.
• dudeinjapan 3y ago
  My goal is to make my company the opposite of this one.
• Groxx 3y ago
  >I am asked to write some PowerPoints, which include phrases like "a careful statistical analysis of user usage patterns indicated an opportunity to more effectively allocate resources", implying that nothing was wrong, we just needed to collect more data before deciding not to let the expensive machines idle all day. Yep. If you don't make them realize it was a Hard Problem™ that was only solved by their smart hiring, funding, and task-deciding, you might shatter their whole world view.
• cainxinth 3y ago
  TLDR: They changed the idle time settings for running queries in their Snowflake database. Originally, they were set to idle for 10 minutes after every query, but most queries only took about 2 seconds to run.
  • 0cf8612b2e1e 3y ago
    I know nothing of Snowflake pricing, but how big is their infrastructure that leaving it on for an extended amount of time could see that much savings? To put it another way, to rack up $500k spend for 30 days of constant provisioned time: $500k/(30*24) = $694/hour Presumably, with the 10 minute blocks, there was idle time where the spend was zero, so the instantaneous amount would be higher.
    • datadrivenangel 3y ago
      $58 per hour over a year on average. Sounds like a big big warehouse. For scalability.
      • ludicity 3y ago
        This is approximately right. They also almost have one warehouse per two data engineers. I've been told there's a reason and usage analysis has been done by actually smart people, but it just sounds incredibly suspicious to me.
    • kwillets 3y ago
      He said a few thousand queries per week, which is maybe 3000, or 156k/year. That's 1.56 million cluster minutes, or $500k/1.56M = $.32/minute. A single snowflake node is $.05/minute, so maybe it's an 8-node cluster. They could shrink the node count down to 4 or 2 to save more money.
  • chinchilla2020 3y ago
    How long does a snowflake machine take to spin up? Why have an idle time at all? I have never used snowflake so I'm not sure how this works.
• SillyUsername 3y ago
  I did something similar. Success and super efficiency was rewarded with additional work, including hinting I should help other departments (apparently a joke). I received no extra remuneration despite asking, yet my company continues to hire new staff weekly. I've learnt my lesson, just like this author.
• justin_oaks 3y ago
  And this is the difference between competence and incompetence in tech. We can argue about whether 10x developers exist, but one thing I hope we can agree on is this: Certain problems can't be solved no matter how many incompetent people we throw at the problem.
  • digging 3y ago
    This is really what people should be looking for in a hire. If this person joins my team, can I reasonably expect them to bring some sense into our code/infra/schedule? Nonsense creep is constant in a big enough organization, and on-ramping is slow, but asking "what does this even do?" is invaluable (wait, no, it can be quantified!)
  • plagiarist 3y ago
    We should be realign around "0.1x developer" terminology. Like assume many developers are in the 1x or 2x range, but then there are plenty of 0.1x developers writing scripts that start with 500 lines of comment. Then job postings would have to be, "we're looking for competence and adequacy. We want to pay you a normal amount and get our money's worth."
• mcv 3y ago
  You didn't accidentally save half a million, you deliberately and intentionally saved them half a million, but now you regret it. That's not the same thing. Large organisations are so woefully inefficient that I'm surprised they're able to compete at all, but they have a ton of money and economy of scale and all that, and along the way there's more than enough money to waste millions on stupid nonsense and inefficiency and nobody really cares.
  • photonbucket 3y ago
    In my mind, there is where inflation really comes from. For every wage paid that wasn't useful, the resulting product becomes more expensive / less profitable. If we were able to optimize out all the waste we'd probably have deflation while computers and business processes keep getting more efficient
    • digging 3y ago
      I mean waste is probably a source of inflation but lots of it just comes from unrestrained profit-seeking behavior. If a company believes it can raise its prices without impacting sales, it will do that. That's why so many massive corporations posted record-breaking profits during the recent bouts of inflation.
      • anoxor 3y ago
        Government shutdowns that made it more difficult or impossible to shop outside of big stores or online would be as big or bigger issue here.
        • digging 3y ago
          It wouldn't explain simultaneous mass layoffs.
    • ElectricalUnion 3y ago
      Except that generalized deflation is one of the largest and widespread economic disasters that can be brought about; you are disincentivizing immediate consumption to on the basis of future larger consumption, you're disincentivizing investing in the economy instead of just holding capital - both things that will make a economy stop working in the long term. It's the reason why most central banks really want some minimal level of inflation going on at all times.
• pgraf 3y ago
  In my opinion this red tape is why startups are more successful than large organisations… At least in the beginning
• CobaltFire 3y ago
  I feel this entire post. My career record (US Navy) for cost savings was something over $50MM. Every time I did something I had to do PowerPoints, present to Flag Officers, etc. Hell, once I almost got hugely punished because I didn't let my boss take the credit (he had no desire to because he had zero idea what it was I even did). Note that some of that was as a Lean Six-Sigma Black Belt doing Enterprise projects (I hate every single bit of terminology in that entire godforsaken sentence), where it was literally my JOB to save the DOD as much money as possible with the least disruption possible. Those were the absolute worst years of my career. That period of my career was my reward for just going rogue and fixing things that saved millions. I'll echo the last part of that post: Beware of doing good things at work; the reward is rarely compensation and is usually more work for the same pay.
  • tehlike 3y ago
    Sometimes I think of joining government org (3 letter agency) to find areas to improve $ efficiency on, or make decent direct contribution to US as a thank you for accepting me as an immigrant. I'm an engineer in a FAANG, who worked directly in money flow, and have had experience in diverse areas where it could come handy. Then I start thinking finding the right person to work with & right area to start at is probably 95% of the job, then give up.
    • wslack 3y ago
      As someone who does this sort of work (but posting personally, not officially), you're 100% right. There are groups of already-networked folks you can join like USDS or 18F that are already connected to the problems. The post reminds me of some of the stuff you might see in gov, though I've never seen someone save money and be punished like this.
      • brainbag 3y ago
        What is 18F? That's not the easiest thing to search if you're not looking for the obvious.
• deleted 3y ago
  [deleted]
• orliesaurus 3y ago
  i loved the style of writing! what do you call it? Realism? Sarcasm? Idk It had me stuck reading until the very end - usually that never happens!!
• pedro_hab 3y ago
  I had a friend in a similar situation, he explained the issue and how he would fix it. I was baffled and urged him to be careful and assume he was wrong that it had to be wrong. Anyways, he is still fixing it, getting some people to validate it. I would think he will save at least hundreds of thousands a year. But seriously, why a company that spends millions of dollars in an area would not hire an expert to try to save some money in it is beyond me.
  • marcos100 3y ago
    Because "we don't have the budget".
    • esafak 3y ago
      That "we" may be too low on the org chart, and the qualified manager may not be aware of the problem.
  • freeopinion 3y ago
    Why would a manager of a multi-million dollar department reduce the department to a single-million dollar department? That would reduce the prestige of their own job. I've seen managers who were overstaffed by double work hard to double their staff again. And the only staff that actually accomplished anything were punished and culled. It doesn't make sense until you realize that they operate in an environment where there is no competition that can disrupt their business model. If their costs go up, their rates go up. There is never an incentive for costs to go down.
  • ludicity 3y ago
    Because it's monopoly money or something man, I don't know.
• knodi 3y ago
  you mean someone costed your company half a million dollars for x amount of time.
• dpifke 3y ago
  The author's gallery of poignant HN commentary on their writing is chef's kiss: https://ludic.mataroa.blog/compliments/ https://ludic.mataroa.blog/compliments/
  • chinchilla2020 3y ago
    Some of it is fair. The author does seem a bit egotistical at times. They begin from a premise that "Everyone else is incompetent and I am the savior". There is a high likelihood that many other people in the org were lightyears ahead of the author. It's even more likely that an engineer or manager in their group was "Banking" the inefficiencies to use during a cost-cutting period - and the author ruined that chance... which will inevitably cause massive suffering and possibly poor performance reviews when there is nothing to trim.
    • f5e4 3y ago
      The superior tone of the writing is wild to me, because it isn't exactly a story where I think the author looks great. Certainly others look worse in the story, but still, finding an issue and apparently barely trying to convince others it's an issue is not impressive. It really sounds like they stayed in their lane and did their assigned job, just like every other person on their team.
    • nerdponx 3y ago
      > They begin from a premise that "Everyone else is incompetent and I am the savior". I have worked with someone who was like this. Perhaps unlike the author, they were not in fact the savior in a room full of idiots. This person was no doubt clever and had interesting experiences to learn from. But they were also dysfunctionally arrogant and utterly resistant to ideas that were not their own, to the point where it repeatedly and painfully got in the way of their own progress and other people's progress around them. At the end of the day they got done what they needed to get done, but they left a big mess behind that took weeks to even stabilize (as in, "prod is broken, prioritize above all other work") before the rest of the team could get back to their regular work. May such people forever find themselves in solo contributor roles, so as to spare others the pain of being their close collaborators.
• jjkaczor 3y ago
  Huh, I wonder which of the "big-4" advised them on their practices? Am willing to bet it was 'KTMJ'... (Batman reference, but... strangely similar to one of the "big-4")
  • ludicity 3y ago
    You actually got it. If it isn't a bother, I'd actually like to know how you guessed this, as I thought all the Big 4 were roughly the same.
    • jack_riminton 3y ago
      I think we all want to know!
    • jjkaczor 3y ago
      No comment... whistles and walks away...
• asow92 3y ago
  > I saved my company half a million dollars in about five minutes. This is more money than I've made for my employers over the course of my entire career because this industry is a sham. I clicked about five buttons. I'm sorry, but this needs a privilege/gratitude check. You are guaranteed your salary, and you're welcome to take on the same level of risk your company is by starting your own. If you think it's so easy go ahead.
  • deleted 3y ago
    [deleted]
  • skeaker 3y ago
    I don't understand your point. According to the article it was in fact that easy. The issue was due to hangups in bureaucracy.
    • asow92 3y ago
      This person is asking for a $30k raise to do the baseline of what's expected of them: their job. It's your job to know what needs to be done and assert that it needs to be done. They didn't make $500k, but saved $500k from a mistake that the company themselves made. Why does that deserve a raise? I've seen people implement features confined to a single file that made over $1m/year but they didn't ask for a $30k raise. I could see a one time bonus or stock gift and maybe a more modest raise next year if they stick around. My overall point is this: If you want to make the big bucks, you can go make your own company, but be prepared to take on the mountain of risk that comes along with it. Otherwise, a tech salary with guaranteed income ain't so bad.
• j45 3y ago
  Great post on why LLMs will have Management Consultants looking for their future. Average skill is highly profitable. One skilled senior and 20 juniors at an affordable rate is always most profitable. Developers - Always do this as bonus paid by contingency of saved money and a signed off scope if it works out, especially if you’re d looking into it on your own time. Specifically if you like you can get signatures from everyone on the hierarchy on how much money or time this will save, cost or make them. Do it enough times and the right kind of CEp/President will tell you to stop bringing the business case to them for approval and just do them if they make sense and you have your backup. You will enter a side door, bypassing most politics and c-levels (and maybe triggering some new), reserved for people who say let me see what’s possible instead of coming back with reasons why it can’t be done. From there, paint a picture of what if you built a team of only doers, minus talkers across the organization. :) Tech should never report into Finance, the group that can’t even tame spreadsheets. Yes, business owners pay.
• isoprophlex 3y ago
  > I ask management for a 30K raise after saving 500K and my message is still unread. I suspect I will eventually receive either nothing or 5K. Only one thing left to do. Leave 'sleep(600)' calls all over the place and resign.
• znagengast 3y ago
  I have to play devils advocate here because for every one of these cases, there is probably a dozen of similar stories where the ambitious new guy actually did nuke the system with a risky friday release and then logged off for the weekend xD
  • ludicity 3y ago
    Let me hit you with the extra spicy take - since the department doesn't really produce business value anyway, taking the system offline for a week would have dropped the bill to 0 and been the greatest possible cost saving. Yes, management would have killed me, but I would have been absolved at the Pearly Gates.
  • therealdrag0 3y ago
    Another’s devils advocate is that part of being a senior engineer is writing good public facing one pagers that communicate the problem and the solution. Sure you shouldn’t have to do that for every single change. But when something significant comes along it’s wise to document it and socialize it. If OP had done this they may not have been asked to make PowerPoints or dread conversations or hide anything. They could just point at the public doc that explains it all in technical non political language that also educates other teams how they can improve too. This is leadership and mentorship opportunity.
• cameron_b 3y ago
  I love this for purely self-appeasing reasons. this machine kills imposter syndrome or at least it helps. Having a background in solid CS theory from High School, and having a degree in Art, I find it very hard to apply for engineering roles, and my mixed bag of experience often lands me in Support Engineer / application admin / integration roles, fuming tremendously when the people with SwEng / Developer titles fumble on with implementing some JavaScript change for a feature I need in Service Now for my application's customers. It is incredibly reassuring that it is not simply my organization that is hamstrung by the pretense of complexity, when really someone just made it complicated to make it seem important.
• 23B1 3y ago
  I accidentally cost my company half a million dollars, but it's not really my fault HR misplaced a comma.
• thefourthchime 3y ago
  I saved my company $1 million a year a couple of months ago by noticing that there was an S3 bucket that kept growing and costing 80k a month. I poked around and realized that there was a system that we weren't using anymore that was copying files to the bucket I reached out to the stakeholders and they turned it off and we deleted the files. The higher-ups didn't seem to really care, My boss's boss told me to reach out to another team that should've caught this and that was about it.
  • Foobar8568 3y ago
    This is what is going on to a Saas provider we are using. I told them about but they shrug it off..VC money I guess :D
• nickdothutton 3y ago
  The unsaid part is that this kind of situation is everywhere. It only really hits home when (IMO) when you’ve had a couple of decades in the industry. We have totally squandered the abundance of resources in many sites/situations/use-cases. Whether it be IOPS or threads or memory or network bandwidth and latency. Yes premature optimisation is the root of all evil, but the other side of that coin is pretty ugly looking too. Not even because of monetary waste in many cases, but because of unnecessary complexity and fragility. For what? A slightly higher level of abstraction? A bit more interoperability? A marginal gain in some other metric?
• bitwize 3y ago
  Great! That's worth (let's see... carry the three...) a 1% raise!
• yudhiyudhi 3y ago
  I know of an intern (with a master mentor) who did a one char change and saved our company many millions - the change was to update the threshold of data size of when to not compress small messages - this threshold was updated after many many years - old threshold was based on old compute and network costs.
• throwaway-jim 3y ago
  is this larping?
• joshuahutt 3y ago
  Now extrapolate this out and realize that 25-50% of our global tech economy is basically waste.
• grvdrm 3y ago
  This post hit right something I dissected in my last role. In short, I was revamping/enhancing a property insurance policy pricing system as the lead from the business side (i.e. portfolio manager/actuary). However, much of the development team was truly incapable of writing software in any useful way, so I inevitably dipped into C# and SQL to help diagnose issues as we (as a team) worked on the enhancements. My business coworker and I found something that remains funny to this day: code littered with references or attempts to use the .NET TPL library to make the platform do more things "in parallel," but without actual knowledge of using such a library in practice. Nothing worked in parallel! There were blockers all over the place! As someone else said, TPL might well have been a //TPL TO DO. We suspected that TPL was there as one of those "look what I can do" intrusions that received praise from more senior devs/mgmt. even if it never actually did what it was supposed to do.
• siliconc0w 3y ago
  At $large-scale company this happens pretty regularly if you just look for expensive things and ask: * "Can I just remove or slowly deprecate this?" I.e this job generates data no one uses, or alerts people just bin-bucket or has been already replaced by faster/ better. * "Can I cache this?" * "Can I run this at a lower-priority/off-peak/less available?" * "Can I reduce the frequency or move processing to deltas?" This is just the easy stuff you can usually do in a few lines of change without even getting into basic optimization rewrites. Most internal things at $large-co are built because someone thought they might be useful, they might get promoted and using that hypothesis and moved on, but few things are actually continuously validated as still generating value > their cost.
  • pjc50 3y ago
    Ask whom, though? It's usually really time consuming to determine that nobody is using something, and difficult to get permission. Hence in so many of these hero stories, the critical work is done without permission. This morning on bluesky I saw a story of a very hasty onprem-to-cloud migration that was facilitated, and accelerated, by the rising floodwaters drowning the premises.
• piperswe 3y ago
  Repost of https://news.ycombinator.com/item?id=38064086 https://news.ycombinator.com/item?id=38064086, should probably be merged
• andrewstuart 3y ago
  It’s always the cloud costing ridiculous amounts of money.
• pavlov 3y ago
  It costs a million dollars to run a service that essentially “uploads a 2KB CSV to a database” on Snowflake? And you can cut this in half by changing some defaults in instance lifetime? I’m starting to understand why Snowflake’s market cap is something like $50B. This sounds like a nice money-printing business if you can convince enterprises to use it.
  • ludicity 3y ago
    I should clarify, the cost was coming from computers that engineers used to write queries against our database - the 2KB CSV was just how we managed their permissions, and the cost of that was negligible. The immense cost was coming from someone writing a query that translates to "I need one row of data" and then we get billed like $10-20 in idling compute. With multiple computers and several full-time SQL modellers, it adds up very, very quickly. As the guy that just has to keep Snowflake running smoothly and isn't paying for it, it's a really nice product. I would still prefer something else on principle because it isn't open source, but eh, I guess it reduces my stress at work.
• ynx 3y ago
  > I ask management for a 30K raise after saving 500K and my message is still unread. I suspect I will eventually receive either nothing or 5K. Pain I feel very acutely. As an employee, I saved a FAANG company billions of dollars in potential GDPR fines after they carelessly declared 'mission accomplished', weeks before the deadline, and never even got a 'thank you', much less a raise.
  • gv83 3y ago
    Mind your own business next time, we need the money here over the pond :P
• jb3689 3y ago
  Whenever this happens, I really have to wonder about all of the people I call "good" on my team. Like surely someone gave a shit enough to know this is how it works, right? ...right?
• ilc 3y ago
  Reason number 543895 why I like small company life: When you find shit like this, people will go "my bad" at the worst, and you all are happy the company has more money. Small teams and small firms have their downside, but honesty and transparency, especially when it comes to cost savings... tend not to be one of them.
• deleted 3y ago
  [deleted]
• RadixDLT 3y ago
  if you want to learn more about consulting firms watch John Oliver https://www.theguardian.com/tv-and-radio/2023/oct/23/john-oliver-mckinsey-consulting https://www.theguardian.com/tv-and-radio/2023/oct/23/john-ol...
• greedo 3y ago
  I'm on the tail end of my career (hopefully), so I can share some unsolicited advice for younger workers. 1. Remember who you work for. Yourself. Not your boss, not your grandpa who always wanted you to become a writer. You only have so many breaths before you get your ticket punched, so make them count. 2. Don't seek work as something you should love. Those should be your loved ones, and hobbies etc that stimulate you. Sure there's a small minority who get to do what they love, but over time love can turn to disgust. 3. When starting a job, figure out the organization's incentives. Some orgs want change, some will fight any change. Figure out what the org wants and your life will be much easier. My job is like the Maytag repairman, kind of waiting for stuff to break. I refer to it as being a digital janitor. 4. Nothing you build digitally will last. Hell, between linkrot, bitrot, and the heat death of the universe, nothing lasts. So don't expect what you've built to last, or to hold value. It'll all be re-factored/re-engineered/re-architected and become obsolete. 5. Large organizations are toxic and often sociopathic. So are small orgs. If you can find a way to start your own business, try to avoid becoming that way. Good luck, most businesses either fail or become that way. But at least you have more choice in the matter. 6. Find a mentor (sometimes referred to as a rabbi) who can provide guidance to you when times get tough. Don't choose a rabbi from your reporting chain. You will find yourself adopting a worldview that might not be aligned to your own interests (in other words, the bastards will manipulate you). This rabbi can be a friend, or just someone you vibe with. Be careful in choosing your rabbi, and take care of this relationship. 7. Illegitmi Non Carborundum
  • ludicity 3y ago
    Great advice, especially on finding a mentor. I hadn't thought about the importance of finding someone outside your reporting chain. Illegitmi Non Carborundum.
  • preciousoo 3y ago
    How do you maintain such mentor relationships? I found a very inspiring mentor at a place I interned at this summer, that I'm due to return to full time in coming months. But I'm not sure on how to continue this moving forward, as the previous was an intern program relationship, and I'm now full time.
• lsh123 3y ago
  Long time ago in a galaxy far far away, I replaced the need for N * (Oracle license + Sun server) with a simple perl script (< 100 lines) and one Sun server for a total cost saving in $300M+ / year. While doing it, I also invented map-reduce (it was before Google time). The problem was to calculate bunch of stats from web servers logs (e.g. 10 most popular pages). The original solution was loading it all into Oracle database running on multiple servers since logs were huge. And then running bunch of SQL queries. Rinse and repeat daily.
• arein2 3y ago
  good management would fire half the org
• Ikatza 3y ago
  > They hired some incredibly talented people to make this happen, and then like five times as many idiots. I can relate to this.
• arein2 3y ago
  In my company we worked with a platform launched on local machines that had an admin console where you could execute java code. Pair that with almost everyone not turning on the firewall and all engineers connected to the same wifi network, anyone could do whatever they wanted. I showed a demo how easy it is to read private ssh keys to the head of infrastructure, and after some months people could connect to network only using custom credentials (ldap) which was good, but also asked us to install "spyware" that among other things checked the firewall. I never installed the "spyware" but nobody pushed me. I didn't think I somehow prevented a disaster or did some heroic deed because everyone in the company was professional and nobody would exploit this. But of course I didn't tell about this to anyone except the infra because such information should not be disclosed until is fixed. And once is fixed why disclose it? I really miss the Mac checkbox to enable the firewall. On linux I use nftables which is really powerful, but with so many possibilities it is easy to miss something during configuration. I observed a lot of senior engineers don't have sufficient network knowledge. A lot of people on linux don't use the firewall which is really bad if you work on shared wifi. Also when running docker images, if you map a port when using docker run (ex. docker run -p 80:80), docker will automatically add firewall rules and bypass the enabled firewall, exposing that port publicly.
• afro88 3y ago
  Sounds like something out of a Gene Kim book. Definitely a pattern through the places I've worked. The ones that are smart, have good intentions and go a little bit rogue tend to make the biggest impact. Executives are too concerned with risk, and honestly, it playing out this way suits them better: if someone goes rogue and it works, they can claim the win, if it doesn't, they can blame and reprimand.
• geephroh 3y ago
  "My team has spun this as a huge cost saving, when really we just applied a fire extinguisher to the pile of money that we had set alight." I love this so much
• mateo411 3y ago
  I think the author should take a vacation.
• effnorwood 3y ago
  [dead]
• mikewarot 3y ago
  This kind of thing has been going on forever. Once upon a time, I walked into a room full of VT100 terminals, and bored CS students waiting for their compiles to finish. (It was a weekend near the end of term) I took a look at the system, and realized they had pushed all the compiles into a batch queue, but that queue defaulted to BELOW interactive priority, so any keystroke anywhere had higher priority... so all the people checking their position of the compile job in the queue, slowed it down even more. Over the next 15 minutes, I kept bumping the priority of the top job in the queue up, an hour later everyone had their work done, and went home. I had the room to myself. Rogue sysadmin for the win. ;-)
• kwillets 3y ago
  "For example, you've run 234,745 INSERTs into [table name deleted]_HOURLY and ALL of them were under 1000 rows." That's an actual quote from our org's snowflake support slack, about pretty much the same problem: lighting up the cluster with a bunch of tiny transactions spread out in time. The product is not made for common use cases like trickle loading. I'm an actual experienced data warehouse admin, so I'm watching them rediscover my job description one cost overrun at a time. They unironically say things like "It's self-managing, but you need to monitor costs and rewrite loads and queries to run more efficiently". I'm afraid to ask them what they think I do all day.
• high_pathetic 3y ago
  Also, you lost Snowflake half a million dollars.
• DoreenMichele 3y ago
  I think the most charitable interpretation of this phenomenon is that bureaucracy is hell and the entire organization tends to get put on a leash pegged to the level of incompetence of whomever is in charge of #thing. (Peter Principle writ large.)
• dkarl 3y ago
  This reminds me of the company I worked at that was paying for dozens of unused Redshift clusters at any one time, peaking over a hundred. There were common data science (read: ETL configuration) tasks that required spinning up a Redshift cluster, and the "data engineers" (read: fresh grads with a few weeks of training) sometimes forgot to shut them down afterwards, or got diverted to different work temporarily and couldn't find the original cluster when they returned to complete the task. The first proposed solution was to ask the forgetful data engineers to manually tag all the resources they created, so un-tagged resources could be deleted promptly. When we asked how the tagging scheme would allow us to distinguish between clusters that had been forgotten and clusters that were being actively used, we were told that it was not necessarily a complete solution, but it was "a step in the right direction." Ultimately the problem was declared solved, but I wonder if it really was. The company was swimming in VC money, but even so, that level of waste (which was consistent across all of our astronomical AWS spend) was impossible to ignore. Checking back now, all of the engineering jobs they're hiring for are overseas, so they must have ultimately nuked the entire department.
• BohuTANG 3y ago
  Indeed, under a pay-as-you-go model, if there's a lack of precise control over the warehouse, such as a 10-minute suspension, it could lead to significant waste. This is because most queries might only take a few seconds, and the rest of the time is wasted. If you find Snowflake expensive, consider Databend. It's an open-source, cost-efficient alternative to Snowflake, and it maintains a consistent product experience with Snowflake. Open-source: https://github.com/datafuselabs/databend https://github.com/datafuselabs/databend Databend vs. Snowflake: https://github.com/datafuselabs/databend/issues/13059 https://github.com/datafuselabs/databend/issues/13059 Cloud: https://app.databend.com/ https://app.databend.com/
• _diq5 3y ago
  Tried to save my company 100k per month by cutting down idle compute, clearing out petabytes of unused buckets, and using on-demand compute in testing envs. This resuted in me getting pummelled by directors asking questions, staff engineers saying random what-ifs, until I eventually gave up. I used to think startups would be one of the few places that actually gave a shit about being lean and efficient - but turns out that's only true if they're bootstrapped.
• xyst 3y ago
  $500K per year saved. Now add in some layoffs in next quarter and pump those financial reports. Should have just let them burn dude
• ivolimmen 3y ago
  Apart from the apparent topic: If the architecture of the software is a mess like he describes I would bail or ask to make significant changes (and threaten to leave if I wasn't allowed to). I would definitely not just try to optimize using settings.
• zubairq 3y ago
  Yep, can definitely relate to this
• isaacfrond 3y ago
  what they actually needed to do was fire most of the staff in every team, leaving behind the two people who actually had good domain knowledge, then allow them to collaborate with good engineering teams to build sensible processes and systems Well duh...of course you'd want that. But it is not so easy is it? I'd also instantly fire half my staff and hire hard working geniuses instead. But that just isn't an option, unfortunately.
• Fanmade 3y ago
  This is the first time that I saw this blog and I've now read a bunch of those posts. There is so much gold there, it's incredible. So much is going wrong in all those companies, that it is just frustrating to be working in those. A colleague of mine and me got the opportunity a while back to basically work as wanted to in one side project. So we just worked truly agile, without any of the "Scrum" bs around it. Everyone involved was blown away how successful it was in terms of implementation. Two companies tried it before in the span of a year, and they didn't get it working at all. We worked just some evenings and weekends outside of our full-time job over the span of six weeks, and we got 90% done, with everything essential included to be able to start working with it. We then implemented everything else the customer wanted within the next weeks by following the same principles. There were no problems at all (apart from the customer basically almost running out of money because they spent most of on the previous companies which couldn't deliver). At my day job, the managers usually argue against anything we want to do, question every decision from our side, want to have regular big meetings with 16+ people to talk about everything at length until no one wants to do it anymore, and then micro-manage everyone to death. We tried to at least keep these guys out of the daily by pointing to the Scrum guide where it says that only people working on tickets should participate, so one of these guys just created a bs task for himself, then talked every day about it for a lengthy amount of time without providing any real value at all. I never got those silly shows before, which play in offices where everyone just acts silly and they never really work. But now I know why they are so popular. They just provide comic relief for people who really have to work in those environments and it is really as bad as in the television shows, if not worse.
• gmerc 3y ago
  one of my employees saved an unnamed tech company 250M in regulatory fines. They were flagged as top performer and subsequently laid off in the first wave of the tech layoffs … probably because stock compensation clawback made it very juicy.
• lloydatkinson 3y ago
  Like others said, I absolutely feel this post. I don’t think I can say any other post I’ve ever read feels this relatable. Every one of his posts is both accurate and a depressing assessment of the overwhelming incompetence in this industry. > I literally can't remember what was said, there was some Agile bullshit about doing a discovery piece, then it just never happened. I know he doesn’t work the same place I did but damn, it sure feels like it.
• nobodyandproud 3y ago
  Line manager here. When I find someone very talented, I go out of my way to promote them because good engineers who want to stick around and not jump when the times are good is hard. The sad reality is that I’m limited to how much I can bump their pay, give them time off, etc.
• Yoric 3y ago
  I'm sad to read that. I can only testify that this is not my experience at all, so it might be that you're in the wrong place. In every company I've joined, I've found issues, often large ones, managed to get assigned to them, and had fun fixing them (or at least the satisfaction of seeing them fixed). Then moved on to the next issue. I haven't always received much recognition from the powers that be (although me and a few colleagues have once been wined and dined at one of the most expensive restaurants in the city for having saved the company 1M$+ with a few days of work each, that was nice), but I've gained respect from fellow engineers and from myself, which was nice, plus lots of things to tell during job interviews. So, may I suggest getting out while you can and finding a better place to work?
• consultSKI 3y ago
  "We have always done it this way" comes to mind.
• johnny99k 3y ago
  I saved my company $50,000/year. A previous consultant (that they were still paying monthly) had written an application that was the backbone of the entire ordering system. If it went down, no work could be done and they needed to call this guy and pay him $150/hour. It was a .net app that I reverse-engineered that was basically calling 10 or so encrypted stored procedures in SQL server. Because the older versions of SQL don't actually use encryption, but a type of encoding, I was able to decode them all and replace the entire app with a script that I wrote. The whole process with testing took about a month (they thought it would take me a year). The consultant that made it wasn't very happy when he was told they would no longer be needing his services.
  • fennecfoxy 3y ago
    Pretty sure that behaviour is as illegal as taking company code with you when you leave, inserting kill codes into software you work on, etc. Being paid to write software for a company and intentionally gimping it in that way probably also invalidates any liability insurance that a contractor may have as well.
• jatin085 3y ago
  I could fully relate with this post. I personally share a similar mindset: someone who can take criticism with pleasure, who has sorted out thoughts about how the industry operates, who has not subscribed to the belief that people in leadership positions are world apart from us, and who can openly express that wrong is wrong. I think many of the people who disagree have either been serving in non-engineering roles since past couple of years, or have bought in the pitch of rosy world created by managers and so-called leaders.
• potta_coffee 3y ago
  At a small company, I found a bug in the sales system that was giving away $10k per month by miscalculating customer's tax. CEO of the company essentially pretended that he couldn't hear anything when he was told.
• wesleyd 3y ago
  No good deed goes unpunished.
• up_o 3y ago
  It's good to know there are worse places than mine. some things are reminiscent of what I've seen, but nothing so extreme as even half of what's described.
• rglover 3y ago
  The amount of time and money wasted on incompetent people who shouldn't be running or working on things is outstanding. Fire the majority of people leaving the A players, unmanaged, and the world will begin to work near-flawlessly in a few weeks to months. But this line of thinking hurts feelings which are (presumably) paramount to actually solving the problems people set out to solve.