4 ms·

This 100%. I just requested a hackathon for performance improvement (might be wasted effort). We are pushing tons of code and now our CPU usage has grown expon

• ryandrake 2mo ago
  Exactly. For most of my career, bad engineers (or juniors who were earnestly learning) would struggle to even create output that compiled, let alone ran. And it would take them a long time to implement something poorly. So the blast radius from their incompetence would be limited. Today, a bad engineer can churn out ten thousand lines of garbage that actually compiles and runs, before my first cup of coffee. If not on a tight leash, they'll sink the whole codebase or inundate every senior person with garbage code review requests. The blast radius is unlimited!
  • palmotea 2mo ago
    > Today, a bad engineer can churn out ten thousand lines of garbage that actually compiles and runs, before my first cup of coffee. If not on a tight leash, they'll sink the whole codebase or inundate every senior person with garbage code review requests. The blast radius is unlimited! The term for that is "10x AI engineer." Anyone who has anything negative to say about such people is just jealous of their insane productivity and speed.
    • florianherrengt 2mo ago
      [dead]
      • palmotea 2mo ago
        FYI, your comment showed up as [dead] like you were banned until I vouched for it, but I don't see any reason for that. Just thought I'd let you know in case you want to follow up.
        • florianherrengt 2mo ago
          Thanks for letting me know. I had no idea. I’ll follow up with the moderators.
      • cdud3 2mo ago
        > You can make your own numbers look incredible while reducing the throughput of the entire team. Taken how some companies award promotions and bonus this is actually a double win. You not only improve your own numbers but also make this of your competition worse! /s
        • SauciestGNU 2mo ago
          I just left a backstabby stack ranked company and so many people were sabotaging colleagues or teams they were in competition with. AI psychosis has been enabling terrible management practices as well as terrible engineering practices.
  • budsniffer952 2mo ago
    If your company has bad engineers cranking out 10,000 line PRs your engineering culture and product was already bad, I guarantee it.
    • RSHEPP 2mo ago
      It's not the whole culture and product, it's specific teams that do it. And trying to push back on teams that are "producing" is not a simple task.
      • Silhouette 2mo ago
        Sadly AI also amplifies both the good and the bad managers and tech leads. If you have managers who fail to set a clear direction or tech leads who fail to define a good framework for developers to operate within then AI just means the poorly directed developer effort goes further in the wrong direction faster. Meanwhile teams with well-specified goals and better structure and processes can exploit the benefits of AI tools much more effectively.
    • oriolid 2mo ago
      In the before times, a small dedicated team could hold back against bad engineering culture and intentional enshittification. Now the balance has changed.
  • watwut 2mo ago
    > or most of my career, bad engineers (or juniors who were earnestly learning) would struggle to even create output that compiled, let alone ran Really? I have never seen that. The expectation was that you could create the code that compiled on day one.
• cube00 2mo ago
  > We are pushing tons of code and now our CPU usage has grown exponentially over the past year because the bad engineers just ship whatever Claude gives them and do not think about the consequences. Sometimes this can be a death by a thousand cuts. Any individual change may not impact performance to a noticeable degree but when they're pumping out a 10x increase in commits it can be a slow decline. Just look at how they're merging ~300 commits a week into bun. https://github.com/oven-sh/bun/graphs/commit-activity https://github.com/oven-sh/bun/graphs/commit-activity
• Shorel 2mo ago
  That really sounds like the kind of people who will complain in university about being forced to study calculus. Natural consequence: Then they never grasped the concept of computational complexity. O(n) Vs O(n²)? They have n, what's the difference? Python is fast enough. The only thing that matters is shipping features fast! Features! Our competitor will have this next week, we need to write code fast, everything else is a matter of adding more compute, which we will pay with revenue!
  • soperj 2mo ago
    computational complexity was never part of calculus. Calculus is the math of continuous change.
    • Shorel 2mo ago
      I should add that text to the rant at my last paragraph!
    • moregrist 2mo ago
      You don’t learn Big-O in Calculus because it requires CS algorithmic concepts. But the core question of “how does this behave as N -> \infty?” is asymptotic behavior (ie: limits) which were developed for calculus and are very much part of the foundational calculus canon.
      • BigTTYGothGF 2mo ago
        > You don’t learn Big-O in Calculus because it requires CS algorithmic concepts Big O notation was invented in 1894.
    • deleted 2mo ago
      [deleted]
    • WorldMaker 2mo ago
      Big O notation predates Computational Complexity. It was originally used to guesstimate asymptotic behavior (limits as they approach infinity). The related Little o notation (even more specifically limits of already large values as they approach infinity) is directly related to early attempts at defining differentiation and differentiability and still sometimes show up in Calculus next to derivatives to explain how derivatives work. The visual interpretation/intuition of Big O notation can be a useful way to build a visual intuition of what a function's derivative and integral "shapes" may look like, so some Calculus books teach Big O notation, too.
• mirmor23 2mo ago
  If your company has sketchy process and minimal checks on the software quality, of course it naturally follows that they hire bad engineers as well. fortunately, that also opens up an opportunity to be a emergency leader with an eye towards promotion.
  • palmotea 2mo ago
    > If your company has sketchy process and minimal checks on the software quality, of course it naturally follows that they hire bad engineers as well. fortunately, that also opens up an opportunity to be a emergency leader with an eye towards promotion. AI disease is encouraging "sketchy process and minimal checks on the software quality." QA has been eliminated from my team, and the QA engineers that are left have been declared to be developers now. Gotta move fast, and I guess making sure the stuff we ship works was "slowing us down."
    • satisfice 2mo ago
      More like developers are now testers and nobody is doing QA.
  • pydry 2mo ago
    The bigger issue (I find) is that the pipeline for finding those engineers is completely fucked. Hiring pipelines that tested the wrong thing have existed for years but the problem is magnified 10x when you test for something that weakly correlates with ability at best which an AI can do better than a human. This is leading to stuff like incompetent junior-level engineers being hired as principals.
    • lubujackson 2mo ago
      My company's hiring process is basically: Can you code a product quickly with AI? Can you understand the generated code? That's basically it. It is surprising hard to find people that can do both, but engineers are becoming much, much better at the first gate while flaming out on the second.
  • 2mo ago
• whilenot-dev 2mo ago
  The handling of backpressure is such a good example to distinguish bad from good engineering. A good implementation even presents all the typical "clean" code indicators: DRY, KISS etc.
• noncoml 2mo ago
  Sounds like bad management to me
  • jurgenburgen 2mo ago
    My company got rid of line management. I now report to a director with over 20 reports and barely any time for career development. Our performance reviews are AI generated and we’re losing engineers. The industry has gone completely insane.
• soperj 2mo ago
  This has less to do with bad engineers than bad policies and procedures. You don't performance test before shipping to prod? you get what you get.
  • RSHEPP 2mo ago
    Startups have limited resources, we prioritize the best we can. As a staff engineer it's my responsibility to bring this to my company, but it's not an overnight process.
    • soperj 2mo ago
      why wouldn't you use those limited resources to test things before they go to production, instead of creating more crappy things that go to production?
      • rhdunn 2mo ago
        What do you test? How much time/effort do you spend testing each of those things? How do you know those will be performance issues? There's only so much you can do -- making reasonable assumptions, choosing suitable data structures, database indices, testing various workloads, etc. -- in a limited test environment. You may spend a week optimizing a feature that only 10 people use or that never gets to the point where it becomes an issue. Or you may have something that can't easily be tested at scale, such as various counts or other dynamic data that are determined by complex queries that you may (likely) find you need to cache but don't necessarily know which values will become issues until you start using the system.
        • soperj 2mo ago
          We have a test team that runs various test suites(automated) and manual testers that test functionality. They spend their entire working day, every day testing.
    • fn-mote 2mo ago
      From this argument, it sounds like your company is making a rational choice to go with what you’re seeing. Even if you don’t like it.
• notakio 2mo ago
  When I worked at a particular fruit company in Cupertino, my boss had been asked repeatedly by the developer teams to provide a tour of the data center we'd recently finished building out, which housed the servers their code ran on. He gave them a lengthy, grueling, hyper-detailed tour of the entire facility, encompassing the HVAC systems, electrical systems, network and computing systems, finishing with about 20 minutes where he had them stand inside a hot aisle that he was just outside of, giving a fantastic soliloquy on the importance of code efficiency, and the consequences of ignoring it. It was hilarious to watch from the comfort of the cold aisle, knowing full well what he was doing.
  • dcrazy 2mo ago
    What did the software engineers hope to learn from this tour?
    • jackyinger 2mo ago
      The scale of a cloud datacenter is pretty awesome to witness. I had the opportunity to visit a large cloud company’s data center years ago, it was immense in scale and (mostly) very well thought out.
      • ltsui 2mo ago
        You just reminded me of this video from Google which toured one of their data centers back in 2009: https://www.youtube.com/watch?v=zRwPSFpLX8I https://www.youtube.com/watch?v=zRwPSFpLX8I
    • sgarland 2mo ago
      Because hardware is cool. I’m personally of the opinion that everyone who works in software should have to rack a server and bootstrap it. Physically mount it, cable it, and get the *nix distribution of your choice running on it, serving Hello, World. Everywhere I’ve worked, there is a marked difference in the quality of engineers who had played with hardware - even those who merely had expressed interest in it, and maybe had an RPi - and those who had not. There is something about physically touching the thing that runs your code that makes you better at it. Maybe it’s a correlation between “wants to play with something unnecessary but adjacent” and “curious enough to ask why more frequently,” but I swear, it exists.
• pengaru 2mo ago
  Mind sharing where you're at?
• butlike 2mo ago
  A hackathon for performance improvement screams to me "you'll be re-interviewing for your own job." I don't foresee this going well for the company. Best case scenario: you wildfire churn all the chaff engineers which limits the company's throughput until they're rehired... with no expectation the new hires will be any better. You also lose A LOT of domain experience and are self-inflicting brain drain. I'd try and push for some "lunch and learn" meeting where the engineers get lunch catered and in exchange sit in on a meeting where you explain your point of view. Without monetary incentive it'll be hard to change the culture, but not impossible (and food goes a long way in greasing the wheels).
  • wonnage 2mo ago
    they meant application performance
  • ivanmontillam 2mo ago
    > A hackathon for performance improvement screams to me "you'll be re-interviewing for your own job." Not necessarily, great engineers do also ship temporary code they didn't have the time to trim. Our process is of 1) make it work, 2) make it right and 3) make it fast; not necessarily that engineer had time for the 3rd step.
    • scj 2mo ago
      The older I get, the more I feel that #3 should be "make it work well". Where the definition of "well" can be situationally interpreted.
• bmurphy1976 2mo ago
  Do you have SLO/SLAs defined? Are you monitoring performance? CPU usage? Cost increases? You should have plenty of data to review regularly and push back on any teams that are causing problems. That's a process problem and you need a process for it. Trace the increase back to specific deployments, call out those teams, and make them fix their shit.
  • RSHEPP 2mo ago
    Yes to all, but it's just me responsible for monitoring performance/cost. I either push back or fix it myself, but I am behind. So this is an attempt to give/push more ownership to the teams.
• ponector 2mo ago
  The actual issue is not the "bad" engineers but the bad organization. While people are allowed to push and merge whatever crap is generated there is no point to do otherwise. Even if you do care about performance, "good" code your teammates don't and close more tickets and are better by many metrics. If you are closing one ticket per week with "good" code but your teammate does three with "bad" code - it's actually you are a bad employee. Also they may say you are a toxic one.
  • florianherrengt 2mo ago
    It means the company is measuring the wrong thing. I would much rather have someone on my team who ships less but whose work I can trust than someone much faster whose changes leave me wondering what problems we’re going to discover later. And when production breaks (and it will), I need the person who made the change to actually understand it well enough to help fix it, instead of showing up with no idea what is going on. You can obviously be an asshole about how you do it but I don’t think pushing back makes someone toxic. You need to be flexible and compromise when the business trade-off makes sense. But you also need a backbone. If you think something is going to cause real problems, bringing it up is part of the job.
    • ponector 2mo ago
      >>I would much rather have someone on my team who ships less but whose work I can trust Me too, but it is not what happening across the industry. Instead they are pushing for more LLM usage as well as more features. And faster, faster!