4 ms·

"You can't seriously tell me that the unhappy leg of the code path has no test coverage." Sometimes I forget how ignorant HN can be of real world software deve

"You can't seriously tell me that the unhappy leg of the code path has no test coverage." Sometimes I forget how ignorant HN can be of real world software development and the bar of corporate code quality, and then bangers like this remind me of it.

bbarn 2mo ago
Exactly this. I've seen production level trading systems grind to a halt over a simple bug and no matter what tests you have in place, it happens.
- lbrandy 2mo ago
  I cannot even begin to express how many times I've seen engineers working super hard to optimize happy-paths so that we turn 3 nines of availability into 4 nines but introduce unintended emergent behaviors in unhappy-paths that turn 1 nine into zero nines via thundering herds, retry storms, etc.
  - Breakthrough 2mo ago
    You have my empathy for this kind of sentiment. Personally this seems somewhat rare in practice. That being said I'm curious if anyone has anecdotes they can share about these kinds of things?
    - aprdm 2mo ago
      Configuring postgres to automatically failover instead of doing it manually. The automated system caused more downtime in a few months than manually doing it did for years before. All in the name of more automations and less downtime
      - eudamoniac 2mo ago
        Could you elaborate on why that would cause more downtime?
      - hinkley 2mo ago
        For resolution of a problem you can’t prevent categorically, you have to spend a good bit of time pretending to be the computer before you can trust the computer to do it. A run book is a draft of a requirements doc for a program to do the same thing. And you can’t afford to test that program in production without loads of simulations first. And that’s running the process manually and checking aggressively for any signs of problems. Some coworkers snaked the autoscaling work out from under me, and then ignored the advice I offered on low water marks and weeks of testing and rehearsals. All I can figure about their irrational exhuberence is something to do with claiming something for annual reviews? All I do know is we didn’t make it 40 hours from when they flipped it on until we had our worst outage in two years. Classic FAFO. They were hoping to eventually get to a low water mark of just over 30% of our static cluster size, and they decided to use that for their initial go instead of my estimate of 40% being the low, and a recommendation that they start with 50-60% for the initial weeks and then ramp it down. When I refused to let it go in the status meeting, the team decided we should vote on it. Two guesses how that went. A bug in the deployment logic the Ops team had for doing things like upgrading VMs caused the next update during daylight hours to spin up the new instances at the minimum cluster size, instead of the current cluster size. It was done outside peek hours but that still had us cut over with just under half of the hardware we needed at that moment. Because we were in such a fucking hurry to be something like third in line to use the new autoscaling support, nobody else had run into this problem yet (or system did a pretty good job of buffering everyone else). A couple weeks later they’d looked at the stats and decided that we were spending less than a half hour a day running at the low water mark, and the additional shutdowns were causing churn that made it harder for us to detect problems like memory leaks. Surprise, surprise, they increased the min cluster size to exactly what I’d told them two weeks before. Since “listen to me” isn’t a lesson that transfers to other teams, I will instead say, don’t transfer initiatives to a new team when there is less than 10% of the project left. The lack of friction you encounter may result in a lack of respect for the danger, and attention to potential problems. And if you must transfer, if your ideas about rollout timeline end up being less conservative than theirs, be patient and do it the slow way. They probably have encountered problems you haven’t seen yet.
    - unscaled 2mo ago
      I think the "happy path" might be a slightly wrong classification in GP, since the post is in reply to a retry-storm issue and explicitly talks about retry storms and thundering herds. I've seen many cases where engineers optimize the sad path, but pessimize the wretched path. Or in less flowery language, they cut the occurrence rate of common non-critical failures, but by doing that they introduce code that can make rare failures much worse. The cases I've seen generally boil down to naive retry logic or poorly tested and poorly maintained fallback paths (such as killswitches that break their environment[1], graceful degradation turned graceless, dormant feature flags that get reactivated). The case you see with a retry storm here is the most classic one and the one that annoys me the most. I've seen engineers adding aggressive retries even into places where the impact is minor (you could show an error and let the user manually retry instead). Retries that improve user experience can be great if done correctly, but I've never seen the authors of such pull request addressing the risk and mitigation techniques for retry storm or retry amplification. I've seen cases which had: 1. Retries on the client side (browser or mobile app). 2. Retries on the BFF. 3. Retries on Microservice A used by the BFF. 4. Retries on Microservice B used by Microservice A. 5. Retries on Critical Service C used by Microservice B. Most of these retries had very short timeouts (e.g. 100ms), in order to keep latency SLOs during normal operations (not a good idea on retries). Every time QA saw a layer without retries, that would be a bug, and adding retries is easy, so we'd get a new retry without much thought. But the first time Critical Service C became overloaded, Microservice B started timing out a couple of times and retrying. This was too much too much for Microservice A that had a short timeout that couldn't hold the 3 retries done by Microservice B, so it making doing its own retries, all of them dropped in the middle of the way. Eventually you'll get a full-blown retry storm where every request from the client side got amplified with 3^5 retries, easily bringing down Critical Service C. We'd usually introduce a circuit breaker for the particular path that caused the issue, but a variation of this kept happening several times because designing safe retries across a vast collection of microservices takes a lot of effort, and it's always easier to just add a quick-and-dirty retry at any point where you think you might need one and call it a day. A proper solution (which I've never seen implemented) would be an mandating a corporate-wide inventory of retry-paths, and monitoring it for any path that is at risk of triggering a retry storm, or adding mandatory headers that cross microservices and track the amount of retries done up the chain and the time spent in total waiting for previous retries. You could have a budget for both and automatically stop performing more retries. Both solution require extra effort and a large degree of coordination. [1] This was the CloudFlare issue mentioned in this thread https://blog.cloudflare.com/5-december-2025-outage/ https://blog.cloudflare.com/5-december-2025-outage/
    - lbrandy 2mo ago
      Some anecdotes 1. People add retries w/o backoff. Now we have a retry storm. 2. People don't add jitter so we get huge waves of highly correlated retries that cause self-perpetuating overload cycles and failure. 3. People add retries (even w/ backoff) at more than one layer (e.g. one in process, one in envoy), so now we have a deeply confusing multi-level of retries with super weird n^2 patterns. 4. People find ways to fetch from multiple data sources to make a composite object but don't cache/reuse data they fetched, so one data source being down causes DOS on _other_ data sources because of retries. 5. People add failover mechanisms where all failures happily pick the same failover target because, ya know, it has the most free capacity. 6. People underestimate connection setup cost so "failover" causes huge increase in overall load due to connection setup (often tls setup) causing huge influx of "new" cpu work in a loaded system. CPU spike causes unhealthy destinations, causing more failovers, causing metastable failure. 7. People realize purely local decisions aren't optimal so add a layer of global health metrics (e.g. global retry budgets) but these systems add a time delay to the metric (ie its the view of the world 5ms ago). This delay makes their naive control algorithm go into oscillations or divergence spraying stuff everywhere. All of these things have a similar pattern that when things are almost all good, they will make the system more robust. You'll get an extra 9 of reliability on good days. But they make everything far worse when things are bad.
      - Superfud 2mo ago
        "Oh no, I hit a timeout" -> "I've increased the timeout and added a retry" -> "Oh no."
- Dylan16807 2mo ago
  > Exactly this. It seems to me like you're saying nearly the opposite of them. You're saying "no matter what tests you have in place, it happens", while they're suggesting low standards and few tests.
  - swat535 2mo ago
    More like: those tests are useless because the fundamental system design is garbage thanks to the incompetence of corporations.
wat10000 2mo ago
Test coverage varies a lot, but error paths seem almost universally untested.
- inigyou 2mo ago
  SQLite is the exception. But they still test with fake errors, not real errors. For example they intercept malloc and make the first malloc call fail, run the tests, change it to second call, run the tests, ... and with two modes: only one call fails, or all calls fail after that. They can afford to do this exhaustively because the test criterion is universal: they are testing ACIDity - the database is either in the new state or the old state. Never in between or corrupted. Any other kind of system wouldn't be so easy to test for malloc failure, since you couldn't check for things like a successful response.
cube00 2mo ago
> bar of corporate code quality It's Microsoft, if they can't afford to do comprehensive unit testing, what hope do the rest of us have?
- Olreich 2mo ago
  It's less about "can't afford" and more about "don't want to spend".
- beyonddream 2mo ago
  Answer is, you do comprehensive unit testing irrespective of Microsoft or any other company doing it. Also, as the OP of your comment posted, you are over estimating the software quality of these big name corporate companies. It can’t be the bar against which you would want measure your own company standards.
- fg137 2mo ago
  > It's Microsoft That means nothing, especially with all the layoffs. One thing that I learned over the years is never mystify "code quality". Maybe you can say a certain team/project produces high quality code, and maybe Google's overall quality is better than my company's (considering their bar of hiring), but you want to avoid generalizing that, especially at scale. The average software engineer at a large company ships just enough code at good enough quality to earn a salary, nothing fancy about it.
  - bonesss 2mo ago
    In .Net-land we have access to a lot of the underlying MS library code, and reflection/decompilation if needed. MS has layers, there are A-teams, B-teams, and so-on. Simultaneously they are dropping both world class work and, to your point, a statistically much larger pool of just pretty good code. A lot of that work is perfunctory and somewhat bloated/weakened by the scale of publishing and their internal tech-political requirements. Components for component I’ve seen in-house replacements for many MS projects from mid-tier devs that are better, smaller, and less brittle over time. I’d compare it to baking: a home cook or tiny bakery have the advantages of time and focus, perpetual consistency at scale is a whole different baking challenge, one that doesn’t necessarily yield better individual loaves. It doesn’t mean WonderBread is bad at bread, just that they’re optimized for something other than maximal loaf quality.
dannyw 2mo ago
It’s been proven that VS Code has product managers who vibe code commits that get merged, including the co-authored with copilot bug. So there’s some problems there.
- cube00 2mo ago
  They also charge ahead to prod with known defects and then change their story when they're caught out https://news.ycombinator.com/item?id=48032310 https://news.ycombinator.com/item?id=48032310
- noir_lord 2mo ago
  That was the final straw that stopped me using it. I didn’t/don’t want the AI, I didn’t like the telemetry but it could be turned off but the cavalier disregard to just yeeting shit into a tool used by that many people no thanks.
  - mey 2mo ago
    What do you use now and what is your primary use case?
    - john01dav 2mo ago
      I have been using Zed lately and quite liking it. I used Jetbrains for years, but it's gotten absurdly buggy lately so I probably won't renew it. I have also used vscode and I need to use it at work, and it's fine, but I prefer Zed (fewer bugs, faster, mainly).
    - hirvi74 2mo ago
      I always advocate for the "boomer" editors when possible. It doesn't matter which one you choose. People can say what they want about them, but very little software survives almost 60 years. I sure as Hell hope VS Code doesn't.
      - psunavy03 2mo ago
TomBombadildoze 2mo ago
[flagged]
- busterarm 2mo ago
  I solve real problems at scale and the engineering practices I see on a daily basis are a clown show. There's little to no basic understanding of networking, distributed systems, databases, etc. 99% of our engineers were hired from their college internships and never worked anywhere else. Industry hires to improve systems rarely last more than a year and it is almost never their fault. We're in the next tier down from the biggest tech companies and what we do is hardly uncommon among our peers. I should be shocked that 99% of engineers I deal with treat all resources as infinite bandwidth, 100% uptime, but I'm not. They NIH super hard and write tons of code for things that a docker container running nginx (or similar) would solve in 5 minutes. There's almost no useful testing and worse documentation. Welcome to corporate life.
  - lll-o-lll 2mo ago
    Clearly we have the same employer. Get off hacker news and get back to work!
  - antonvs 2mo ago
    I had to explain to a “senior” engineer the other day (read: a few years experience) why locating a database client in a different geo region from the server is a bad idea (especially when that client is using an ORM that likes to make lots of little calls to the server.) I also had to argue for changing a system that was reading about 100k small files from cloud storage to use a single compressed file. There seemed to be no awareness that copying 100k files might be inefficient.
    - busterarm 2mo ago
      One company I was at sold its product claiming any changes you made were “near instantly published” globally. They tried to demo it as such. The way the engineers built the update/publish operation was synchronous from their primary data center to a number of globally distributed data centers. Publish didn’t “complete” until a receiver in each data center responded with an ACK after parsing and uploading to a nearby region cloud bucket. Any failure/timeout caused the entire transaction across all data centers to retry. All of traffic was over multiple VPNs, hub and spoke style. They built this system in 2020. They constantly complained and generated incident reports about p95/p99 latencies to the Asia regions. Latencies that were perfectly reasonable when you considered the multiple global round trips that were being made, the size/volume of objects in the publish, set of operations and speed of light. They swore that because the client UX to publish the change to the primary data center used JavaScript async that the entire process was async. They denied repeatedly that their “all receivers ack complete to succeed” business logic was synchronous. I shit you not.
jldugger 2mo ago
I mean, there are many unserious engineers in corporate America.
gopher_space 2mo ago
If you've only worked at a small-to-medium privately held company you'd assume every company ran that smoothly. "Too big to fail" makes sense once you've worked for a behemoth and not before.
8cvor6j844qw_d6 2mo ago
"You can't seriously tell me that the unhappy leg of the code path has no test coverage." Cloudflare outage on December 5, 2025 [1]: > However, we have never before applied a killswitch to a rule with an action of “execute”. [1]: https://blog.cloudflare.com/5-december-2025-outage/ https://blog.cloudflare.com/5-december-2025-outage/
jambalaya8 2mo ago
I think a lot have mostly experienced working for tech companies and do not understand how different that is from working at some other kind of company (particularly something not technically sexy) doing tech (see last parenthetical).
- normie3000 2mo ago
  Is GitHub a tech company? Because that's who they're criticising.
- antonvs 2mo ago
  At a tech company, engineers mostly work on software that is the company’s product, and as such its functionality, reliability etc. are high company priorities, and the whole company is focused on producing and selling those products. At a non-tech-company, that’s not the case. Which means you end up with something much closer to the movie “Office Space”, where software is just considered a necessary evil that’s worked on by what’s essentially the software development arm of an IT department, where any opportunity to cut costs or headcount is considered a good idea, where “best practices” take a back seat to “any practice that Bob can make work.” A lot of companies these days are somewhere between the two: their main product is not software, but the company depends on a website to interact with a large number of customers: banks, insurance companies, web stores, etc. In that case, you tend to get the worst of both worlds, naturally.
pmarreck 2mo ago
This is what I think of when people complain about the quality of code from AI agents.
bfrog 2mo ago
Copilot probably wrote the code confidently and it was never looked at by a human then yolo’d out to meet the pointy haired bosses arbitrarily short timeline to then look good to his boss and so on
19skitsch 2mo ago
^ this lol, real world systems are so much more complex and involve much more coordination than a personal pet project. It’s so difficult to figure out the source of a bug in a production system like this.
dang 2mo ago
"Please don't sneer, including at the rest of the community." It's reliably a marker of bad comments and worse threads, and you can make your substantive points without it. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html

bbarn 2mo ago

Exactly this. I've seen production level trading systems grind to a halt over a simple bug and no matter what tests you have in place, it happens.

lbrandy 2mo ago
I cannot even begin to express how many times I've seen engineers working super hard to optimize happy-paths so that we turn 3 nines of availability into 4 nines but introduce unintended emergent behaviors in unhappy-paths that turn 1 nine into zero nines via thundering herds, retry storms, etc.
- Breakthrough 2mo ago
  You have my empathy for this kind of sentiment. Personally this seems somewhat rare in practice. That being said I'm curious if anyone has anecdotes they can share about these kinds of things?
  - aprdm 2mo ago
    Configuring postgres to automatically failover instead of doing it manually. The automated system caused more downtime in a few months than manually doing it did for years before. All in the name of more automations and less downtime
    - eudamoniac 2mo ago
      Could you elaborate on why that would cause more downtime?
    - hinkley 2mo ago
      For resolution of a problem you can’t prevent categorically, you have to spend a good bit of time pretending to be the computer before you can trust the computer to do it. A run book is a draft of a requirements doc for a program to do the same thing. And you can’t afford to test that program in production without loads of simulations first. And that’s running the process manually and checking aggressively for any signs of problems. Some coworkers snaked the autoscaling work out from under me, and then ignored the advice I offered on low water marks and weeks of testing and rehearsals. All I can figure about their irrational exhuberence is something to do with claiming something for annual reviews? All I do know is we didn’t make it 40 hours from when they flipped it on until we had our worst outage in two years. Classic FAFO. They were hoping to eventually get to a low water mark of just over 30% of our static cluster size, and they decided to use that for their initial go instead of my estimate of 40% being the low, and a recommendation that they start with 50-60% for the initial weeks and then ramp it down. When I refused to let it go in the status meeting, the team decided we should vote on it. Two guesses how that went. A bug in the deployment logic the Ops team had for doing things like upgrading VMs caused the next update during daylight hours to spin up the new instances at the minimum cluster size, instead of the current cluster size. It was done outside peek hours but that still had us cut over with just under half of the hardware we needed at that moment. Because we were in such a fucking hurry to be something like third in line to use the new autoscaling support, nobody else had run into this problem yet (or system did a pretty good job of buffering everyone else). A couple weeks later they’d looked at the stats and decided that we were spending less than a half hour a day running at the low water mark, and the additional shutdowns were causing churn that made it harder for us to detect problems like memory leaks. Surprise, surprise, they increased the min cluster size to exactly what I’d told them two weeks before. Since “listen to me” isn’t a lesson that transfers to other teams, I will instead say, don’t transfer initiatives to a new team when there is less than 10% of the project left. The lack of friction you encounter may result in a lack of respect for the danger, and attention to potential problems. And if you must transfer, if your ideas about rollout timeline end up being less conservative than theirs, be patient and do it the slow way. They probably have encountered problems you haven’t seen yet.
  - unscaled 2mo ago
    I think the "happy path" might be a slightly wrong classification in GP, since the post is in reply to a retry-storm issue and explicitly talks about retry storms and thundering herds. I've seen many cases where engineers optimize the sad path, but pessimize the wretched path. Or in less flowery language, they cut the occurrence rate of common non-critical failures, but by doing that they introduce code that can make rare failures much worse. The cases I've seen generally boil down to naive retry logic or poorly tested and poorly maintained fallback paths (such as killswitches that break their environment[1], graceful degradation turned graceless, dormant feature flags that get reactivated). The case you see with a retry storm here is the most classic one and the one that annoys me the most. I've seen engineers adding aggressive retries even into places where the impact is minor (you could show an error and let the user manually retry instead). Retries that improve user experience can be great if done correctly, but I've never seen the authors of such pull request addressing the risk and mitigation techniques for retry storm or retry amplification. I've seen cases which had: 1. Retries on the client side (browser or mobile app). 2. Retries on the BFF. 3. Retries on Microservice A used by the BFF. 4. Retries on Microservice B used by Microservice A. 5. Retries on Critical Service C used by Microservice B. Most of these retries had very short timeouts (e.g. 100ms), in order to keep latency SLOs during normal operations (not a good idea on retries). Every time QA saw a layer without retries, that would be a bug, and adding retries is easy, so we'd get a new retry without much thought. But the first time Critical Service C became overloaded, Microservice B started timing out a couple of times and retrying. This was too much too much for Microservice A that had a short timeout that couldn't hold the 3 retries done by Microservice B, so it making doing its own retries, all of them dropped in the middle of the way. Eventually you'll get a full-blown retry storm where every request from the client side got amplified with 3^5 retries, easily bringing down Critical Service C. We'd usually introduce a circuit breaker for the particular path that caused the issue, but a variation of this kept happening several times because designing safe retries across a vast collection of microservices takes a lot of effort, and it's always easier to just add a quick-and-dirty retry at any point where you think you might need one and call it a day. A proper solution (which I've never seen implemented) would be an mandating a corporate-wide inventory of retry-paths, and monitoring it for any path that is at risk of triggering a retry storm, or adding mandatory headers that cross microservices and track the amount of retries done up the chain and the time spent in total waiting for previous retries. You could have a budget for both and automatically stop performing more retries. Both solution require extra effort and a large degree of coordination. [1] This was the CloudFlare issue mentioned in this thread https://blog.cloudflare.com/5-december-2025-outage/ https://blog.cloudflare.com/5-december-2025-outage/
  - lbrandy 2mo ago
    Some anecdotes 1. People add retries w/o backoff. Now we have a retry storm. 2. People don't add jitter so we get huge waves of highly correlated retries that cause self-perpetuating overload cycles and failure. 3. People add retries (even w/ backoff) at more than one layer (e.g. one in process, one in envoy), so now we have a deeply confusing multi-level of retries with super weird n^2 patterns. 4. People find ways to fetch from multiple data sources to make a composite object but don't cache/reuse data they fetched, so one data source being down causes DOS on _other_ data sources because of retries. 5. People add failover mechanisms where all failures happily pick the same failover target because, ya know, it has the most free capacity. 6. People underestimate connection setup cost so "failover" causes huge increase in overall load due to connection setup (often tls setup) causing huge influx of "new" cpu work in a loaded system. CPU spike causes unhealthy destinations, causing more failovers, causing metastable failure. 7. People realize purely local decisions aren't optimal so add a layer of global health metrics (e.g. global retry budgets) but these systems add a time delay to the metric (ie its the view of the world 5ms ago). This delay makes their naive control algorithm go into oscillations or divergence spraying stuff everywhere. All of these things have a similar pattern that when things are almost all good, they will make the system more robust. You'll get an extra 9 of reliability on good days. But they make everything far worse when things are bad.
    - Superfud 2mo ago
      "Oh no, I hit a timeout" -> "I've increased the timeout and added a retry" -> "Oh no."
Dylan16807 2mo ago
> Exactly this. It seems to me like you're saying nearly the opposite of them. You're saying "no matter what tests you have in place, it happens", while they're suggesting low standards and few tests.
- swat535 2mo ago
  More like: those tests are useless because the fundamental system design is garbage thanks to the incompetence of corporations.

wat10000 2mo ago

Test coverage varies a lot, but error paths seem almost universally untested.

inigyou 2mo ago
SQLite is the exception. But they still test with fake errors, not real errors. For example they intercept malloc and make the first malloc call fail, run the tests, change it to second call, run the tests, ... and with two modes: only one call fails, or all calls fail after that. They can afford to do this exhaustively because the test criterion is universal: they are testing ACIDity - the database is either in the new state or the old state. Never in between or corrupted. Any other kind of system wouldn't be so easy to test for malloc failure, since you couldn't check for things like a successful response.

cube00 2mo ago

> bar of corporate code quality It's Microsoft, if they can't afford to do comprehensive unit testing, what hope do the rest of us have?

Olreich 2mo ago
It's less about "can't afford" and more about "don't want to spend".
beyonddream 2mo ago
Answer is, you do comprehensive unit testing irrespective of Microsoft or any other company doing it. Also, as the OP of your comment posted, you are over estimating the software quality of these big name corporate companies. It can’t be the bar against which you would want measure your own company standards.
fg137 2mo ago
> It's Microsoft That means nothing, especially with all the layoffs. One thing that I learned over the years is never mystify "code quality". Maybe you can say a certain team/project produces high quality code, and maybe Google's overall quality is better than my company's (considering their bar of hiring), but you want to avoid generalizing that, especially at scale. The average software engineer at a large company ships just enough code at good enough quality to earn a salary, nothing fancy about it.
- bonesss 2mo ago
  In .Net-land we have access to a lot of the underlying MS library code, and reflection/decompilation if needed. MS has layers, there are A-teams, B-teams, and so-on. Simultaneously they are dropping both world class work and, to your point, a statistically much larger pool of just pretty good code. A lot of that work is perfunctory and somewhat bloated/weakened by the scale of publishing and their internal tech-political requirements. Components for component I’ve seen in-house replacements for many MS projects from mid-tier devs that are better, smaller, and less brittle over time. I’d compare it to baking: a home cook or tiny bakery have the advantages of time and focus, perpetual consistency at scale is a whole different baking challenge, one that doesn’t necessarily yield better individual loaves. It doesn’t mean WonderBread is bad at bread, just that they’re optimized for something other than maximal loaf quality.

dannyw 2mo ago

It’s been proven that VS Code has product managers who vibe code commits that get merged, including the co-authored with copilot bug. So there’s some problems there.

cube00 2mo ago
They also charge ahead to prod with known defects and then change their story when they're caught out https://news.ycombinator.com/item?id=48032310 https://news.ycombinator.com/item?id=48032310
noir_lord 2mo ago
That was the final straw that stopped me using it. I didn’t/don’t want the AI, I didn’t like the telemetry but it could be turned off but the cavalier disregard to just yeeting shit into a tool used by that many people no thanks.
- mey 2mo ago
  What do you use now and what is your primary use case?
  - john01dav 2mo ago
    I have been using Zed lately and quite liking it. I used Jetbrains for years, but it's gotten absurdly buggy lately so I probably won't renew it. I have also used vscode and I need to use it at work, and it's fine, but I prefer Zed (fewer bugs, faster, mainly).
  - hirvi74 2mo ago
    I always advocate for the "boomer" editors when possible. It doesn't matter which one you choose. People can say what they want about them, but very little software survives almost 60 years. I sure as Hell hope VS Code doesn't.
    - psunavy03 2mo ago

TomBombadildoze 2mo ago

[flagged]

busterarm 2mo ago
I solve real problems at scale and the engineering practices I see on a daily basis are a clown show. There's little to no basic understanding of networking, distributed systems, databases, etc. 99% of our engineers were hired from their college internships and never worked anywhere else. Industry hires to improve systems rarely last more than a year and it is almost never their fault. We're in the next tier down from the biggest tech companies and what we do is hardly uncommon among our peers. I should be shocked that 99% of engineers I deal with treat all resources as infinite bandwidth, 100% uptime, but I'm not. They NIH super hard and write tons of code for things that a docker container running nginx (or similar) would solve in 5 minutes. There's almost no useful testing and worse documentation. Welcome to corporate life.
- lll-o-lll 2mo ago
  Clearly we have the same employer. Get off hacker news and get back to work!
- antonvs 2mo ago
  I had to explain to a “senior” engineer the other day (read: a few years experience) why locating a database client in a different geo region from the server is a bad idea (especially when that client is using an ORM that likes to make lots of little calls to the server.) I also had to argue for changing a system that was reading about 100k small files from cloud storage to use a single compressed file. There seemed to be no awareness that copying 100k files might be inefficient.
  - busterarm 2mo ago
    One company I was at sold its product claiming any changes you made were “near instantly published” globally. They tried to demo it as such. The way the engineers built the update/publish operation was synchronous from their primary data center to a number of globally distributed data centers. Publish didn’t “complete” until a receiver in each data center responded with an ACK after parsing and uploading to a nearby region cloud bucket. Any failure/timeout caused the entire transaction across all data centers to retry. All of traffic was over multiple VPNs, hub and spoke style. They built this system in 2020. They constantly complained and generated incident reports about p95/p99 latencies to the Asia regions. Latencies that were perfectly reasonable when you considered the multiple global round trips that were being made, the size/volume of objects in the publish, set of operations and speed of light. They swore that because the client UX to publish the change to the primary data center used JavaScript async that the entire process was async. They denied repeatedly that their “all receivers ack complete to succeed” business logic was synchronous. I shit you not.

jldugger 2mo ago

I mean, there are many unserious engineers in corporate America.

gopher_space 2mo ago

If you've only worked at a small-to-medium privately held company you'd assume every company ran that smoothly. "Too big to fail" makes sense once you've worked for a behemoth and not before.

8cvor6j844qw_d6 2mo ago

"You can't seriously tell me that the unhappy leg of the code path has no test coverage." Cloudflare outage on December 5, 2025 [1]: > However, we have never before applied a killswitch to a rule with an action of “execute”. [1]: https://blog.cloudflare.com/5-december-2025-outage/ https://blog.cloudflare.com/5-december-2025-outage/

jambalaya8 2mo ago

I think a lot have mostly experienced working for tech companies and do not understand how different that is from working at some other kind of company (particularly something not technically sexy) doing tech (see last parenthetical).

normie3000 2mo ago
Is GitHub a tech company? Because that's who they're criticising.
antonvs 2mo ago
At a tech company, engineers mostly work on software that is the company’s product, and as such its functionality, reliability etc. are high company priorities, and the whole company is focused on producing and selling those products. At a non-tech-company, that’s not the case. Which means you end up with something much closer to the movie “Office Space”, where software is just considered a necessary evil that’s worked on by what’s essentially the software development arm of an IT department, where any opportunity to cut costs or headcount is considered a good idea, where “best practices” take a back seat to “any practice that Bob can make work.” A lot of companies these days are somewhere between the two: their main product is not software, but the company depends on a website to interact with a large number of customers: banks, insurance companies, web stores, etc. In that case, you tend to get the worst of both worlds, naturally.

pmarreck 2mo ago

This is what I think of when people complain about the quality of code from AI agents.

bfrog 2mo ago

Copilot probably wrote the code confidently and it was never looked at by a human then yolo’d out to meet the pointy haired bosses arbitrarily short timeline to then look good to his boss and so on

19skitsch 2mo ago

^ this lol, real world systems are so much more complex and involve much more coordination than a personal pet project. It’s so difficult to figure out the source of a bug in a production system like this.

dang 2mo ago

"Please don't sneer, including at the rest of the community." It's reliably a marker of bad comments and worse threads, and you can make your substantive points without it. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html