5 ms·

Nobody cares what your code looks like. Nobody cares what your architecture looks like. … until it breaks. Nobody cared how the FAA NOTAM database was impl

Nobody cares what your code looks like. Nobody cares what your architecture looks like. … until it breaks. Nobody cared how the FAA NOTAM database was implemented either, until it went down. Part of being a professional engineer is thinking about these things, so your users won't have to.

AnIdiotOnTheNet 4y ago
Forgive me if I'm wrong about this, but it would seem to me that the FAA NOTAM system had orders of magnitude more uptime than literally any product produced by so called "professional" "engineers" at big software companies.
- alfalfasprout 4y ago
  I think the point still stands but in a different way. The architecture was designed to be quite reliable.
- nfRfqX5n 4y ago
  AWS uptime is pretty insane
  - barbazoo 4y ago
    According to their own status page, yes
  - ILike42Cookies 4y ago
    yes, some part of aws somewhere is always online. fully functional? 7.12.2021 alone was enough to kill that statement for the next few years.
- discussDev 4y ago
  If it wasn't then who built it? Or is it the quotes we should focus on.
- bluGill 4y ago
  Sure, but we do make intentional tradeoffs, at least sometimes.
- mardifoufs 4y ago
  More than google.com? Maybe, but it's close. Also, I'd be surprised if those behind NOTAM were all "professional engineers" since software PEs are pretty rare, relatively speaking. Especially in the US.
  - hbrn 4y ago
    You're comparing database to a service. NOTAM is real-time, google.com is not. By that standard google.com is always down.
- strken 4y ago
  It would seem to me that the FAA NOTAM system had professional engineers working on it to produce that uptime.
pixl97 4y ago
Yep, this here. I work with a product that sees a massive difference in use between the low end customers and the very highest end of customers. For the low end customers it doesn't matter what your architecture is at all. Could be a single small machine in a closet and it would suffice. For our large customers it matters 100%. They are doing throughput 4 to 5 orders of magnitude higher than our most basic customers per day. They will push NAS and database performance to the limit. Every time we think "eh, no one is going to do that much" our customers come back and prove us wrong. Databases expand to billions and billions of rows. Query performance falls apart. Design decisions lead to lack of ability of horizontal scaling. And when the systems get to this size the costs of operating the system add up quickly, and if your competitors choices lead to much lower operational costs, then expect to lose customers to them, especially as your product equalizes in capability, or at least meets their set of needs.
- limitedsupply 4y ago
  The argument is that nobody cares if you use a single file with 10K LOC of outdated JavaScript code or a beautiful Rust library to compute whatever. Of course customers care about performance - if your app is unusably slow it is going to be an issue.
hbrn 4y ago
The risk of focusing on code and architecture too much is that it will never break. Not because it's very reliable: because you have no users. Part of being a professional engineer is recognizing that your knowledge is always limited. No matter how much you think about these things, they will always break. Being too careful is as bad as being reckless. Instead of trying to prevent failure, accept that it's inevitable and make sure you are fast at detecting and fixing.
taeric 4y ago
While not wrong, everything breaks at scale. Everything. Anyone that says otherwise is just not interested in talking about where what they built will break. And note that this is more than just scale of use. It is also scale of development. Keep adding changes to something, and it will break. Stability of application requiring stability of development is just not something we care to admit, that often. That is, don't hide when/why something will break. But also don't get blinded looking for how to avoid all breakage. If you can, find ways to isolate failures and block off entire sections.
- selcuka 4y ago
  > everything breaks at scale This is good advice. You should be aware of who you are building for. Designing your system to run on 100 containers orchestrated by Kubernetes is overkill for your average SaaS MVP, while counting on shell scripts is a recipe for disaster at FAANG scale. Scale matters.
  - taeric 4y ago
    Don't just look at the execution you are building for, but also the team that will be maintaining it. That is, there are several values for "who" that you are building for. Each will have cause for you to reach to more complicated seeming solutions, as you go.
adrianmonk 4y ago
Nobody cares if your warehouse is neat and clean. Your customers never go in there. They just receive orders that are shipped out of it. But if your warehouse is a mess, if the aisles of your warehouse are blocked by crates strewn everywhere (that you've been meaning to put away for ages), then it does matter if your warehouse is neat and clean because it takes forever to maneuver your forklift in to retrieve products to ship them out. If your warehouse is disorganized, and you can't find anything when you need it, then it matters. The real trouble, I think, is when people have reasons to choose not to keep the warehouse organized and functioning smoothly (like workers who see it as drudgery or managers who want workers working on something else) and they try to excuse it by saying, "Our customers never visit." (I don't think the person above is doing that, incidentally. I think they're just agreeing with the keep your eyes on the prize message.)
- lll-o-lll 4y ago
  The difference between software and most other forms of engineering is that the internals of software are invisible to everyone but the developers. A manager walks in to the warehouse you describe above and goes “what in the literal hell? No one goes home until we clean this $&@$ up!”. A machine that has been poorly designed looks like a piece crap. Software, if the UI looks nice the massive tangled mess inside is completely invisible to customer and manager alike. Gross inefficiencies, evil hacks, everything entangled with everything? No one can see it. “Why does it take 3 months for a tiny feature?” “Technical debt”, “ok, you can have 5% of your time budget to sort that out”… If people could really see the software, viscerally, they would approve budgets to fix it.
  - ignoramous 4y ago
    > If people could really see the software, viscerally, they would approve budgets to fix it. Tools for this exist; ex: https://semgrep.dev/docs/writing-rules/data-flow/data-flow-overview/ https://semgrep.dev/docs/writing-rules/data-flow/data-flow-o... I suspect the information overload for any substantially large code base would be such that, non-engineers would look right past the complexity after the enthusiasm wears off.
  - 4y ago
nunez 4y ago
yeah but that's borrowing trouble. we can spend ages solving every theoretical. and to be fair NOTAMs were rock solid until that happened, so if anything this failure was a good thing long-term
toast0 4y ago
> … until it breaks. My boss once told me, "[toast0], you can't have that kind of attitude if your shit doesn't work" Rather than change my attitude, I made sure my shit worked.
unity1001 4y ago
For each case like NOTAM database going down, there are tens of thousands of applications running on the web and outside the web without people obsessing about its architecture.
PaybackTony 4y ago
My point on this isn't that you should write ugly, unmaintainable code because who cares. It's that, if you're an experienced and intelligent engineer, you're going to do just fine. There's no need to complicate it, get lost in analysis paralysis and so forth. I have seen top engineering teams make software that's hard to maintain with questionable performance as a _result_ of some of the things this article brings up. Often times too much engineering is just as bad as not enough. Here is how I personally define good software engineering: - Does it do the job? - Is there low cognitive load in regards to following the code? (Can you jump back in the code in 6 months and get your feet quickly... AKA: Don't over-abstract). - Is it performant? - Can you easily make changes? If those 4 things are true, the rest doesn't matter from my experience.

AnIdiotOnTheNet 4y ago

Forgive me if I'm wrong about this, but it would seem to me that the FAA NOTAM system had orders of magnitude more uptime than literally any product produced by so called "professional" "engineers" at big software companies.

alfalfasprout 4y ago
I think the point still stands but in a different way. The architecture was designed to be quite reliable.
nfRfqX5n 4y ago
AWS uptime is pretty insane
- barbazoo 4y ago
  According to their own status page, yes
- ILike42Cookies 4y ago
  yes, some part of aws somewhere is always online. fully functional? 7.12.2021 alone was enough to kill that statement for the next few years.
discussDev 4y ago
If it wasn't then who built it? Or is it the quotes we should focus on.
bluGill 4y ago
Sure, but we do make intentional tradeoffs, at least sometimes.
mardifoufs 4y ago
More than google.com? Maybe, but it's close. Also, I'd be surprised if those behind NOTAM were all "professional engineers" since software PEs are pretty rare, relatively speaking. Especially in the US.
- hbrn 4y ago
  You're comparing database to a service. NOTAM is real-time, google.com is not. By that standard google.com is always down.
strken 4y ago
It would seem to me that the FAA NOTAM system had professional engineers working on it to produce that uptime.

pixl97 4y ago

Yep, this here. I work with a product that sees a massive difference in use between the low end customers and the very highest end of customers. For the low end customers it doesn't matter what your architecture is at all. Could be a single small machine in a closet and it would suffice. For our large customers it matters 100%. They are doing throughput 4 to 5 orders of magnitude higher than our most basic customers per day. They will push NAS and database performance to the limit. Every time we think "eh, no one is going to do that much" our customers come back and prove us wrong. Databases expand to billions and billions of rows. Query performance falls apart. Design decisions lead to lack of ability of horizontal scaling. And when the systems get to this size the costs of operating the system add up quickly, and if your competitors choices lead to much lower operational costs, then expect to lose customers to them, especially as your product equalizes in capability, or at least meets their set of needs.

limitedsupply 4y ago
The argument is that nobody cares if you use a single file with 10K LOC of outdated JavaScript code or a beautiful Rust library to compute whatever. Of course customers care about performance - if your app is unusably slow it is going to be an issue.

hbrn 4y ago

The risk of focusing on code and architecture too much is that it will never break. Not because it's very reliable: because you have no users. Part of being a professional engineer is recognizing that your knowledge is always limited. No matter how much you think about these things, they will always break. Being too careful is as bad as being reckless. Instead of trying to prevent failure, accept that it's inevitable and make sure you are fast at detecting and fixing.

taeric 4y ago

While not wrong, everything breaks at scale. Everything. Anyone that says otherwise is just not interested in talking about where what they built will break. And note that this is more than just scale of use. It is also scale of development. Keep adding changes to something, and it will break. Stability of application requiring stability of development is just not something we care to admit, that often. That is, don't hide when/why something will break. But also don't get blinded looking for how to avoid all breakage. If you can, find ways to isolate failures and block off entire sections.

selcuka 4y ago
> everything breaks at scale This is good advice. You should be aware of who you are building for. Designing your system to run on 100 containers orchestrated by Kubernetes is overkill for your average SaaS MVP, while counting on shell scripts is a recipe for disaster at FAANG scale. Scale matters.
- taeric 4y ago
  Don't just look at the execution you are building for, but also the team that will be maintaining it. That is, there are several values for "who" that you are building for. Each will have cause for you to reach to more complicated seeming solutions, as you go.

adrianmonk 4y ago

Nobody cares if your warehouse is neat and clean. Your customers never go in there. They just receive orders that are shipped out of it. But if your warehouse is a mess, if the aisles of your warehouse are blocked by crates strewn everywhere (that you've been meaning to put away for ages), then it does matter if your warehouse is neat and clean because it takes forever to maneuver your forklift in to retrieve products to ship them out. If your warehouse is disorganized, and you can't find anything when you need it, then it matters. The real trouble, I think, is when people have reasons to choose not to keep the warehouse organized and functioning smoothly (like workers who see it as drudgery or managers who want workers working on something else) and they try to excuse it by saying, "Our customers never visit." (I don't think the person above is doing that, incidentally. I think they're just agreeing with the keep your eyes on the prize message.)

lll-o-lll 4y ago
The difference between software and most other forms of engineering is that the internals of software are invisible to everyone but the developers. A manager walks in to the warehouse you describe above and goes “what in the literal hell? No one goes home until we clean this $&@$ up!”. A machine that has been poorly designed looks like a piece crap. Software, if the UI looks nice the massive tangled mess inside is completely invisible to customer and manager alike. Gross inefficiencies, evil hacks, everything entangled with everything? No one can see it. “Why does it take 3 months for a tiny feature?” “Technical debt”, “ok, you can have 5% of your time budget to sort that out”… If people could really see the software, viscerally, they would approve budgets to fix it.
- ignoramous 4y ago
  > If people could really see the software, viscerally, they would approve budgets to fix it. Tools for this exist; ex: https://semgrep.dev/docs/writing-rules/data-flow/data-flow-overview/ https://semgrep.dev/docs/writing-rules/data-flow/data-flow-o... I suspect the information overload for any substantially large code base would be such that, non-engineers would look right past the complexity after the enthusiasm wears off.
- 4y ago

nunez 4y ago

yeah but that's borrowing trouble. we can spend ages solving every theoretical. and to be fair NOTAMs were rock solid until that happened, so if anything this failure was a good thing long-term

toast0 4y ago

> … until it breaks. My boss once told me, "[toast0], you can't have that kind of attitude if your shit doesn't work" Rather than change my attitude, I made sure my shit worked.

unity1001 4y ago

For each case like NOTAM database going down, there are tens of thousands of applications running on the web and outside the web without people obsessing about its architecture.

PaybackTony 4y ago

My point on this isn't that you should write ugly, unmaintainable code because who cares. It's that, if you're an experienced and intelligent engineer, you're going to do just fine. There's no need to complicate it, get lost in analysis paralysis and so forth. I have seen top engineering teams make software that's hard to maintain with questionable performance as a _result_ of some of the things this article brings up. Often times too much engineering is just as bad as not enough. Here is how I personally define good software engineering: - Does it do the job? - Is there low cognitive load in regards to following the code? (Can you jump back in the code in 6 months and get your feet quickly... AKA: Don't over-abstract). - Is it performant? - Can you easily make changes? If those 4 things are true, the rest doesn't matter from my experience.