4 ms·

Software had it way too easy for way too long. You could ship faulty code to billions without anyone blinking an eye. It was just harmless ideas after all. The

Software had it way too easy for way too long. You could ship faulty code to billions without anyone blinking an eye. It was just harmless ideas after all. The stakes are now higher with data being so important and the advent of algorithms that affect people directly. From health insurance claims, to automated trading, social media drugs and ai companions, bad code today can and does ruin lives. Software engineers, like every other engineer have to be held accountable for code they sign off and ship. Their livelihoods should be on the line.

gjsman-1000 2y ago
There are 2.8 trillion lines of code in this world. If I’m an engineer hired to work on a preexisting project, like 95% of jobs are, do I want to take liability for code I didn’t write? Or for if I make a mistake when interacting with hundreds of thousands of lines of code I also didn’t write? No. What you’re suggesting is about as plausible as the Aesop fable about mice saying they should put a bell on the cat. Sounds great, completely impossible. So what about only new code then? In that case, does old code get grandfathered in? If so, Google gets to take tens of billions of lines with them for free, while startups face the audit burden which would be insurmountable to reach a similar scale. Heck, Google does not have enough skilled labor themselves to audit it all. Also completely unfeasible. And even if, even if, some country decided to audit all the code, and even if there was enough talent and labor in this world to get it done by the next decade, what does that mean? It means all research, development, and investment just moves to China and other countries that don’t require it. Also completely unfeasible. > “Their livelihoods should be on the line.” This fundamentally relies on the subject being so demonstrably knowable and predictable, that only someone guilty of negligence or malice could possibly make a mistake. This absolutely does not apply to software development, and for the reasons above, probably never will. The moment such a requirement comes into existence, any software developer who isn’t suicidal abandons the field.
- whatever1 2y ago
  Let’s say you are a civil engineer and your calculator had a problem spitting out wrong results the day you were calculating the amount of reinforcement for the school you were designing. If the school collapses on the kids, you are going to jail in most countries . It does not matter the calculator had an issue, you chose to use it and not verify the results.
  - gjsman-1000 2y ago
    How many structural parts does a school have that need to be considered? How many iron beams? How many floors? Several thousand at most? Everything else on the BOM doesn’t matter - wallpaper isn’t a structural priority. In computer code, every last line is possibly structural. It also only takes a single missing = in the 1.2 million line codebase to kill. Comparing it to school engineering is an oversimplification. You should be comparing it to verifying the structural integrity of every skyscraper ever built; for each project.
    - aeonik 2y ago
      Chemical and elemental properties of the walls and wall paper can matter though. Leaded paint, Aresenic, flammability, weight (steel walls vs sheet rock). The complexity is still less than software though, and there are much better established standards of things that work together. Even if a screw is slightly wrong, it can still work. In software it's more like, every screw must have monocrystalline design, any grain boundary must be properly accounted for, and if not, that screw can take out an entire section of the building, possibly the whole thing, possibly the entire city.
    - whatever1 2y ago
      The claim was not that it will be easy, but since the stakes are high the buck has to stop somewhere. You cannot have Devin AI shipping crap and nobody picking up the phone when it hits the fan.
  - yuvalr1 2y ago
    When handling critical software and hardware, for example automated cars, it should be the case that it's never ever the sole responsibility of a single individual. People make mistakes, and will always do. There should be a lot of safety mechanisms existing to ensure nothing critically bad ever happens due to a bug. If this is not the case, then management is to blame, and even the state, for not ensuring high quality of critical equipment. When something like that does happen, it is very hard to know the measure of responsibility every entity holds. This will most certainly be decided in court.
  - 2y ago
- thfuran 2y ago
  >do I want to take liability I don't suppose most doctors or civil engineers want to take liability either.
- hulitu 2y ago
  > do I want to take liability for code I didn’t write? That's the problem, isn't it ? Nobody wants to assume liability. They all want just the profits. Think about, if your car will come with a disclaimer like Microsoft's EULA.
jandrewrogers 2y ago
Software engineering is not like physical engineering. I've done both and they necessarily operate from different assumptions about the world. In software, a single bit-flip or typo can lead to catastrophic failure, it is sensitive to defects in a way physical systems are not. The behavior of software is dependent on the behavior of the hardware it runs on; unlike physical engineering, the hardware for software is not engineered to the specifications of the design, it has to run on unknown hardware with unknown constraints and work correctly the first time. Physical systems are inherently resilient in the presence of many small defects, a property that physical engineering relies on greatly. Software is much less tolerant of defects, and there is a limited ability to "over-engineer" software to add safety margin, which is done all the time in physical engineering but is qualitatively more expensive in software engineering. Being broken is often a binary state in software. I've written software for high-assurance environments. Every detail of the implementation must be perfect to a degree that would make engineers with even the most perfectionist tendencies blush. Physical engineering requires nothing like this degree of perfectionism. In my experience, the vast majority of engineers are not cognitively equipped to engineer to that standard, and in physical engineering they don't have to.
- hn367621 2y ago
  > physical engineering they don't have to Totally agree with what you wrote but in any well run engineering organization one has to consider the cost/benefit of extra testing and engineering. If you're making a USB powered LED desk lamp, you get into diminishing returns fairly quickly. If you're making basically anything on JWST, less so.
  - kevin_thibedeau 2y ago
    There is no excuse for making defective products, no matter how low the profit margins are. The key why properly engineered anything works is that safety margins are baked into the design. Every real engineering field uses repeatable processes with known likelihood of success. Software "engineering" doesn't do that.
    - 2y ago
hn367621 2y ago
> You could ship faulty code to billions without anyone blinking an eye. Not all software is equivalent and there is plenty of code that gets treated with the precision you're asking for. But the ability to ship out code to billions in the blink of an eye is both the strength and the weakness of modern software. It allows few engineers to tackle very complex problems without the huge investment in man hours that more rigor would require. This keeps costs down enabling many projects that wouldn't see the light of day if the NRE was equivalent to HW. On the other hand, you get a lot of "preliminary" code in the world, for lack of a better word. At the end of the day all engineering is about balancing trade-offs and there are no right answers, only wrong ones. :)
BirAdam 2y ago
I actually think code quality has decreased over time. With the rise of high bandwidth internet, shipping fixes for faulty garbage became trivial and everyone now does so. Heck, some software is shipped before it’s even finished. Just to use Microsoft as an example, I can still install Windows 2000 and it will be rock solid running Office 2K3, and it won’t “need” much of anything. Windows 10 updates made some machines I used fail to boot.
hulitu 2y ago
> Software had it way too easy for way too long. You could ship faulty code to billions without anyone blinking an eye. It was just harmless ideas after all. Like Crowdstrike ? Or Microsoft ? > The stakes are now higher with data being so important and the advent of algorithms that affect people directly. From health insurance claims, to automated trading, social media drugs and ai companions, bad code today can and does ruin lives. See above and the UK Post scandal. Nobody gives a shit. There is no acountability for bad software. Software is not a product from a liability point of view. > Software engineers, like every other engineer have to be held accountable for code they sign off and ship. They managers more than them. Software engineers are not responsible for the software lifecycle. Somebody must test the code. Somebody must review the code.

gjsman-1000 2y ago

There are 2.8 trillion lines of code in this world. If I’m an engineer hired to work on a preexisting project, like 95% of jobs are, do I want to take liability for code I didn’t write? Or for if I make a mistake when interacting with hundreds of thousands of lines of code I also didn’t write? No. What you’re suggesting is about as plausible as the Aesop fable about mice saying they should put a bell on the cat. Sounds great, completely impossible. So what about only new code then? In that case, does old code get grandfathered in? If so, Google gets to take tens of billions of lines with them for free, while startups face the audit burden which would be insurmountable to reach a similar scale. Heck, Google does not have enough skilled labor themselves to audit it all. Also completely unfeasible. And even if, even if, some country decided to audit all the code, and even if there was enough talent and labor in this world to get it done by the next decade, what does that mean? It means all research, development, and investment just moves to China and other countries that don’t require it. Also completely unfeasible. > “Their livelihoods should be on the line.” This fundamentally relies on the subject being so demonstrably knowable and predictable, that only someone guilty of negligence or malice could possibly make a mistake. This absolutely does not apply to software development, and for the reasons above, probably never will. The moment such a requirement comes into existence, any software developer who isn’t suicidal abandons the field.

whatever1 2y ago
Let’s say you are a civil engineer and your calculator had a problem spitting out wrong results the day you were calculating the amount of reinforcement for the school you were designing. If the school collapses on the kids, you are going to jail in most countries . It does not matter the calculator had an issue, you chose to use it and not verify the results.
- gjsman-1000 2y ago
  How many structural parts does a school have that need to be considered? How many iron beams? How many floors? Several thousand at most? Everything else on the BOM doesn’t matter - wallpaper isn’t a structural priority. In computer code, every last line is possibly structural. It also only takes a single missing = in the 1.2 million line codebase to kill. Comparing it to school engineering is an oversimplification. You should be comparing it to verifying the structural integrity of every skyscraper ever built; for each project.
  - aeonik 2y ago
    Chemical and elemental properties of the walls and wall paper can matter though. Leaded paint, Aresenic, flammability, weight (steel walls vs sheet rock). The complexity is still less than software though, and there are much better established standards of things that work together. Even if a screw is slightly wrong, it can still work. In software it's more like, every screw must have monocrystalline design, any grain boundary must be properly accounted for, and if not, that screw can take out an entire section of the building, possibly the whole thing, possibly the entire city.
  - whatever1 2y ago
    The claim was not that it will be easy, but since the stakes are high the buck has to stop somewhere. You cannot have Devin AI shipping crap and nobody picking up the phone when it hits the fan.
- yuvalr1 2y ago
  When handling critical software and hardware, for example automated cars, it should be the case that it's never ever the sole responsibility of a single individual. People make mistakes, and will always do. There should be a lot of safety mechanisms existing to ensure nothing critically bad ever happens due to a bug. If this is not the case, then management is to blame, and even the state, for not ensuring high quality of critical equipment. When something like that does happen, it is very hard to know the measure of responsibility every entity holds. This will most certainly be decided in court.
- 2y ago
thfuran 2y ago
>do I want to take liability I don't suppose most doctors or civil engineers want to take liability either.
hulitu 2y ago
> do I want to take liability for code I didn’t write? That's the problem, isn't it ? Nobody wants to assume liability. They all want just the profits. Think about, if your car will come with a disclaimer like Microsoft's EULA.

jandrewrogers 2y ago

Software engineering is not like physical engineering. I've done both and they necessarily operate from different assumptions about the world. In software, a single bit-flip or typo can lead to catastrophic failure, it is sensitive to defects in a way physical systems are not. The behavior of software is dependent on the behavior of the hardware it runs on; unlike physical engineering, the hardware for software is not engineered to the specifications of the design, it has to run on unknown hardware with unknown constraints and work correctly the first time. Physical systems are inherently resilient in the presence of many small defects, a property that physical engineering relies on greatly. Software is much less tolerant of defects, and there is a limited ability to "over-engineer" software to add safety margin, which is done all the time in physical engineering but is qualitatively more expensive in software engineering. Being broken is often a binary state in software. I've written software for high-assurance environments. Every detail of the implementation must be perfect to a degree that would make engineers with even the most perfectionist tendencies blush. Physical engineering requires nothing like this degree of perfectionism. In my experience, the vast majority of engineers are not cognitively equipped to engineer to that standard, and in physical engineering they don't have to.

hn367621 2y ago
> physical engineering they don't have to Totally agree with what you wrote but in any well run engineering organization one has to consider the cost/benefit of extra testing and engineering. If you're making a USB powered LED desk lamp, you get into diminishing returns fairly quickly. If you're making basically anything on JWST, less so.
- kevin_thibedeau 2y ago
  There is no excuse for making defective products, no matter how low the profit margins are. The key why properly engineered anything works is that safety margins are baked into the design. Every real engineering field uses repeatable processes with known likelihood of success. Software "engineering" doesn't do that.
  - 2y ago

hn367621 2y ago

> You could ship faulty code to billions without anyone blinking an eye. Not all software is equivalent and there is plenty of code that gets treated with the precision you're asking for. But the ability to ship out code to billions in the blink of an eye is both the strength and the weakness of modern software. It allows few engineers to tackle very complex problems without the huge investment in man hours that more rigor would require. This keeps costs down enabling many projects that wouldn't see the light of day if the NRE was equivalent to HW. On the other hand, you get a lot of "preliminary" code in the world, for lack of a better word. At the end of the day all engineering is about balancing trade-offs and there are no right answers, only wrong ones. :)

BirAdam 2y ago

I actually think code quality has decreased over time. With the rise of high bandwidth internet, shipping fixes for faulty garbage became trivial and everyone now does so. Heck, some software is shipped before it’s even finished. Just to use Microsoft as an example, I can still install Windows 2000 and it will be rock solid running Office 2K3, and it won’t “need” much of anything. Windows 10 updates made some machines I used fail to boot.

hulitu 2y ago

> Software had it way too easy for way too long. You could ship faulty code to billions without anyone blinking an eye. It was just harmless ideas after all. Like Crowdstrike ? Or Microsoft ? > The stakes are now higher with data being so important and the advent of algorithms that affect people directly. From health insurance claims, to automated trading, social media drugs and ai companions, bad code today can and does ruin lives. See above and the UK Post scandal. Nobody gives a shit. There is no acountability for bad software. Software is not a product from a liability point of view. > Software engineers, like every other engineer have to be held accountable for code they sign off and ship. They managers more than them. Software engineers are not responsible for the software lifecycle. Somebody must test the code. Somebody must review the code.