19 ms·

Why it's hard to trust software, but you mostly have to anyway

119 points by ekr____ 2y ago

geokon 2y ago
The only person tackling the verifiable hardware side of things seems to be Bunnie Huang with his work on the Precursor If you're going to be militant and absolutist about things, that seems like the best place to start And then probably updating your software incredibly slowly at a rate that can actually be reviewed Software churn is so incredibly high that my impression is that only some core encryption algo really get scrutinized
- kfreds 2y ago
  > The only person tackling the verifiable hardware side of things seems to be Bunnie Huang with his work on the Precursor Bunnie's work is inspiring, but he is not alone. As far as verifiable hardware goes, I would argue that Tillitis TKey is more open source than the Precursor. However, they are very different products, and Precursor is a lot more complex and capable. The only reason TKey is more open than Precursor is because TKey is able to use a completely open source FPGA flow, whereas Precursor cannot.
Timber-6539 2y ago
In many dimensions the software you can trust is the one you author, compile and ship yourself. Vulnerabilities cannot be avoided only mitigated.
- n_plus_1_acc 2y ago
  I don't trust myself to do many things correctly.
- BriggyDwiggs42 2y ago
  Hey terry
- sltkr 2y ago
  Even then you are depending on the integrity of your development environment (see: Ken Thompson's compiler hack).
  - moffkalast 2y ago
    And the libraries, and the language, and the platform, and the hardware it runs on. It's blind trust all the way down.
- 7373737373 2y ago
  Vulnerabilities CAN be avoided, including in software you write yourself, by reducing the attack surface introduced by dangerous, superfluous, given-by-default https://en.wikipedia.org/wiki/Ambient_authority https://en.wikipedia.org/wiki/Ambient_authority by insisting on the usage of operating systems, virtual machines and programming languages that use https://en.wikipedia.org/wiki/Capability-based_security https://en.wikipedia.org/wiki/Capability-based_security and allow programmers to apply the https://en.wikipedia.org/wiki/Principle_of_least_privilege https://en.wikipedia.org/wiki/Principle_of_least_privilege easily, correctly and on a, if desired, ever finer-grained level. People figured this out 50 years ago but the rest of the world prefers to suffer I guess https://github.com/void4/notes/issues/41 https://github.com/void4/notes/issues/41 https://en.wikipedia.org/wiki/E_(programming_language) https://en.wikipedia.org/wiki/E_(programming_language)
missing-acumen 2y ago
While it certainly does not solve everything, the work being done with verifiable VMs is very interesting. Today's most advanced projects are able to compile pretty much arbitrary rust code into provable RISC-V programs (using SNARKs). Imo that solves a good chunk of the problem of proving to software users that what they get is what they asked for.
- jt2190 2y ago
  TIL > … zero-knowledge succinct non-interactive argument of knowledge (zkSNARK), which is a type of zero-knowledge proof system with short proofs and fast verification times. [1] [1] Microsoft Spartan: High-speed zkSNARKs without trusted setup https://github.com/microsoft/Spartan https://github.com/microsoft/Spartan
- sabas123 2y ago
  > Today's most advanced projects are able to compile pretty much arbitrary rust code into provable RISC-V programs Provable does not imply secure.
  - missing-acumen 2y ago
    Care to expand? Happy to answer your point which is interesting but I'm unsure of the dimension you are thinking of.
- yokem55 2y ago
  There's a lot of good cryptography and game theory and economic incentive alignment that can be done to constrain and limit the trust assumptions people have to make. But ultimately, all this does is redistribute and dilute those trust assumptions. It doesn't eliminate them. There is no such thing as "trustlessness".
  - missing-acumen 2y ago
    I do think there is. For instance, I can convince you that two graphs are not isomorphic while avoiding you the burden of having to do the computation yourself.
yu3zhou4 2y ago
Maybe in the future we will agree on using only standardized, verified, shared software so we can really trust software?
- lazide 2y ago
  Thank god I’ll have NSA certified chat apps to trust in the future!
  - yu3zhou4 2y ago
    I rather think about really verifiable, formally correct, openly available (maybe similar to how SQLite is governed - open source but closed to external contributions). This would take so much more effort to build and maintain this software, but would bring a reliability and trust that we don't have today. Lifespan of a typical software is very short, it's more like build and throw out mostly (data from institute of making things up - it's just my general observation over the years). We could pivot into having narrow set of specialized and trusted software. It wouldn't prevent anyone from building their own stuff. I just mean that something provably trusted would change how our systems can work (from untrusted shaky stuff to the infra we really rely on)
    - lazide 2y ago
      The FBI et al have been lobbying against even basic crypto for decades, and for backdoors. Do you think they’d be okay with that?
- deleted 2y ago
  [deleted]
BlueTemplar 2y ago
It doesn't help that this article starts with a strawman : it's like making fun of people that want political deliberations and decisions to be out in the open : "what, you don't trust representatives that you, yourself, voted for ?" "you're never going to read the transcripts anyway!"
ghjfrdghibt 2y ago
This reminds me of a short fiction story I read on HN ages ago about two programmers that find some word code in a program that turns out to be some AI hiding itself in all known compilers so when ever any software was created it was present. Can't for the life of me remember the name of the story or author though.
- reshlo 2y ago
  https://www.teamten.com/lawrence/writings/coding-machines/ https://www.teamten.com/lawrence/writings/coding-machines/
  - ghjfrdghibt 2y ago
    Succeeded where AI failed.
  - moffkalast 2y ago
    One of the great old classics.
woadwarrior01 2y ago
Lately there's been a surge in the number of open source in name only software, which hoodwink gullible (and often technical) users into downloading crapware laden binaries from their github releases page, which have little or nothing to do with the source code on the repo.
- bippihippi1 2y ago
  build from source or bust!
  - jay_kyburz 2y ago
    Do you have to read the entire source first?
    - bippihippi1 2y ago
      if a bear poops in the woods, does it make a sound?
lesuorac 2y ago
While it seems mostly about the individual level. The thing that always bugged me was that organizations seemed to fail to get a warranty on software. If you're going to be forking over millions of dollars like get a real warranty that it's going to work or spend that millions doing it yourself ... Of course, warranty still has the counter-party risk that they go out of business (probably because of all the lawsuits about a bad software ...).
- actionfromafar 2y ago
  Spending those millions doing it themselves may be much riskier. (Depending on a bunch of stuff.)
- gonzo41 2y ago
  You don't really get a warranty on heavy machinery either. Instead you get good support from the OEM. But at the end of the day you have the RTFM and deal with the muddy problem you're in. IMO, Software, and what we expect it to do is too complex to offer something like warranty.
  - MichaelZuo 2y ago
    Huh? IBM offers literal written warranties on plenty of their software products. It’s just usually bundled with expensive hardware or consulting products too.
- jasode 2y ago
  >was that organizations seemed to fail to get a warranty on software. The corporate buyer paying millions didn't "fail to get a warranty". What happened is that the market equilibrium price for that transaction for that seller-and-buyer is one that does not come with a warranty. In other words, if the software seller doesn't provide a warranty AND still able to find willing buyers, then the market price becomes "software sold without a warranty". Likewise, SpaceX sells rocket launches for companies that need to get their payloads up into orbit. SpaceX does not reimburse or provide insurance for the monetary value of the payload (satellite, etc) if it blows up. Why would companies pay millions for launches if SpaceX won't cover damages to the payload (akin to FedEX/UPS insuring monetary value of packages) ?!? Because the competitors don't cover payload reimbursements either. If you really really want to get your satellite up into orbit, you have to eat the cost if the launch destroys your satellite. The market clearing price for launching satellites is a shared risk model between buyer and seller. Someday in the future when space missions become 99.99% reliable and routine, an aerospace company may be the first to offer payload insurance as a differentiating feature to attract customers. Until then, buyers get 3rd-party coverage or self-insure. >If you're going to be forking over millions of dollars like get a real warranty that it's going to work or spend that millions doing it yourself x = price of 3rd-party software without a warranty y = price of developing in-house software (which your employee programmers also code without warranties) if (x < y) : purchase(x)
superkuh 2y ago
The real problem are the web applications and encasulated web applications (electron, etc) which download their executable code entirely anew each time you run them. They can just add something like require('fs').readFileSync(process.env.HOME + '/.ssh/id_rsa').toString() and send this to their servers, and you won't even notice that (since it doesn't require an update on client because the client is just a browser with full permissions that loads obfuscated code from their servers every time you launch it). An installed binary is much more verifiable and secure and trustworthy.
- kccqzy 2y ago
  A long time ago I had this (not very original) idea that software would be installed in /usr/bin and it would be mounted as a read-only file system. All other mount points that aren't read only like /tmp or /home ignore execute bit for all files. These days I don't think that's much of an improvement. And the problem is not just JavaScript. Python apps can also just download new code from the server and execute them. You can even do it in bash. The real problem is the fact that software that can be used locally needs to connect the vendor's server in the first place. The other real problem is that by and large desktop software is not sufficiently sandboxed and does not have effective security policy (like SELinux) to restrict their permission.
  - ninkendo 2y ago
    If an app has root (which it would need to write to /usr), then mount -o remount,rw /usr Can defeat that pretty trivially. Probably why nobody really bothers with it. /usr (and root access in general) is such a distraction from the real issue though, which is that all the stuff I care about is accessible under my account. My browser data, photos, etc etc… if a malicious app isn’t sandboxed and is running as me, it’s game over. Stopping it from writing to /usr (or other places outside my homedir) is basically meaningless.
  - superkuh 2y ago
    Setting all the exectuable and lib files in an application's directory as immutable might work: chattr +i.
spenrose 2y ago
Excellent. Wish he had cited Thompson's "Reflections on Trusting Trust" from half a century ago: * https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_ReflectionsonTrustingTrust.pdf https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_Ref...
- ncruces 2y ago
  Yeah, this many words, and not even talking about having to trust the compiler, and how hard that is.
whatever1 2y ago
Software had it way too easy for way too long. You could ship faulty code to billions without anyone blinking an eye. It was just harmless ideas after all. The stakes are now higher with data being so important and the advent of algorithms that affect people directly. From health insurance claims, to automated trading, social media drugs and ai companions, bad code today can and does ruin lives. Software engineers, like every other engineer have to be held accountable for code they sign off and ship. Their livelihoods should be on the line.
- gjsman-1000 2y ago
  There are 2.8 trillion lines of code in this world. If I’m an engineer hired to work on a preexisting project, like 95% of jobs are, do I want to take liability for code I didn’t write? Or for if I make a mistake when interacting with hundreds of thousands of lines of code I also didn’t write? No. What you’re suggesting is about as plausible as the Aesop fable about mice saying they should put a bell on the cat. Sounds great, completely impossible. So what about only new code then? In that case, does old code get grandfathered in? If so, Google gets to take tens of billions of lines with them for free, while startups face the audit burden which would be insurmountable to reach a similar scale. Heck, Google does not have enough skilled labor themselves to audit it all. Also completely unfeasible. And even if, even if, some country decided to audit all the code, and even if there was enough talent and labor in this world to get it done by the next decade, what does that mean? It means all research, development, and investment just moves to China and other countries that don’t require it. Also completely unfeasible. > “Their livelihoods should be on the line.” This fundamentally relies on the subject being so demonstrably knowable and predictable, that only someone guilty of negligence or malice could possibly make a mistake. This absolutely does not apply to software development, and for the reasons above, probably never will. The moment such a requirement comes into existence, any software developer who isn’t suicidal abandons the field.
  - whatever1 2y ago
    Let’s say you are a civil engineer and your calculator had a problem spitting out wrong results the day you were calculating the amount of reinforcement for the school you were designing. If the school collapses on the kids, you are going to jail in most countries . It does not matter the calculator had an issue, you chose to use it and not verify the results.
rkagerer 2y ago
Before the modern Cloud took shape, developers (or at least, the software they created) used to be more trustworthy. I love those classic tools from the likes of Sysinternals or Nirsoft. I didn't hesitate to give them full access to my machine, because I was confident they'd (mostly) work as expected. Although I couldn't inspect their source, I could reason about how they should behave, and the prevailing culture of the time was one where I knew the developers and myself shared a common set of expectations. Their creators didn't tend to pull stunts like quietly vacuuming all your data up to themselves. When they did want feedback, they asked you for it first. There wasn't such a potent "extract value" anti-culture, and successful companies recognized enduring value came from working in the user's best interest (eg. early Google resisted cluttering their search results). Although silos existed (like proprietary data formats), there was at least an implicit acknowledgement and expectation you retained ownership and control over the data itself. Distribution wasn't locked behind appstores. Heck, license enforcement in early Office and Windows was based on the honour system - talk about an ecosystem of trust. One way to work toward a healthier zeitgeist is to advocate tirelessly for the user at every opportunity you get, and stand by your gut feeling of what is right - even when faced with opposing headwinds.
- gjsman-1000 2y ago
  “talk about an ecosystem of trust” I’m actually looking back at the past, and realizing why app stores took over. For the developers, it was indeed an ecosystem of trust. For regular users, it was hell. The app stores, on a basic level, were absolutely in their best interest. Why does the phone, even now, have more apps than desktop? I answer that it was because users could try software for the first time, and could afford to take risks, knowing with certainty it wouldn’t steal their bank account. Users were implicitly trained on the “free” platforms to trust no one, take no risks, that .exe (or .deb) could ruin your life. For the average user, there has never been such an ecosystem of trust as there is now. That’s a sobering indictment about how a “free” or “open” platform can simultaneously be, for most people, user-hostile. Or another example: We like owning our data, knowing that Docx is ours, as you also complain above. But if you talk to many families; so many families have horror stories of digital losses. Lost photos, lost tax documents, lost memories. Apple charges $2.99/mo. and they’ll never lose it (or at least, the odds are lower than a self-inflicted disaster)? For them, the cloud has never felt so freeing.
ryukafalz 2y ago
The section titled "Verifying the Build" describes recompilation of the software with the same build toolchain and so on as a difficult task, but that's exactly what tools like Guix do for you. It's true that a build that's nondeterministic will trip you up, but if the build process is deterministic and avoids including timestamps for example then we do have tools that can ensure the build environment is consistent. But aside from that, yes, we still need to trust our software to a large degree especially on desktop operating systems. I would like to see more object capability systems start to show up so we can more effectively isolate software that we don't fully trust. (WebAssembly and WASI feel like they might be particularly interesting in that regard.)
paulnpace 2y ago
Fortunately, we have Bitcoin, which is trustless.
egypturnash 2y ago
clicks on link "image by chatgpt" I'm just gonna assume the rest of this post is also AI-generated waffle. closes tab
- chamomeal 2y ago
  Also not to nitpick but the image does not at all capture the spirit of “two children under a trench coat”. The one on the bottom is just lying on the floor lol.
mikewarot 2y ago
All of this effort is like putting Lipstick on a Pig. Imagine if we ran the electrical grid this way... with inspections, certifications, and all manner of paperwork. That world would be hell. Instead we carefully capabilities at the source, with circuit breakers, fuses, and engineering of same so that the biggest circuit breakers trip last. Capabilities based operating systems limit capabilities at the source, and never trust the application. CapROS, KeyKOS, and EROS have lead the way. I'm hopeful Hurd or Genode can be our daily driver in the future. Wouldn't it be awesome to be able to just use software without trusting it?
mwkaufma 2y ago
Another article immediately skipped for leading with GenAI image slop.
cbxjksls 2y ago
For most software, I trust the supply chain more than the developers. And I don't trust the supply chain. A big problem is user hostile software (and products in general). I'm not able to walk into a Walmart and buy a TV or walk into a dealership and buy a new car, because there are no options that aren't user hostile. Options exist, but I have to go out of my way to buy them.
localghost3000 2y ago
I’m probably gonna get downvoted into oblivion for this but did anyone else notice the kid in the trench coat has six fingers on his right hand?
rini17 2y ago
I consider myself quite promiscuous when trusting software but sometimes just can't. Seeing how signal desktop does 100MB updates every week, or the big ball of coalesced mud that is typescript compiler, made me avoid these. Why there isn't more pushback against that complexity?
- purplecats 2y ago
  interesting perspective. i suppose complex minifiers would also be an attack vector, as they don't as readily afford even eyeballing obvious deviances due to the obfuscation
- wruza 2y ago
  When something complex exists it’s usually because alternatives are worse. Would you have less issues with 10MB updates? 1MB? One megabyte is a lot of text, a good novel for a week of evening reading can be less than that.
  - mschild 2y ago
    I think the concern OP has is why a lot of the updates are so large. I use signal desktop and the UI hasn't changed in years. It begs the question what those 100mb are and whether it's actually necessary.
    - sumuyuda 2y ago
      I believe the Signal desktop app is an electron app. That’s probably why the updates are so big, has to update the bundled browser.
      - fisf 2y ago
        Yes, but that is a choice. It doesn't have to do that. Because, instead of "trusting" the update (or rather codebase) of a messenger, we now have to trust the complete browser bundle.

119 points by ekr____ 2y ago

The only person tackling the verifiable hardware side of things seems to be Bunnie Huang with his work on the Precursor If you're going to be militant and absolutist about things, that seems like the best place to start And then probably updating your software incredibly slowly at a rate that can actually be reviewed Software churn is so incredibly high that my impression is that only some core encryption algo really get scrutinized

kfreds 2y ago
> The only person tackling the verifiable hardware side of things seems to be Bunnie Huang with his work on the Precursor Bunnie's work is inspiring, but he is not alone. As far as verifiable hardware goes, I would argue that Tillitis TKey is more open source than the Precursor. However, they are very different products, and Precursor is a lot more complex and capable. The only reason TKey is more open than Precursor is because TKey is able to use a completely open source FPGA flow, whereas Precursor cannot.

Timber-6539 2y ago

In many dimensions the software you can trust is the one you author, compile and ship yourself. Vulnerabilities cannot be avoided only mitigated.

n_plus_1_acc 2y ago
I don't trust myself to do many things correctly.
BriggyDwiggs42 2y ago
Hey terry
sltkr 2y ago
Even then you are depending on the integrity of your development environment (see: Ken Thompson's compiler hack).
- moffkalast 2y ago
  And the libraries, and the language, and the platform, and the hardware it runs on. It's blind trust all the way down.
7373737373 2y ago
Vulnerabilities CAN be avoided, including in software you write yourself, by reducing the attack surface introduced by dangerous, superfluous, given-by-default https://en.wikipedia.org/wiki/Ambient_authority https://en.wikipedia.org/wiki/Ambient_authority by insisting on the usage of operating systems, virtual machines and programming languages that use https://en.wikipedia.org/wiki/Capability-based_security https://en.wikipedia.org/wiki/Capability-based_security and allow programmers to apply the https://en.wikipedia.org/wiki/Principle_of_least_privilege https://en.wikipedia.org/wiki/Principle_of_least_privilege easily, correctly and on a, if desired, ever finer-grained level. People figured this out 50 years ago but the rest of the world prefers to suffer I guess https://github.com/void4/notes/issues/41 https://github.com/void4/notes/issues/41 https://en.wikipedia.org/wiki/E_(programming_language) https://en.wikipedia.org/wiki/E_(programming_language)

missing-acumen 2y ago

While it certainly does not solve everything, the work being done with verifiable VMs is very interesting. Today's most advanced projects are able to compile pretty much arbitrary rust code into provable RISC-V programs (using SNARKs). Imo that solves a good chunk of the problem of proving to software users that what they get is what they asked for.

jt2190 2y ago
TIL > … zero-knowledge succinct non-interactive argument of knowledge (zkSNARK), which is a type of zero-knowledge proof system with short proofs and fast verification times. [1] [1] Microsoft Spartan: High-speed zkSNARKs without trusted setup https://github.com/microsoft/Spartan https://github.com/microsoft/Spartan
sabas123 2y ago
> Today's most advanced projects are able to compile pretty much arbitrary rust code into provable RISC-V programs Provable does not imply secure.
- missing-acumen 2y ago
  Care to expand? Happy to answer your point which is interesting but I'm unsure of the dimension you are thinking of.
yokem55 2y ago
There's a lot of good cryptography and game theory and economic incentive alignment that can be done to constrain and limit the trust assumptions people have to make. But ultimately, all this does is redistribute and dilute those trust assumptions. It doesn't eliminate them. There is no such thing as "trustlessness".
- missing-acumen 2y ago
  I do think there is. For instance, I can convince you that two graphs are not isomorphic while avoiding you the burden of having to do the computation yourself.

yu3zhou4 2y ago

Maybe in the future we will agree on using only standardized, verified, shared software so we can really trust software?

lazide 2y ago
Thank god I’ll have NSA certified chat apps to trust in the future!
- yu3zhou4 2y ago
  I rather think about really verifiable, formally correct, openly available (maybe similar to how SQLite is governed - open source but closed to external contributions). This would take so much more effort to build and maintain this software, but would bring a reliability and trust that we don't have today. Lifespan of a typical software is very short, it's more like build and throw out mostly (data from institute of making things up - it's just my general observation over the years). We could pivot into having narrow set of specialized and trusted software. It wouldn't prevent anyone from building their own stuff. I just mean that something provably trusted would change how our systems can work (from untrusted shaky stuff to the infra we really rely on)
  - lazide 2y ago
    The FBI et al have been lobbying against even basic crypto for decades, and for backdoors. Do you think they’d be okay with that?
deleted 2y ago
[deleted]

BlueTemplar 2y ago

It doesn't help that this article starts with a strawman : it's like making fun of people that want political deliberations and decisions to be out in the open : "what, you don't trust representatives that you, yourself, voted for ?" "you're never going to read the transcripts anyway!"

ghjfrdghibt 2y ago

This reminds me of a short fiction story I read on HN ages ago about two programmers that find some word code in a program that turns out to be some AI hiding itself in all known compilers so when ever any software was created it was present. Can't for the life of me remember the name of the story or author though.

reshlo 2y ago
https://www.teamten.com/lawrence/writings/coding-machines/ https://www.teamten.com/lawrence/writings/coding-machines/
- ghjfrdghibt 2y ago
  Succeeded where AI failed.
- moffkalast 2y ago
  One of the great old classics.

woadwarrior01 2y ago

Lately there's been a surge in the number of open source in name only software, which hoodwink gullible (and often technical) users into downloading crapware laden binaries from their github releases page, which have little or nothing to do with the source code on the repo.

bippihippi1 2y ago
build from source or bust!
- jay_kyburz 2y ago
  Do you have to read the entire source first?
  - bippihippi1 2y ago
    if a bear poops in the woods, does it make a sound?

lesuorac 2y ago

While it seems mostly about the individual level. The thing that always bugged me was that organizations seemed to fail to get a warranty on software. If you're going to be forking over millions of dollars like get a real warranty that it's going to work or spend that millions doing it yourself ... Of course, warranty still has the counter-party risk that they go out of business (probably because of all the lawsuits about a bad software ...).

actionfromafar 2y ago
Spending those millions doing it themselves may be much riskier. (Depending on a bunch of stuff.)
gonzo41 2y ago
You don't really get a warranty on heavy machinery either. Instead you get good support from the OEM. But at the end of the day you have the RTFM and deal with the muddy problem you're in. IMO, Software, and what we expect it to do is too complex to offer something like warranty.
- MichaelZuo 2y ago
  Huh? IBM offers literal written warranties on plenty of their software products. It’s just usually bundled with expensive hardware or consulting products too.
jasode 2y ago
>was that organizations seemed to fail to get a warranty on software. The corporate buyer paying millions didn't "fail to get a warranty". What happened is that the market equilibrium price for that transaction for that seller-and-buyer is one that does not come with a warranty. In other words, if the software seller doesn't provide a warranty AND still able to find willing buyers, then the market price becomes "software sold without a warranty". Likewise, SpaceX sells rocket launches for companies that need to get their payloads up into orbit. SpaceX does not reimburse or provide insurance for the monetary value of the payload (satellite, etc) if it blows up. Why would companies pay millions for launches if SpaceX won't cover damages to the payload (akin to FedEX/UPS insuring monetary value of packages) ?!? Because the competitors don't cover payload reimbursements either. If you really really want to get your satellite up into orbit, you have to eat the cost if the launch destroys your satellite. The market clearing price for launching satellites is a shared risk model between buyer and seller. Someday in the future when space missions become 99.99% reliable and routine, an aerospace company may be the first to offer payload insurance as a differentiating feature to attract customers. Until then, buyers get 3rd-party coverage or self-insure. >If you're going to be forking over millions of dollars like get a real warranty that it's going to work or spend that millions doing it yourself x = price of 3rd-party software without a warranty y = price of developing in-house software (which your employee programmers also code without warranties) if (x < y) : purchase(x)

superkuh 2y ago

The real problem are the web applications and encasulated web applications (electron, etc) which download their executable code entirely anew each time you run them. They can just add something like require('fs').readFileSync(process.env.HOME + '/.ssh/id_rsa').toString() and send this to their servers, and you won't even notice that (since it doesn't require an update on client because the client is just a browser with full permissions that loads obfuscated code from their servers every time you launch it). An installed binary is much more verifiable and secure and trustworthy.

kccqzy 2y ago
A long time ago I had this (not very original) idea that software would be installed in /usr/bin and it would be mounted as a read-only file system. All other mount points that aren't read only like /tmp or /home ignore execute bit for all files. These days I don't think that's much of an improvement. And the problem is not just JavaScript. Python apps can also just download new code from the server and execute them. You can even do it in bash. The real problem is the fact that software that can be used locally needs to connect the vendor's server in the first place. The other real problem is that by and large desktop software is not sufficiently sandboxed and does not have effective security policy (like SELinux) to restrict their permission.
- ninkendo 2y ago
  If an app has root (which it would need to write to /usr), then mount -o remount,rw /usr Can defeat that pretty trivially. Probably why nobody really bothers with it. /usr (and root access in general) is such a distraction from the real issue though, which is that all the stuff I care about is accessible under my account. My browser data, photos, etc etc… if a malicious app isn’t sandboxed and is running as me, it’s game over. Stopping it from writing to /usr (or other places outside my homedir) is basically meaningless.
- superkuh 2y ago
  Setting all the exectuable and lib files in an application's directory as immutable might work: chattr +i.

spenrose 2y ago

Excellent. Wish he had cited Thompson's "Reflections on Trusting Trust" from half a century ago: * https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_ReflectionsonTrustingTrust.pdf https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_Ref...

ncruces 2y ago
Yeah, this many words, and not even talking about having to trust the compiler, and how hard that is.

whatever1 2y ago

Software had it way too easy for way too long. You could ship faulty code to billions without anyone blinking an eye. It was just harmless ideas after all. The stakes are now higher with data being so important and the advent of algorithms that affect people directly. From health insurance claims, to automated trading, social media drugs and ai companions, bad code today can and does ruin lives. Software engineers, like every other engineer have to be held accountable for code they sign off and ship. Their livelihoods should be on the line.

gjsman-1000 2y ago
There are 2.8 trillion lines of code in this world. If I’m an engineer hired to work on a preexisting project, like 95% of jobs are, do I want to take liability for code I didn’t write? Or for if I make a mistake when interacting with hundreds of thousands of lines of code I also didn’t write? No. What you’re suggesting is about as plausible as the Aesop fable about mice saying they should put a bell on the cat. Sounds great, completely impossible. So what about only new code then? In that case, does old code get grandfathered in? If so, Google gets to take tens of billions of lines with them for free, while startups face the audit burden which would be insurmountable to reach a similar scale. Heck, Google does not have enough skilled labor themselves to audit it all. Also completely unfeasible. And even if, even if, some country decided to audit all the code, and even if there was enough talent and labor in this world to get it done by the next decade, what does that mean? It means all research, development, and investment just moves to China and other countries that don’t require it. Also completely unfeasible. > “Their livelihoods should be on the line.” This fundamentally relies on the subject being so demonstrably knowable and predictable, that only someone guilty of negligence or malice could possibly make a mistake. This absolutely does not apply to software development, and for the reasons above, probably never will. The moment such a requirement comes into existence, any software developer who isn’t suicidal abandons the field.
- whatever1 2y ago
  Let’s say you are a civil engineer and your calculator had a problem spitting out wrong results the day you were calculating the amount of reinforcement for the school you were designing. If the school collapses on the kids, you are going to jail in most countries . It does not matter the calculator had an issue, you chose to use it and not verify the results.

rkagerer 2y ago

Before the modern Cloud took shape, developers (or at least, the software they created) used to be more trustworthy. I love those classic tools from the likes of Sysinternals or Nirsoft. I didn't hesitate to give them full access to my machine, because I was confident they'd (mostly) work as expected. Although I couldn't inspect their source, I could reason about how they should behave, and the prevailing culture of the time was one where I knew the developers and myself shared a common set of expectations. Their creators didn't tend to pull stunts like quietly vacuuming all your data up to themselves. When they did want feedback, they asked you for it first. There wasn't such a potent "extract value" anti-culture, and successful companies recognized enduring value came from working in the user's best interest (eg. early Google resisted cluttering their search results). Although silos existed (like proprietary data formats), there was at least an implicit acknowledgement and expectation you retained ownership and control over the data itself. Distribution wasn't locked behind appstores. Heck, license enforcement in early Office and Windows was based on the honour system - talk about an ecosystem of trust. One way to work toward a healthier zeitgeist is to advocate tirelessly for the user at every opportunity you get, and stand by your gut feeling of what is right - even when faced with opposing headwinds.

gjsman-1000 2y ago
“talk about an ecosystem of trust” I’m actually looking back at the past, and realizing why app stores took over. For the developers, it was indeed an ecosystem of trust. For regular users, it was hell. The app stores, on a basic level, were absolutely in their best interest. Why does the phone, even now, have more apps than desktop? I answer that it was because users could try software for the first time, and could afford to take risks, knowing with certainty it wouldn’t steal their bank account. Users were implicitly trained on the “free” platforms to trust no one, take no risks, that .exe (or .deb) could ruin your life. For the average user, there has never been such an ecosystem of trust as there is now. That’s a sobering indictment about how a “free” or “open” platform can simultaneously be, for most people, user-hostile. Or another example: We like owning our data, knowing that Docx is ours, as you also complain above. But if you talk to many families; so many families have horror stories of digital losses. Lost photos, lost tax documents, lost memories. Apple charges $2.99/mo. and they’ll never lose it (or at least, the odds are lower than a self-inflicted disaster)? For them, the cloud has never felt so freeing.

ryukafalz 2y ago

The section titled "Verifying the Build" describes recompilation of the software with the same build toolchain and so on as a difficult task, but that's exactly what tools like Guix do for you. It's true that a build that's nondeterministic will trip you up, but if the build process is deterministic and avoids including timestamps for example then we do have tools that can ensure the build environment is consistent. But aside from that, yes, we still need to trust our software to a large degree especially on desktop operating systems. I would like to see more object capability systems start to show up so we can more effectively isolate software that we don't fully trust. (WebAssembly and WASI feel like they might be particularly interesting in that regard.)

paulnpace 2y ago

Fortunately, we have Bitcoin, which is trustless.

egypturnash 2y ago

clicks on link "image by chatgpt" I'm just gonna assume the rest of this post is also AI-generated waffle. closes tab

chamomeal 2y ago
Also not to nitpick but the image does not at all capture the spirit of “two children under a trench coat”. The one on the bottom is just lying on the floor lol.

mikewarot 2y ago

All of this effort is like putting Lipstick on a Pig. Imagine if we ran the electrical grid this way... with inspections, certifications, and all manner of paperwork. That world would be hell. Instead we carefully capabilities at the source, with circuit breakers, fuses, and engineering of same so that the biggest circuit breakers trip last. Capabilities based operating systems limit capabilities at the source, and never trust the application. CapROS, KeyKOS, and EROS have lead the way. I'm hopeful Hurd or Genode can be our daily driver in the future. Wouldn't it be awesome to be able to just use software without trusting it?

mwkaufma 2y ago

Another article immediately skipped for leading with GenAI image slop.

cbxjksls 2y ago

For most software, I trust the supply chain more than the developers. And I don't trust the supply chain. A big problem is user hostile software (and products in general). I'm not able to walk into a Walmart and buy a TV or walk into a dealership and buy a new car, because there are no options that aren't user hostile. Options exist, but I have to go out of my way to buy them.

localghost3000 2y ago

I’m probably gonna get downvoted into oblivion for this but did anyone else notice the kid in the trench coat has six fingers on his right hand?

rini17 2y ago

I consider myself quite promiscuous when trusting software but sometimes just can't. Seeing how signal desktop does 100MB updates every week, or the big ball of coalesced mud that is typescript compiler, made me avoid these. Why there isn't more pushback against that complexity?

purplecats 2y ago
interesting perspective. i suppose complex minifiers would also be an attack vector, as they don't as readily afford even eyeballing obvious deviances due to the obfuscation
wruza 2y ago
When something complex exists it’s usually because alternatives are worse. Would you have less issues with 10MB updates? 1MB? One megabyte is a lot of text, a good novel for a week of evening reading can be less than that.
- mschild 2y ago
  I think the concern OP has is why a lot of the updates are so large. I use signal desktop and the UI hasn't changed in years. It begs the question what those 100mb are and whether it's actually necessary.
  - sumuyuda 2y ago
    I believe the Signal desktop app is an electron app. That’s probably why the updates are so big, has to update the bundled browser.
    - fisf 2y ago
      Yes, but that is a choice. It doesn't have to do that. Because, instead of "trusting" the update (or rather codebase) of a messenger, we now have to trust the complete browser bundle.