4 ms·

What everyone seems to forget is that "trusting trust" has a counter: https://www.dwheeler.com/trusting-trust/ https://www.dwheeler.com/trusting-trust/

• carapace 9y ago
  Yes, but... You have to have at least two independent compiler stack development processes occurring in separate light-cones. If one happens far enough within the cone of the other you cannot trust it. (Not actually speed-of-light-cone of course. You have the lead-time required to develop general "nanites" and then their travel time to reach the opposite side of the Earth (assuming no one is working on this off-planet. The first thing a paranoid nanotech-haver would does is detect and suborn all other nanotech labs. I call this the "Matter-Lock".)
  • Jach 9y ago
    That goes a bit beyond the "trusting trust" scenario, though, and into subsuming "hardware" rather than "compiler software", which DDC admits it doesn't control for... I don't have enough backing physics to be confident on the possibility of a "matter lock". How do you detect without being detected a nanomachine that was designed vs one of the nanomachines that already exist in living organisms? Can you also expand on the goal of subsuming all the nanotech of other labs? If things obviously start breaking, that's especially going away from the "trusted trust" scenario which implies things like trojans used to passively sniff secrets and gain advantage through information when an opportunity to export the information arises. I would expect physics might allow some workarounds for that, on top of methods of outright detection like conservation laws, spectrometry... I've finished the first of three in a sci-fi book series that has introduced the problem of what to do with an adversary that subsumes basic physics research to halt general advances, perhaps there's an entertaining hard sci-fi book you'd recommend for the "matter lock" idea? Or arxiv papers if you have any.
    • carapace 9y ago
      The problem with nanotech is that unlike software it can transmit itself. That's what make the "trusting trust" problem so severe in this context. On the scale of hypothetical nanites the world is really really huge, so the first hurdle is figuring out how to integrate the incoming information and control the machines. > How do you detect without being detected a nanomachine that was designed vs one of the nanomachines that already exist in living organisms? If there are already other machines to detect then you're too late and the scenario is "toner war" as per "Diamond Age" (probably the best nanotech sci-fi novel; or maybe "Blood Music" by Greg Bear.) If you do get there first (and you've correctly identified this as a huge existential challenge: how can you know that you're not being fed false information by the person who got there before you? You can't. If "matter lock" is possible there's no way to know if you're really first, except to try some shit and see if anyone notices and can stop you) then you have the relatively easy task of locating the other nanotech labs in the world and infecting them with your malware. > Can you also expand on the goal of subsuming all the nanotech of other labs? Well, if you're reading "The Three Body Problem" then that's one way. Eventually some people would start to get wise. But nanotech: you detect them and alter their brains to forget. There's always another way to contain the information if you get there early enough. It would be easy to infect the other labs because you would be infecting every lab everywhere already. And of course, you can always just declare yourself. Wear a purple silk cape and call yourself the Robot King. Who's going to stop you? Anyhow, if you wanted to keep your "matter lock" a secret you would have to minimize your interventions, restrict yourself to subtle sabotage, and program every instrument to ignore the fact that every computer and robot in the entire world had a massive Trojan in it. More than that, to actively lie about it and alert you if anyone starts doing weird experiments. Even then I suspect things would come to a head somehow and... and then I don't know what would happen. > perhaps there's an entertaining hard sci-fi book you'd recommend for the "matter lock" idea? Nah. There is one novel about a megalomaniacal mad scientist who achieves "matter lock" and immediately begins editing the world as he pleases. It's grotesque. FWIW it's called "The Goliath Stone" by Matthew Joseph Harrington with some sort of involvement of Larry Niven (who is otherwise one of my favorite authors, but this book is a stinker.) Just one example: the mad scientist is violently opposed to rape (okay) but he makes womens' breasts larger without asking them. I do recommend these if you haven't read them already: https://en.wikipedia.org/wiki/The_Diamond_Age https://en.wikipedia.org/wiki/The_Diamond_Age https://en.wikipedia.org/wiki/Blood_Music_(novel) https://en.wikipedia.org/wiki/Blood_Music_(novel) > Or arxiv papers if you have any. No. People working on this do not publish. ;-)