4 ms·

You trust machine code because it was produced by a deterministic system that was written and tested thoroughly by other engineers. AI systems are not the same

You trust machine code because it was produced by a deterministic system that was written and tested thoroughly by other engineers. AI systems are not the same. You can't guarantee deterministic output. Also, code is still the best language we have to convey our intentions as engineers into function. Natural language is too imprecise and ambiguous. Understanding the code allows us to understand the intention behind the code and identify bugs, plan and architect better. Inspecting prompts gives you a blinkered view of the system meaning you're more likely to make incorrect assumptions leading to serious bugs.

Terr_ 22d ago
Right, it's a chain of trust between the layers. The link I choose next to me helps determine the quality of successive links, whether I inspect them or not. When I buy a wooden shack from a carpenter, I'd never expect them to core-sample every plank to ensure it's good wood. That's because a good carpenter habitually buys wood from a good supplier, and will have the experience to detect other suspicious indicators. Similarly, my coworker might not have verified the machine code, but I can be pretty confident they didn't vibe-install a compiler by Ken Thompson that puts backdoors into the binary.
faangguyindia 22d ago
Are your teammates deterministic output machines?
- atomicnumber3 22d ago
  In a sense yes. I trust that my coworkers will be accountable for their outcomes and accomplish them with the high quality bar I know they intrinsically hold themselves to. And that they understand our shared goals and if they don't, they will work to become aligned. And if I can't say that for a coworker, well, that's performance feedback.
  - faangguyindia 22d ago
    Highly likely, your trust is just based on your feelings, and they don't do much for the product quality.
  - scrollaway 22d ago
    Your coworkers are if anything less reliably deterministic than an AI. And the human version of a prompt injection is called social engineering and it’s far more of an issue than actual prompt injection and has been for an extremely long time.
    - atomicnumber3 22d ago
      I don't know what to say except that this has not been the case for me. Claude is a different person for every single prompt and has absolutely no core sense of what our goals are, or even who the "our" is that's having goals. While my coworkers are very stable personalities of a consistent work drive, sense of pragmatism, things that they find more and less interesting, career aspirations, etc. they're not all the exact same, and I depend on that as I fit tasks and ownership to people.
      - scrollaway 21d ago
        Ok but why are you comparing different Claude sessions then? Of course they’re different, they have different experiences. Just because they have the same “name” doesn’t make them the same persona. Compare a single session. It’ll be extremely consistent.
    - icedchai 21d ago
      Prompt injection is often not the issue: it's prompt quality. If given a vague spec (prompt), the human is more likely to understand the problem domain, see what's missing, ask clarifying questions. AI is more likely to assume and take a probable path, which may not always be correct. It's (probably) not going to tell you no, what you're asking makes no sense.
- lolakutty 22d ago
  At least the tools they used to use used to be so...Now the tools do this https://blog.coredump.cx/p/recursion-into-madness https://blog.coredump.cx/p/recursion-into-madness
nobodywillobsrv 22d ago
But in chemical science you would not "understand" everything. You cook with ingredients where you don't fully understand then. There are always limits. Engineering human types are often guilty of saying something is low cost but then not doing it themselves. Great, "understand everything". Trust us trust. You trust an entity. What the author is describing is a new kind of transaction. It's not that AI causes people to lie suddenly (although if you trusted the wrong people this might happen to you, but YOU were wrong then) but now you have a lot of work being done more at the edge of uncertainty and recommendations. Ideas are not ideas anymore, they are working implementations. So instead of submitting discussions or requests for features, you literally submit implemented features because the cost is lower. This is good. You just have to realize that you are starting the discussion with a working implementation. If you continually try to act like this is not the case, without someone literally lying to you about it, this is more about just accepting a now workflow.
theoreticalmal 22d ago
What about a team of human coders is more deterministic than an AI coder?

Terr_ 22d ago

Right, it's a chain of trust between the layers. The link I choose next to me helps determine the quality of successive links, whether I inspect them or not. When I buy a wooden shack from a carpenter, I'd never expect them to core-sample every plank to ensure it's good wood. That's because a good carpenter habitually buys wood from a good supplier, and will have the experience to detect other suspicious indicators. Similarly, my coworker might not have verified the machine code, but I can be pretty confident they didn't vibe-install a compiler by Ken Thompson that puts backdoors into the binary.

faangguyindia 22d ago

Are your teammates deterministic output machines?

atomicnumber3 22d ago
In a sense yes. I trust that my coworkers will be accountable for their outcomes and accomplish them with the high quality bar I know they intrinsically hold themselves to. And that they understand our shared goals and if they don't, they will work to become aligned. And if I can't say that for a coworker, well, that's performance feedback.
- faangguyindia 22d ago
  Highly likely, your trust is just based on your feelings, and they don't do much for the product quality.
- scrollaway 22d ago
  Your coworkers are if anything less reliably deterministic than an AI. And the human version of a prompt injection is called social engineering and it’s far more of an issue than actual prompt injection and has been for an extremely long time.
  - atomicnumber3 22d ago
    I don't know what to say except that this has not been the case for me. Claude is a different person for every single prompt and has absolutely no core sense of what our goals are, or even who the "our" is that's having goals. While my coworkers are very stable personalities of a consistent work drive, sense of pragmatism, things that they find more and less interesting, career aspirations, etc. they're not all the exact same, and I depend on that as I fit tasks and ownership to people.
    - scrollaway 21d ago
      Ok but why are you comparing different Claude sessions then? Of course they’re different, they have different experiences. Just because they have the same “name” doesn’t make them the same persona. Compare a single session. It’ll be extremely consistent.
  - icedchai 21d ago
    Prompt injection is often not the issue: it's prompt quality. If given a vague spec (prompt), the human is more likely to understand the problem domain, see what's missing, ask clarifying questions. AI is more likely to assume and take a probable path, which may not always be correct. It's (probably) not going to tell you no, what you're asking makes no sense.
lolakutty 22d ago
At least the tools they used to use used to be so...Now the tools do this https://blog.coredump.cx/p/recursion-into-madness https://blog.coredump.cx/p/recursion-into-madness

nobodywillobsrv 22d ago

But in chemical science you would not "understand" everything. You cook with ingredients where you don't fully understand then. There are always limits. Engineering human types are often guilty of saying something is low cost but then not doing it themselves. Great, "understand everything". Trust us trust. You trust an entity. What the author is describing is a new kind of transaction. It's not that AI causes people to lie suddenly (although if you trusted the wrong people this might happen to you, but YOU were wrong then) but now you have a lot of work being done more at the edge of uncertainty and recommendations. Ideas are not ideas anymore, they are working implementations. So instead of submitting discussions or requests for features, you literally submit implemented features because the cost is lower. This is good. You just have to realize that you are starting the discussion with a working implementation. If you continually try to act like this is not the case, without someone literally lying to you about it, this is more about just accepting a now workflow.

theoreticalmal 22d ago

What about a team of human coders is more deterministic than an AI coder?