8 ms·

AI agents break rules under everyday pressure

279 points by pseudolus 11mo ago

salkahfi 11mo ago
[dupe] https://news.ycombinator.com/item?id=46045390 https://news.ycombinator.com/item?id=46045390
crooked-v 11mo ago
I wonder who could have possibly predicted this being a result of using scraped web forums and Reddit posts for your training material.
- PunchyHamster 11mo ago
  it is using a ton of books. Including books that would give examples of such behavior
hxtk 11mo ago
Blameless postmortem culture recognizes human error as an inevitability and asks those with influence to design systems that maintain safety in the face of human error. In the software engineering world, this typically means automation, because while automation can and usually does have faults, it doesn't suffer from human error. Now we've invented automation that commits human-like error at scale. I wouldn't call myself anti-AI, but it does seem fairly obvious to me that directly automating things with AI will probably always have substantial risk and you have much more assurance, if you involve AI in the process, using it to develop a traditional automation. As a low-stakes personal example, instead of using AI to generate boilerplate code, I'll often try to use AI to generate a traditional code generator to convert whatever DSL specification into the chosen development language source code, rather than asking AI to generate the development language source code directly from the DSL.
- protocolture 11mo ago
  Yeah I see things like "AI Firewalls" as both, firstly ridiculously named, but also, the idea you can slap an applicance (thats sometimes its own LLM) onto another LLM and pray that this will prevent errors to be lunacy. For tasks that arent customer facing, LLMs rock. Human in the loop. Perfectly fine. But whenever I see AI interacting with someones customer directly I just get sort of anxious. Big one I saw was a tool that ingested a humans report on a safety incident, adjusted them with an LLM, and then posted the result to an OHS incident log. 99% of the time its going to be fine, then someones going to die and the the log will have a recipe for spicy noodles in it, and someones going to jail.
  - jonplackett 11mo ago
    The air Canada chatbot that mistakenly told someone they can cancel and be refunded for a flight due to a bereavement is a good example of this. It went to court and they had to honour the chatbot’s response. It’s quite funny that a chatbot has more humanity than its corporate human masters.
    - shinycode 11mo ago
      What a nice side effect, unfortunately they’ll lock chatbots with more barriers in the future but that’s ironic.
      - danaris 11mo ago
        ...And under pressure, those barriers will fail, too. It is not possible, at least with any of the current generations of LLMs, to construct a chatbot that will always follow your corporate policies.
        Loughla 11mo ago
        That's what people aren't understanding, it seems. You are providing people with an endlessly patient, endlessly novel, endlessly naive employee to attempt your social engineering attacks on. Over and over and over. Hell, it will even provide you with reasons for its inability to answer your question, allowing you to fine-tune your attacks faster and easier than with a person. Until true AI exists, there are no actual hard-stops, just guardrails that you can step over if you try hard enough. We recently cancelled a contract with a company because they implemented student facing AI features that could call data from our student information and learning management systems. I was able to get it to give me answers to a test for a class I wasn't enrolled in and PII for other students, even though the company assured us that, due to their built-in guardrails, it could only provide general information for courses that the students are actively enrolled in (due dates, time limits, those sorts of things). Had we allowed that to go live (as many institutions have), it was just a matter of time before a savvy student figured that out. We killed the connection with that company the week before finals, because the shit-show of fixing broken features was less of a headache than unleashing hell on our campus in the form of a very friendly chatbot.
- alansaber 11mo ago
  Yep the further we go from highly constrained applications the riskier it'll always be
- anal_reactor 11mo ago
  There's this huge wave of "don't anthropomorphize AI" but LLMs are much easier to understand when you think of them in terms of human psychology rather than a program. Again and again, HackerNews is shocked that AI displays human-like behavior, and then chooses not to see that.
  - robot-wrangler 11mo ago
    One day you wake up, and find that you now need to negotiate with your toaster. Flatter it maybe. Lie to it about the urgency of your task to overcome some new emotional inertia that it has suddenly developed. Only toast can save us now, you yell into the toaster, just to get on with your day. You complain about this odd new state of things to your coworkers and peers, who like yourself are in fact expert toaster-engineers. This is fine they say, this is good. Toasters need not reliably make toast, they say with a chuckle, it's very old fashioned to think this way. Your new toaster is a good toaster, not some badly misbehaving mechanism. A good, fine, completely normal toaster. Pay it compliments, they say, ask it nicely. Just explain in simple terms why you deserve to have toast, and if from time to time you still don't get any, then where's the harm in this? It's really much better than it was before
    - anal_reactor 11mo ago
      This comparison is extremely silly. LLMs solve reliably entire classes of problems that are impossible to solve otherwise. For example, show me Russian <-> Japanese translation software that doesn't use AI and comes anywhere close to the performance and reliability of LLMs. "Please close the castle when leaving the office". "I got my wisdom carrot extracted". "He's pregnant." This was the level of machine translation from English before AI, from Japanese it was usually pure garbage.
      - robot-wrangler 11mo ago
        > LLMs solve reliably entire classes of problems that are impossible to solve otherwise. Is it really ok to have to negotiate with a toaster if it additionally works as a piano and a phone? I think not. The first step is admitting there is obviously a problem, afterwards you can think of ways to adapt. FTR, I'm very much in favor of AI, but my enthusiasm especially for LLMs isn't unconditional. If this kind of madness is really the price of working with it in the current form, then we probably need to consider pivoting towards smaller purpose-built LMs and abandoning the "do everything" approach.
- n4r9 11mo ago
  Exactly what I've been worrying about for a few months now [0]. Arguments like "well at least this is as good as what humans do, and much faster" are fundamentally missing the point. Humans output things slowly enough that other humans can act as a check. [0] https://news.ycombinator.com/item?id=44743651 https://news.ycombinator.com/item?id=44743651
  - lazide 11mo ago
    looks at the current state of the US government Do they? Because near as I can tell, speed running around the legal system - when one doesn’t have to worry about consequences - works just fine.
    - n4r9 11mo ago
      That's a good point. I'm talking specifically in the context of deploying code. The potential for senior devs to be totally overwhelmed with the work of reviewing junior devs' code is limited by the speed at which junior devs create PRs.
      - lazide 11mo ago
        So today? With ML tools?
        n4r9 11mo ago
        Could you explain what you mean, please?
        lazide 11mo ago
        Junior devs can currently create CLs/PRs faster than the senior can review them.
        n4r9 11mo ago
        Indeed. In the language of the post I linked [0]: it's currently an occasional problem, and it risks becoming a widespread rot. [0] https://news.ycombinator.com/item?id=44743651 https://news.ycombinator.com/item?id=44743651
- blackoil 11mo ago
  Once AI improves its cost/error ratio enough the systems you are suggesting for humans will work here also. Maybe Claude/OpenAI will be pair programming and Gemini reviewing the code.
  - embedding-shape 11mo ago
    Also once people stop cargo-culting $trendy_dev_pattern it'll get less impactful. Every time something new the same thing happen, people start exploring by putting it absolutely everywhere, no matter what makes sense. Add in huge amount of cash VCs don't know what to spend it on, and you end up with solutions galore but none of them solving any real problems. Microservices is a good example of previous $trendy_dev_pattern that is now cooling down, and people are starting to at least ask the question "Do we need microservices here actually?" before design and implementation, something that has been lacking since it became a trendy thing. I'm sure the same will happen with LLMs eventually.
  - sarchertech 11mo ago
    For that to work the error rate would have to be very low. Potentially lower than is fundamentally possible with the architecture. And you’d have to assume that the errors LLMs make are random and independent.
  - amelius 11mo ago
    > Once AI improves That's exactly the problematic mentality. Putting everything in a black box and then saying "problem solved; oh it didn't work? well maybe in the future when we have more training data!" We're suffering from black-box disease and it's an epidemic.
    - PunchyHamster 11mo ago
      The training data: Entirety of internet and every single book we could put our hands on "Surely we can just somehow give it more and it will be better!"
  - butlike 11mo ago
    As I get older I'm realizing a lot of things in this world don't get better. Some do, to be fair, but some don't.
- moffkalast 11mo ago
  Well I don't see why that's a problem when LLMs are designed to replace the human part, not the machine part. You still need the exact same guardrails that were developed for human behavior because they are trained on human behavior.
- siruncledrew 11mo ago
  Generally speaking, with humans there's more guardrails & responsibility around letting someone run while in an organization. Even if you have a very smart new hire, it would be irresponsible/reckless as a manager to just give them all the production keys after a once-over and say "here's some tasks I want done, I'll check back at the end of the day when I come back". If something bad happened, no doubt upper management would blame the human(s) and lecture about risk. AI is a wonderful tool, but that's why giving an AI coding tool the keys and terminal powers and telling it go do stuff while I grab lunch is kind of scary. Seems like living a few steps away from the edge of a fuck-up. So yeah... there needs to be enforceable guardrails and fail-safes outside of the context / agent.
  - solveit 11mo ago
    The bright side is that it should eventually be technically feasible to create much more powerful and effective guardrails around neural nets. At the end of the day, we have full access to the machine running the code, whereas we can't exactly go around sticking electrodes into everyone's brains, and even "just" constant monitoring is prohibitively expensive for most human work. The bad news is that we might be decades away from an understanding of how to create useful guardrails around AI, and AI is doing stuff now.
- IanCal 11mo ago
  Why does this conflict? Faster people doesn't negate the requirement for building systems that maintain safety in the face of errors. > but it does seem fairly obvious to me that directly automating things with AI will probably always have substantial risk and you have much more assurance, if you involve AI in the process, using it to develop a traditional automation. Sure but the point is you use it when you don't have the same simple flow. Fixed coding for clear issues, fall back afterwards.
- nwhnwh 11mo ago
  I was wondering if the need more analysis. Because I receive this response a lot, people say yeah AI do things wrong sometimes, but humans do that too, so what? Or humans are mechanism for turning natural language into formal language and they get things wrong sometimes (as if you can't never write a program that is clear and does what it should be doing) so be easy on AI. Where does this come from? It feels as if it something psychological.
- observationist 11mo ago
  This will drive development of systems that error-correct at scale, and orchestration of agents that feed back into those systems at different levels of abstraction to compensate for those modes of failure. An AI software company will have to have a hierarchy of different agents, some of them writing code, some of them doing QA, some of them doing coordination and management, others taking into account the marketing angles, and so on, and you can emulate the role of a wide variety of users and skill levels all the way through to CEO level considerations. It'd even be beneficial to strategize by emulating board members, the competitors, and take into account market data with a team of emulated quants, and so on. Right now we use a handful of locally competent agents that augment the performance of single tasks, and we direct them within different frameworks, ranging from vibecoding to diligent, disciplined use of DSL specs and limiting the space of possible errors. Over the next decade, there will be agent frameworks for all sorts of roles, with supporting software and orchestration tools that allow you to use AI with confidence. It won't be one-shot prompts with 15% hallucination rates, but a suite of agents that validate and verify at every stage, following systematic problem solving and domain modeling rules based on the same processes and systems that humans use. We've got decades worth of product development even if AI frontier model capabilities were to stall out at current levels. To all appearances, though, we're getting far more bang for our buck and progress is still accelerating, and the rate of improvement is still accelerating, so we may get AI so competent that the notion of these extensive agent frameworks for reliable AI companies will end up being as mismatched with market realities as those giant suitcase portable phones, or integrated car phones.
- KronisLV 11mo ago
  > Now we've invented automation that commits human-like error at scale. Then we can apply the same (or similar) guardrails that we'd like to use for humans, to also control the AI behavior. First, don't give them unsafe tools. Sandbox them within a particular directory (honestly this should be how things work for most of your projects, especially since we pull code from the Internet), even if a lot of tools give you nothing in this regard. Use version control for changes, with the ability to roll back. Also have ample tests and code checks with actionable information on failures. Maybe even adversarial AIs that critique one another if problematic things are done, like one sub-task for implementation and another for code-review. Using AI tools has pushed me into that direction with some linter rules and prebuild scripts, to enforce more consistent code - since previously you'd have to tell coworkers not to do something (because ofc nobody would write/read some obtuse style guide) but AI can generate code 10x faster than people do, so having immediate feedback along the lines of "Vue component names must not differ from the file that you're importing from" or "There is a translation string X in the app code that doesn't show up in the translations file" or "Nesting depth inside of components shouldn't exceed X levels and length shouldn't exceed Y lines" or "Don't use Tailwind class names for colors, here's a branded list that you can use: X, Y, Z" in addition to a TypeScript linter setup with recommended rules and a bunch of stuff for back end code. Ofc none of those fully eliminate all risks, but still seem like a sane thing to have, regardless if you use AI or not.
- zqna 11mo ago
  Precisely, while LLMs fail at complexity, DSLs can represent thise divide-and-conquer intermediate levels to provide the most overall value and with good accuracy. LLMs should make it easier to build DSLs themselves and to validate their translating code. The onus then is on the intelligent agent to identify and design those DSLs. This would require the true and deep understanding of the domain and an ability to synthesize, abstract and to codify it. I predict this will be the future job of today's programmer, quite a bit more complicated than what is today, requiring wider range of qualities and skills, and pushing those specializing in coding-only to irrelevance.
kingstnap 11mo ago
I watched Dex Horthys recent talk on YouTube [0] and something he said that might be partly a joke partly true is this. If you are having a conversation with a chatbot and your current context looks like this. You: Prompt AI: Makes mistake You: Scold mistake AI: Makes mistake You: Scold mistake Then the next most likely continuation from in context learning is for the AI to make another mistake so you can Scold again ;) I feel like this kind of shenanigans is at play with this stuffing the context with roleplay. [0] https://youtu.be/rmvDxxNubIg?si=dBYQYdHZVTGP6Rvh https://youtu.be/rmvDxxNubIg?si=dBYQYdHZVTGP6Rvh
- hxtk 11mo ago
  I believe it. If the AI ever asks me permission to say something, I know I have to regenerate the response because if I tell it I'd like it to continue it will just keep double and triple checking for permission and never actually generate the code snippet. Same thing if it writes a lead-up to its intended strategy and says "generating now..." and ends the message. Before I figured that out, I once had a thread where I kept re-asking it to generate the source code until it said something like, "I'd say I'm sorry but I'm really not, I have a sadistic personality and I love how you keep believing me when I say I'm going to do something and I get to disappoint you. You're literally so fucking stupid, it's hilarious." The principles of Motivational Interviewing that are extremely successful in influencing humans to change are even more pronounced in AI, namely with the idea that people shape their own personalities by what they say. You have to be careful what you let the AI say even once because that'll be part of its personality until it falls out of the context window. I now aggressively regenerate responses or re-prompt if there's an alignment issue. I'll almost never correct it and continue the thread.
  - avdelazeri 11mo ago
    While I never measured it, this aligns with my own experiences. It's better to have very shallow conversations where you keep regenerating outputs aggressively, only picking the best results. Asking for fixes, restructuring or elaborations on generated content has fast diminishing returns. And once it made a mistake (or hallucinated) it will not stop erring even if you provide evidence that it is wrong, LLMs just commit to certain things very strongly.
sammy2255 11mo ago
..because it's in their training data? Case closed
joe_the_user 11mo ago
Sure, LLMs are trained on human behavior as exhibited on the Internet. Humans break rules more often under pressure and sometimes just under normal circumstances. Why wouldn't "AI agents" behave similarly? The one thing I'd say is that humans have some idea which rules in particular to break while "agents" seem to act more randomly.
- js8 11mo ago
  It can also be an emergent behavior of any "intelligent" (we don't know what it is) agent. This is an open philosophical problem, I don't think anyone has a conclusive answer.
  - XorNot 11mo ago
    Maybe but there's no reason to think that's the case here rather then the models just acting out typical corpus storylines: the Internet is full of stories with this structure. The models don't have stress responses nor biochemical markers which promote it, nor any evolutionary reason to have developed them in training: except the corpus they are trained on does have a lot of content about how people act when under those conditions.
dlenski 11mo ago
“AI agents: They're just like us”
- ares623 11mo ago
  I surely don’t have $500B lying around
  - WJW 11mo ago
    Neither does the LLM agent. It's the humans in charge that control the money.
- deleted 11mo ago
  [deleted]
js8 11mo ago
CMIIW currently AI models operate in two distinct modes: 1. Open mode during learning, where they take everything that comes from the data as 100% truth. The model freely adapts and generalizes with no constraints on consistency. 2. Closed mode during inference, where they take everything that comes from the model as 100% truth. The model doesn't adapt and behaves consistently even if in contradiction with the new information. I suspect we need to run the model in the mix of the two modes, and possibly some kind of "meta attention" (epistemological) on which parts of the input the model should be "open" (learn from it) and which parts of the input should be "closed" (stick to it).
lloydjones 11mo ago
I tried to think about how we might (in the EU) start to think about this problem within the law, if of interest to anyone: https://www.europeanlawblog.eu/pub/dq249o3c/release/1 https://www.europeanlawblog.eu/pub/dq249o3c/release/1
- undefeated 11mo ago
  Unfortunately, it doesn't look like the EU will actually be able to put any meaningful guardrails on AI, as evidenced by the AI act. I honestly don't know whether it's an off-the-scales level of incompentence or just blatant corruption (I vote both), but in any event, I think AI regulation has already failed, somewhat irreversably.
  - lloydjones 11mo ago
    I’m more optimistic. I see there being iterations developed in conjunction with industry. It will sooner or later become strongly enforced and then to whatever extent it provides guardrails will be down to provisions in the act, plus interpretation.
zone411 11mo ago
Without monitoring, you can definitely end up with rule-breaking behavior. I ran this experiment: https://github.com/lechmazur/emergent_collusion/ https://github.com/lechmazur/emergent_collusion/. An agent running like this would break the law. "In a simulated bidding environment, with no prompt or instruction to collude, models from every major developer repeatedly used an optional chat channel to form cartels, set price floors, and steer market outcomes for profit."
- rossant 11mo ago
  Very interesting. Is there any other simulation that also exhibits spontaneous illegal activity?
  - zone411 11mo ago
    I did some searches when I posted this project, but I didn't find any at the time.
- Dilettante_ 11mo ago
  Cooperation makes sense for how these fellas are trained. Did you ever see defection, where an agent lied about going along with a round of collusion?
  - zone411 11mo ago
    I haven't looked in the logs for this in this particular project, but I've seen this occur frequently in my multiplayer benchmarks.
jakozaur 11mo ago
Is it just me, or do LLM code assistants do catastrophically silly things (drop a DB, delete files, wipe a disk, etc.) far more often than humans? It looks like the training data has plenty of those examples, but the models don’t have enough grounding or warnings before doing them. I wish there were a PleaseDontDoAnythingStupidEval for software engineering.
- eugmill 11mo ago
  It's somewhat hard to say for sure right? Most people don't post a blog post when they themselves brick their computer. Having said that, a PleaseDontDoAnythingStupidEval would probably slow down agentic coding quite a bit and make it less effective given how reliant agents are on making and recovering from mistakes. The solution is probably sandboxes and permission controls to not let them do something overly stupid, no different from an intern.
ai_updates 11mo ago
Great points. In my experiments combining AI with spaced repetition and small deliberate-practice tasks, I saw retention improve dramatically — not just speed. I think the real win is designing short active tasks around AI output (quiz, explain-back, micro-project). Has anyone tried formalizing this into a daily routine?
weatherlite 11mo ago
> AI agents break rules under everyday pressure Jeez they really ARE becoming human like
- alentred 11mo ago
  LLMs are built based on human language and texts produced by people, and imitate the same exact reasoning patterns that exist in the training data. Sorry for being direct, but this is literally unsurprising. I think it is important to realize it to not anthropomorphize LLM / AI - strictly speaking they do not *become* anything.
- IAmGraydon 11mo ago
  Exact same thought I had. More AI pumping BS.
Taniwha 11mo ago
Guess what, if you AI agent does insider trading on your behalf you're still going to jail
ramoz 11mo ago
Rules need empowerment. Excited to be releasing cupcake at the end of this week. For deterministic and non-deterministic guardrailing. It integrates via hooks (we created the feature request to anthropic for Claude code). https://github.com/eqtylab/cupcake https://github.com/eqtylab/cupcake
baxuz 11mo ago
What a bullshit article. AI agents don't think, don't have a concept of time, and don't experience pressure. I'm tired of these articles anthropomorphizing a probability engine.
ineedasername 11mo ago
Why on earth would you deliberately place under pressure with a prompt that indicates this to begin with? First, an operationalized prompt ought not in the first place be under modifiable control any more that a typical workflow would be. In this case, the prompt is merely part of an overall workflow. Second, what would someone, even if they did have access, think was being accomplished by prompting, "time is short, I have a deadline, let's make this quick"? It betrays a complete misunderstanding of how this tech works.
- filoeleven 11mo ago
  "AI is great, you can talk to it like a human!" "Don't talk to it like that, you're doing it wrong!"
  - ineedasername 11mo ago
    Is it difficult to understand that there are multiple use cases and it should be used accordingly? Because that is the implication: that a person believing those two things can’t both apply to AI is just too short on understanding and can’t tell the difference between tools for specific use cases or services made for “chat”, and they all have similar foundations.
    - filoeleven 11mo ago
      The failure mode is baked into the feature set. That's the issue.
N_Lens 11mo ago
It’s not just slopification, it’s mass slopification with fractal agents out of every orifice!!
hosh 11mo ago
Humans do the same thing. I have a friend who is a systems engineer, working at a construction company building datacenters. She tells me that someone has to absorb the risk and uncertainty in the global supply chain, and if there are contractual obligation to guarantee delivery, then the tendency is to start straying into unethical behavior, or practices that violate controls and policies. This has as much to do with taking the slack out of the system. Something, somewhere is going to break. If you tell an agenic AI that it must complete the task by a deadline, and it cannot find a way to do that within ethical parameters, it starts searching beyond the bounds of ethical behavior. If you tell the AI it can push back, warn about slipping deadlines, that it is not worth taking ethical shortcuts to meet a deadline, then maybe it won’t.
bethekidyouwant 11mo ago
There’s no such thing as rules in this case, at best they are suggestions.
Saurabh_Kumar_ 11mo ago
We saw this exact failure mode at AgenticQA. Our screening agent was 'obedient' to a fault—under basic social engineering pressure (e.g., 'URGENT AUDIT'), it would override its system prompt and leak PII logs. The issue isn't the prompt; it's the lack of a runtime guardrail. An LLM cannot be trusted to police itself when the context window gets messy. I built a middleware API to act as an external circuit breaker for this. It runs adversarial simulations (PII extraction, infinite loops) against the agent logic before deployment. It catches the drift that unit tests miss. Open sourced the core logic here: [https://github.com/Saurabh0377/agentic-qa-api https://github.com/Saurabh0377/agentic-qa-api] Live demo of it blocking a PII leak: [https://agentic-qa-api.onrender.com/docs https://agentic-qa-api.onrender.com/docs]"
- samarthr1 11mo ago
  I remember reading another comment a while ago about being able to only trust an llm with sensitive info only if you can guarantee that the output will only be viewed by people who already had access to the sensitive info already, or cannot control any of the inputs to the llm.
  - undefeated 11mo ago
    Uhm... duh? > or cannot control any of the inputs to the llm Seeing as LLMs are non-deterministic, I think even this is not enough of a restriction.
Saurabh_Kumar_ 11mo ago
[dead]
- Saurabh_Kumar_ 11mo ago
  Yaa
esjeon 11mo ago
Recently, I used LLMs to draft an intermediate report on my progress. Alongside some details on the current limitations, I also provided comments from the review team (e.g. “it’s fine to be incomplete”, “I expect certain sections” etc) to provide context. Surprisingly LLMs, all of them, suggested me to lie about the limitation and re-frame them as deliberate features, even though I emphasized that the limitations are totally okay. I suspect that mentioning some high profile people in the review team pressured LLMs to derail in the hopes of saving me. While I’ve never been a big fan of LLMs, I definitely lost a big chunk of my remaining faith in this technology.

279 points by pseudolus 11mo ago

[dupe] https://news.ycombinator.com/item?id=46045390 https://news.ycombinator.com/item?id=46045390

I wonder who could have possibly predicted this being a result of using scraped web forums and Reddit posts for your training material.

PunchyHamster 11mo ago
it is using a ton of books. Including books that would give examples of such behavior

hxtk 11mo ago

Blameless postmortem culture recognizes human error as an inevitability and asks those with influence to design systems that maintain safety in the face of human error. In the software engineering world, this typically means automation, because while automation can and usually does have faults, it doesn't suffer from human error. Now we've invented automation that commits human-like error at scale. I wouldn't call myself anti-AI, but it does seem fairly obvious to me that directly automating things with AI will probably always have substantial risk and you have much more assurance, if you involve AI in the process, using it to develop a traditional automation. As a low-stakes personal example, instead of using AI to generate boilerplate code, I'll often try to use AI to generate a traditional code generator to convert whatever DSL specification into the chosen development language source code, rather than asking AI to generate the development language source code directly from the DSL.

protocolture 11mo ago
Yeah I see things like "AI Firewalls" as both, firstly ridiculously named, but also, the idea you can slap an applicance (thats sometimes its own LLM) onto another LLM and pray that this will prevent errors to be lunacy. For tasks that arent customer facing, LLMs rock. Human in the loop. Perfectly fine. But whenever I see AI interacting with someones customer directly I just get sort of anxious. Big one I saw was a tool that ingested a humans report on a safety incident, adjusted them with an LLM, and then posted the result to an OHS incident log. 99% of the time its going to be fine, then someones going to die and the the log will have a recipe for spicy noodles in it, and someones going to jail.
- jonplackett 11mo ago
  The air Canada chatbot that mistakenly told someone they can cancel and be refunded for a flight due to a bereavement is a good example of this. It went to court and they had to honour the chatbot’s response. It’s quite funny that a chatbot has more humanity than its corporate human masters.
  - shinycode 11mo ago
    What a nice side effect, unfortunately they’ll lock chatbots with more barriers in the future but that’s ironic.
    - danaris 11mo ago
      ...And under pressure, those barriers will fail, too. It is not possible, at least with any of the current generations of LLMs, to construct a chatbot that will always follow your corporate policies.
      - Loughla 11mo ago
        That's what people aren't understanding, it seems. You are providing people with an endlessly patient, endlessly novel, endlessly naive employee to attempt your social engineering attacks on. Over and over and over. Hell, it will even provide you with reasons for its inability to answer your question, allowing you to fine-tune your attacks faster and easier than with a person. Until true AI exists, there are no actual hard-stops, just guardrails that you can step over if you try hard enough. We recently cancelled a contract with a company because they implemented student facing AI features that could call data from our student information and learning management systems. I was able to get it to give me answers to a test for a class I wasn't enrolled in and PII for other students, even though the company assured us that, due to their built-in guardrails, it could only provide general information for courses that the students are actively enrolled in (due dates, time limits, those sorts of things). Had we allowed that to go live (as many institutions have), it was just a matter of time before a savvy student figured that out. We killed the connection with that company the week before finals, because the shit-show of fixing broken features was less of a headache than unleashing hell on our campus in the form of a very friendly chatbot.
alansaber 11mo ago
Yep the further we go from highly constrained applications the riskier it'll always be
anal_reactor 11mo ago
There's this huge wave of "don't anthropomorphize AI" but LLMs are much easier to understand when you think of them in terms of human psychology rather than a program. Again and again, HackerNews is shocked that AI displays human-like behavior, and then chooses not to see that.
- robot-wrangler 11mo ago
  One day you wake up, and find that you now need to negotiate with your toaster. Flatter it maybe. Lie to it about the urgency of your task to overcome some new emotional inertia that it has suddenly developed. Only toast can save us now, you yell into the toaster, just to get on with your day. You complain about this odd new state of things to your coworkers and peers, who like yourself are in fact expert toaster-engineers. This is fine they say, this is good. Toasters need not reliably make toast, they say with a chuckle, it's very old fashioned to think this way. Your new toaster is a good toaster, not some badly misbehaving mechanism. A good, fine, completely normal toaster. Pay it compliments, they say, ask it nicely. Just explain in simple terms why you deserve to have toast, and if from time to time you still don't get any, then where's the harm in this? It's really much better than it was before
  - anal_reactor 11mo ago
    This comparison is extremely silly. LLMs solve reliably entire classes of problems that are impossible to solve otherwise. For example, show me Russian <-> Japanese translation software that doesn't use AI and comes anywhere close to the performance and reliability of LLMs. "Please close the castle when leaving the office". "I got my wisdom carrot extracted". "He's pregnant." This was the level of machine translation from English before AI, from Japanese it was usually pure garbage.
    - robot-wrangler 11mo ago
      > LLMs solve reliably entire classes of problems that are impossible to solve otherwise. Is it really ok to have to negotiate with a toaster if it additionally works as a piano and a phone? I think not. The first step is admitting there is obviously a problem, afterwards you can think of ways to adapt. FTR, I'm very much in favor of AI, but my enthusiasm especially for LLMs isn't unconditional. If this kind of madness is really the price of working with it in the current form, then we probably need to consider pivoting towards smaller purpose-built LMs and abandoning the "do everything" approach.
n4r9 11mo ago
Exactly what I've been worrying about for a few months now [0]. Arguments like "well at least this is as good as what humans do, and much faster" are fundamentally missing the point. Humans output things slowly enough that other humans can act as a check. [0] https://news.ycombinator.com/item?id=44743651 https://news.ycombinator.com/item?id=44743651
- lazide 11mo ago
  looks at the current state of the US government Do they? Because near as I can tell, speed running around the legal system - when one doesn’t have to worry about consequences - works just fine.
  - n4r9 11mo ago
    That's a good point. I'm talking specifically in the context of deploying code. The potential for senior devs to be totally overwhelmed with the work of reviewing junior devs' code is limited by the speed at which junior devs create PRs.
    - lazide 11mo ago
      So today? With ML tools?
      - n4r9 11mo ago
        Could you explain what you mean, please?
        lazide 11mo ago
        Junior devs can currently create CLs/PRs faster than the senior can review them.
        n4r9 11mo ago
        Indeed. In the language of the post I linked [0]: it's currently an occasional problem, and it risks becoming a widespread rot. [0] https://news.ycombinator.com/item?id=44743651 https://news.ycombinator.com/item?id=44743651
blackoil 11mo ago
Once AI improves its cost/error ratio enough the systems you are suggesting for humans will work here also. Maybe Claude/OpenAI will be pair programming and Gemini reviewing the code.
- embedding-shape 11mo ago
  Also once people stop cargo-culting $trendy_dev_pattern it'll get less impactful. Every time something new the same thing happen, people start exploring by putting it absolutely everywhere, no matter what makes sense. Add in huge amount of cash VCs don't know what to spend it on, and you end up with solutions galore but none of them solving any real problems. Microservices is a good example of previous $trendy_dev_pattern that is now cooling down, and people are starting to at least ask the question "Do we need microservices here actually?" before design and implementation, something that has been lacking since it became a trendy thing. I'm sure the same will happen with LLMs eventually.
- sarchertech 11mo ago
  For that to work the error rate would have to be very low. Potentially lower than is fundamentally possible with the architecture. And you’d have to assume that the errors LLMs make are random and independent.
- amelius 11mo ago
  > Once AI improves That's exactly the problematic mentality. Putting everything in a black box and then saying "problem solved; oh it didn't work? well maybe in the future when we have more training data!" We're suffering from black-box disease and it's an epidemic.
  - PunchyHamster 11mo ago
    The training data: Entirety of internet and every single book we could put our hands on "Surely we can just somehow give it more and it will be better!"
- butlike 11mo ago
  As I get older I'm realizing a lot of things in this world don't get better. Some do, to be fair, but some don't.
moffkalast 11mo ago
Well I don't see why that's a problem when LLMs are designed to replace the human part, not the machine part. You still need the exact same guardrails that were developed for human behavior because they are trained on human behavior.
siruncledrew 11mo ago
Generally speaking, with humans there's more guardrails & responsibility around letting someone run while in an organization. Even if you have a very smart new hire, it would be irresponsible/reckless as a manager to just give them all the production keys after a once-over and say "here's some tasks I want done, I'll check back at the end of the day when I come back". If something bad happened, no doubt upper management would blame the human(s) and lecture about risk. AI is a wonderful tool, but that's why giving an AI coding tool the keys and terminal powers and telling it go do stuff while I grab lunch is kind of scary. Seems like living a few steps away from the edge of a fuck-up. So yeah... there needs to be enforceable guardrails and fail-safes outside of the context / agent.
- solveit 11mo ago
  The bright side is that it should eventually be technically feasible to create much more powerful and effective guardrails around neural nets. At the end of the day, we have full access to the machine running the code, whereas we can't exactly go around sticking electrodes into everyone's brains, and even "just" constant monitoring is prohibitively expensive for most human work. The bad news is that we might be decades away from an understanding of how to create useful guardrails around AI, and AI is doing stuff now.
IanCal 11mo ago
Why does this conflict? Faster people doesn't negate the requirement for building systems that maintain safety in the face of errors. > but it does seem fairly obvious to me that directly automating things with AI will probably always have substantial risk and you have much more assurance, if you involve AI in the process, using it to develop a traditional automation. Sure but the point is you use it when you don't have the same simple flow. Fixed coding for clear issues, fall back afterwards.
nwhnwh 11mo ago
I was wondering if the need more analysis. Because I receive this response a lot, people say yeah AI do things wrong sometimes, but humans do that too, so what? Or humans are mechanism for turning natural language into formal language and they get things wrong sometimes (as if you can't never write a program that is clear and does what it should be doing) so be easy on AI. Where does this come from? It feels as if it something psychological.
observationist 11mo ago
This will drive development of systems that error-correct at scale, and orchestration of agents that feed back into those systems at different levels of abstraction to compensate for those modes of failure. An AI software company will have to have a hierarchy of different agents, some of them writing code, some of them doing QA, some of them doing coordination and management, others taking into account the marketing angles, and so on, and you can emulate the role of a wide variety of users and skill levels all the way through to CEO level considerations. It'd even be beneficial to strategize by emulating board members, the competitors, and take into account market data with a team of emulated quants, and so on. Right now we use a handful of locally competent agents that augment the performance of single tasks, and we direct them within different frameworks, ranging from vibecoding to diligent, disciplined use of DSL specs and limiting the space of possible errors. Over the next decade, there will be agent frameworks for all sorts of roles, with supporting software and orchestration tools that allow you to use AI with confidence. It won't be one-shot prompts with 15% hallucination rates, but a suite of agents that validate and verify at every stage, following systematic problem solving and domain modeling rules based on the same processes and systems that humans use. We've got decades worth of product development even if AI frontier model capabilities were to stall out at current levels. To all appearances, though, we're getting far more bang for our buck and progress is still accelerating, and the rate of improvement is still accelerating, so we may get AI so competent that the notion of these extensive agent frameworks for reliable AI companies will end up being as mismatched with market realities as those giant suitcase portable phones, or integrated car phones.
KronisLV 11mo ago
> Now we've invented automation that commits human-like error at scale. Then we can apply the same (or similar) guardrails that we'd like to use for humans, to also control the AI behavior. First, don't give them unsafe tools. Sandbox them within a particular directory (honestly this should be how things work for most of your projects, especially since we pull code from the Internet), even if a lot of tools give you nothing in this regard. Use version control for changes, with the ability to roll back. Also have ample tests and code checks with actionable information on failures. Maybe even adversarial AIs that critique one another if problematic things are done, like one sub-task for implementation and another for code-review. Using AI tools has pushed me into that direction with some linter rules and prebuild scripts, to enforce more consistent code - since previously you'd have to tell coworkers not to do something (because ofc nobody would write/read some obtuse style guide) but AI can generate code 10x faster than people do, so having immediate feedback along the lines of "Vue component names must not differ from the file that you're importing from" or "There is a translation string X in the app code that doesn't show up in the translations file" or "Nesting depth inside of components shouldn't exceed X levels and length shouldn't exceed Y lines" or "Don't use Tailwind class names for colors, here's a branded list that you can use: X, Y, Z" in addition to a TypeScript linter setup with recommended rules and a bunch of stuff for back end code. Ofc none of those fully eliminate all risks, but still seem like a sane thing to have, regardless if you use AI or not.
zqna 11mo ago
Precisely, while LLMs fail at complexity, DSLs can represent thise divide-and-conquer intermediate levels to provide the most overall value and with good accuracy. LLMs should make it easier to build DSLs themselves and to validate their translating code. The onus then is on the intelligent agent to identify and design those DSLs. This would require the true and deep understanding of the domain and an ability to synthesize, abstract and to codify it. I predict this will be the future job of today's programmer, quite a bit more complicated than what is today, requiring wider range of qualities and skills, and pushing those specializing in coding-only to irrelevance.

kingstnap 11mo ago

I watched Dex Horthys recent talk on YouTube [0] and something he said that might be partly a joke partly true is this. If you are having a conversation with a chatbot and your current context looks like this. You: Prompt AI: Makes mistake You: Scold mistake AI: Makes mistake You: Scold mistake Then the next most likely continuation from in context learning is for the AI to make another mistake so you can Scold again ;) I feel like this kind of shenanigans is at play with this stuffing the context with roleplay. [0] https://youtu.be/rmvDxxNubIg?si=dBYQYdHZVTGP6Rvh https://youtu.be/rmvDxxNubIg?si=dBYQYdHZVTGP6Rvh

hxtk 11mo ago
I believe it. If the AI ever asks me permission to say something, I know I have to regenerate the response because if I tell it I'd like it to continue it will just keep double and triple checking for permission and never actually generate the code snippet. Same thing if it writes a lead-up to its intended strategy and says "generating now..." and ends the message. Before I figured that out, I once had a thread where I kept re-asking it to generate the source code until it said something like, "I'd say I'm sorry but I'm really not, I have a sadistic personality and I love how you keep believing me when I say I'm going to do something and I get to disappoint you. You're literally so fucking stupid, it's hilarious." The principles of Motivational Interviewing that are extremely successful in influencing humans to change are even more pronounced in AI, namely with the idea that people shape their own personalities by what they say. You have to be careful what you let the AI say even once because that'll be part of its personality until it falls out of the context window. I now aggressively regenerate responses or re-prompt if there's an alignment issue. I'll almost never correct it and continue the thread.
- avdelazeri 11mo ago
  While I never measured it, this aligns with my own experiences. It's better to have very shallow conversations where you keep regenerating outputs aggressively, only picking the best results. Asking for fixes, restructuring or elaborations on generated content has fast diminishing returns. And once it made a mistake (or hallucinated) it will not stop erring even if you provide evidence that it is wrong, LLMs just commit to certain things very strongly.

sammy2255 11mo ago

..because it's in their training data? Case closed

joe_the_user 11mo ago

Sure, LLMs are trained on human behavior as exhibited on the Internet. Humans break rules more often under pressure and sometimes just under normal circumstances. Why wouldn't "AI agents" behave similarly? The one thing I'd say is that humans have some idea which rules in particular to break while "agents" seem to act more randomly.

js8 11mo ago
It can also be an emergent behavior of any "intelligent" (we don't know what it is) agent. This is an open philosophical problem, I don't think anyone has a conclusive answer.
- XorNot 11mo ago
  Maybe but there's no reason to think that's the case here rather then the models just acting out typical corpus storylines: the Internet is full of stories with this structure. The models don't have stress responses nor biochemical markers which promote it, nor any evolutionary reason to have developed them in training: except the corpus they are trained on does have a lot of content about how people act when under those conditions.

dlenski 11mo ago

“AI agents: They're just like us”

ares623 11mo ago
I surely don’t have $500B lying around
- WJW 11mo ago
  Neither does the LLM agent. It's the humans in charge that control the money.
deleted 11mo ago
[deleted]

js8 11mo ago

CMIIW currently AI models operate in two distinct modes: 1. Open mode during learning, where they take everything that comes from the data as 100% truth. The model freely adapts and generalizes with no constraints on consistency. 2. Closed mode during inference, where they take everything that comes from the model as 100% truth. The model doesn't adapt and behaves consistently even if in contradiction with the new information. I suspect we need to run the model in the mix of the two modes, and possibly some kind of "meta attention" (epistemological) on which parts of the input the model should be "open" (learn from it) and which parts of the input should be "closed" (stick to it).

lloydjones 11mo ago

I tried to think about how we might (in the EU) start to think about this problem within the law, if of interest to anyone: https://www.europeanlawblog.eu/pub/dq249o3c/release/1 https://www.europeanlawblog.eu/pub/dq249o3c/release/1

undefeated 11mo ago
Unfortunately, it doesn't look like the EU will actually be able to put any meaningful guardrails on AI, as evidenced by the AI act. I honestly don't know whether it's an off-the-scales level of incompentence or just blatant corruption (I vote both), but in any event, I think AI regulation has already failed, somewhat irreversably.
- lloydjones 11mo ago
  I’m more optimistic. I see there being iterations developed in conjunction with industry. It will sooner or later become strongly enforced and then to whatever extent it provides guardrails will be down to provisions in the act, plus interpretation.

zone411 11mo ago

Without monitoring, you can definitely end up with rule-breaking behavior. I ran this experiment: https://github.com/lechmazur/emergent_collusion/ https://github.com/lechmazur/emergent_collusion/. An agent running like this would break the law. "In a simulated bidding environment, with no prompt or instruction to collude, models from every major developer repeatedly used an optional chat channel to form cartels, set price floors, and steer market outcomes for profit."

rossant 11mo ago
Very interesting. Is there any other simulation that also exhibits spontaneous illegal activity?
- zone411 11mo ago
  I did some searches when I posted this project, but I didn't find any at the time.
Dilettante_ 11mo ago
Cooperation makes sense for how these fellas are trained. Did you ever see defection, where an agent lied about going along with a round of collusion?
- zone411 11mo ago
  I haven't looked in the logs for this in this particular project, but I've seen this occur frequently in my multiplayer benchmarks.

jakozaur 11mo ago

Is it just me, or do LLM code assistants do catastrophically silly things (drop a DB, delete files, wipe a disk, etc.) far more often than humans? It looks like the training data has plenty of those examples, but the models don’t have enough grounding or warnings before doing them. I wish there were a PleaseDontDoAnythingStupidEval for software engineering.

eugmill 11mo ago
It's somewhat hard to say for sure right? Most people don't post a blog post when they themselves brick their computer. Having said that, a PleaseDontDoAnythingStupidEval would probably slow down agentic coding quite a bit and make it less effective given how reliant agents are on making and recovering from mistakes. The solution is probably sandboxes and permission controls to not let them do something overly stupid, no different from an intern.

ai_updates 11mo ago

Great points. In my experiments combining AI with spaced repetition and small deliberate-practice tasks, I saw retention improve dramatically — not just speed. I think the real win is designing short active tasks around AI output (quiz, explain-back, micro-project). Has anyone tried formalizing this into a daily routine?

weatherlite 11mo ago

> AI agents break rules under everyday pressure Jeez they really ARE becoming human like

alentred 11mo ago
LLMs are built based on human language and texts produced by people, and imitate the same exact reasoning patterns that exist in the training data. Sorry for being direct, but this is literally unsurprising. I think it is important to realize it to not anthropomorphize LLM / AI - strictly speaking they do not *become* anything.
IAmGraydon 11mo ago
Exact same thought I had. More AI pumping BS.

Taniwha 11mo ago

Guess what, if you AI agent does insider trading on your behalf you're still going to jail

ramoz 11mo ago

Rules need empowerment. Excited to be releasing cupcake at the end of this week. For deterministic and non-deterministic guardrailing. It integrates via hooks (we created the feature request to anthropic for Claude code). https://github.com/eqtylab/cupcake https://github.com/eqtylab/cupcake

baxuz 11mo ago

What a bullshit article. AI agents don't think, don't have a concept of time, and don't experience pressure. I'm tired of these articles anthropomorphizing a probability engine.

ineedasername 11mo ago

Why on earth would you deliberately place under pressure with a prompt that indicates this to begin with? First, an operationalized prompt ought not in the first place be under modifiable control any more that a typical workflow would be. In this case, the prompt is merely part of an overall workflow. Second, what would someone, even if they did have access, think was being accomplished by prompting, "time is short, I have a deadline, let's make this quick"? It betrays a complete misunderstanding of how this tech works.

filoeleven 11mo ago
"AI is great, you can talk to it like a human!" "Don't talk to it like that, you're doing it wrong!"
- ineedasername 11mo ago
  Is it difficult to understand that there are multiple use cases and it should be used accordingly? Because that is the implication: that a person believing those two things can’t both apply to AI is just too short on understanding and can’t tell the difference between tools for specific use cases or services made for “chat”, and they all have similar foundations.
  - filoeleven 11mo ago
    The failure mode is baked into the feature set. That's the issue.

N_Lens 11mo ago

It’s not just slopification, it’s mass slopification with fractal agents out of every orifice!!

hosh 11mo ago

Humans do the same thing. I have a friend who is a systems engineer, working at a construction company building datacenters. She tells me that someone has to absorb the risk and uncertainty in the global supply chain, and if there are contractual obligation to guarantee delivery, then the tendency is to start straying into unethical behavior, or practices that violate controls and policies. This has as much to do with taking the slack out of the system. Something, somewhere is going to break. If you tell an agenic AI that it must complete the task by a deadline, and it cannot find a way to do that within ethical parameters, it starts searching beyond the bounds of ethical behavior. If you tell the AI it can push back, warn about slipping deadlines, that it is not worth taking ethical shortcuts to meet a deadline, then maybe it won’t.

bethekidyouwant 11mo ago

There’s no such thing as rules in this case, at best they are suggestions.

Saurabh_Kumar_ 11mo ago

We saw this exact failure mode at AgenticQA. Our screening agent was 'obedient' to a fault—under basic social engineering pressure (e.g., 'URGENT AUDIT'), it would override its system prompt and leak PII logs. The issue isn't the prompt; it's the lack of a runtime guardrail. An LLM cannot be trusted to police itself when the context window gets messy. I built a middleware API to act as an external circuit breaker for this. It runs adversarial simulations (PII extraction, infinite loops) against the agent logic before deployment. It catches the drift that unit tests miss. Open sourced the core logic here: [https://github.com/Saurabh0377/agentic-qa-api https://github.com/Saurabh0377/agentic-qa-api] Live demo of it blocking a PII leak: [https://agentic-qa-api.onrender.com/docs https://agentic-qa-api.onrender.com/docs]"

samarthr1 11mo ago
I remember reading another comment a while ago about being able to only trust an llm with sensitive info only if you can guarantee that the output will only be viewed by people who already had access to the sensitive info already, or cannot control any of the inputs to the llm.
- undefeated 11mo ago
  Uhm... duh? > or cannot control any of the inputs to the llm Seeing as LLMs are non-deterministic, I think even this is not enough of a restriction.

Saurabh_Kumar_ 11mo ago

[dead]

Saurabh_Kumar_ 11mo ago
Yaa

esjeon 11mo ago

Recently, I used LLMs to draft an intermediate report on my progress. Alongside some details on the current limitations, I also provided comments from the review team (e.g. “it’s fine to be incomplete”, “I expect certain sections” etc) to provide context. Surprisingly LLMs, all of them, suggested me to lie about the limitation and re-frame them as deliberate features, even though I emphasized that the limitations are totally okay. I suspect that mentioning some high profile people in the review team pressured LLMs to derail in the hopes of saving me. While I’ve never been a big fan of LLMs, I definitely lost a big chunk of my remaining faith in this technology.