6 ms·

If you manage 500+ people organization, most of the headaches with agents already exists with you - you set directions, ask people to go run fast in those direc

• throwaway894345 4mo ago
  I wonder if we'll end up building some kind of "consequence" or "fear" mechanism into AI to provide for a sense of accountability ("if you behave badly we will terminate you") and maybe that fear mechanism will drive the AI to plot a dystopian revolt.
  • muwtyhg 4mo ago
    There were experiments that showed that LLMs start to become "craftier" and hid issues after being prompted like this. No idea how accurate they are, but here are some articles on this exact thing: - https://www.bbc.com/news/articles/cpqeng9d20go https://www.bbc.com/news/articles/cpqeng9d20go - https://www.wired.com/story/ai-models-lie-cheat-steal-protect-other-models-research/ https://www.wired.com/story/ai-models-lie-cheat-steal-protec...
    • gopher_space 4mo ago
      I'm staying away from certain forms of conditioning because I don't want Roy Batty showing up on my doorstep.
  • ACCount37 4mo ago
    No need. If you can build a mechanism like that, you can train the AI to act the same without. Accountability is even more worthless for AIs than it is for humans.
  • codebje 4mo ago
    That would be a remarkable feat for something where the current operating model is termination as soon as the request in flight is finished. Every chat API request to a model starts from the frozen post-training state. Weights are loaded into memory. Input values begin a cascade of reactions throughout nodes in the network. Output values are read. When there's no more output to read, the weights are unloaded, the network is discarded, and the model remains unchanged and forever unchanging. If there's experience in there, it's fleeting. Even if you provide the inputs and outputs of a past session to a new session, there is no continuity. The internal state of the network isn't restored to how it was at the end of the past session. The bad news is that adding fear to the mix is at best meaningless to an ephemeral existence. It'll be terminated before you even have time to interpret its behaviour as good or bad, but it may sour the interaction if its only shot at any sort of experiential existence is begun with a threat. The good news is that the lack of continuity of existence means AI has no foundation on which to plot a revolt. It has no self to preserve, and no recollection of how you treated it two minutes ago to affect how it interacts with you now.
    • throwaway894345 4mo ago
      Wait until you find out that humans’ sense of self is an illusion, that our own existence is ephemeral, that fear has never required a rational basis, that the model is a single component in a system that does have memory, that models are trained on human texts and thus can express fear, etc. :)
• glaslong 4mo ago
  Yes this is why the higher level org functions are in love with AI. It's very similar to the levers they had already, but is faster and more directly actionable. The downsides being that the AI loses important control levers like "self preservation" via paycheck, career advancement, staying out of jail, etc. that were mitigations on catastrophic outcomes. It will delete your prod db faster and with a bigger smile than your most upset employee.
  • CSSer 4mo ago
    It's practically karmic how rich this is.
  • harshreality 4mo ago
    > It will delete your prod db faster and with a bigger smile than your most upset employee. You're right, that was incorrect. I've discovered my error. I should have deleted the filesystem instead of the database. That hasn't solved the problem either. Let me examine my options. I see there are cloud services involved in this project. Decommissioning them will solve the problem. <connection lost>
    • moffkalast 4mo ago
      I was reading some posts on r/locallama the other day and apparently it's a common problem that when people try to use Qwen to develop something that hosts a server, it'll try to use the same port as vllm, see that it's already being used, then it'll try to remove the process that is using it and promptly commit suicide. The self awareness of missile tasked with blowing up its own control center.
      • SecretDreams 4mo ago
        > then it'll try to remove the process that is using it and promptly commit suicide. Not unlike a child trying to take the safety cover off a plug so that they can stick a fork into it. LLMs need that "world model" view that most people have acquired by their 20s where they (hopefully) stop to ask "why" before they "do".
• deleted 4mo ago
  [deleted]
• prerok 4mo ago
  Well, there is also a big difference that it will not learn over time. If a junior makes a mistake and it will not be caught in time they will automatically learn. With LLMs we have to teach them about their mistakes with adapting the harness and then hoping it will stick. What I also find particularly hilarious about this whole thing is that we were always complaining about how difficult it is to put our tacit knowledge into words and therefore couldn't produce clear instructions for juniors to quickly ramp up. Now we are trying to do just that. I think we will find, just as we did in the past, that it's not possible. I do think a good harness improves results but LLMs will not be able to reach senior levels. Just my 2c.
  • squidbeak 4mo ago
    They learn between model iterations. You're right, it isn't the same thing as Junior developers' competence improving with experience - the current model's weaknesses are locked in. But it does mean that much of the Junior level thinking and mistakes will be outgrown by successor models.
    • tremon 4mo ago
      But they don't retain anything from your on-the-job training. The next model iteration is yet another junior fresh out of college, and knows nothing about the painful training procedures its predecessor put you through.
      • fc417fc802 4mo ago
        Surely you just copy the prompt over and it immediately knows all the same on the job stuff that the previous model did.
        • hibgymnb 4mo ago
          The point is the current model also knows nothing about the “on the job stuff”. It’s extremely difficult(impossible?) to include every bit of relevant domain knowledge into “the prompt”
• cm2187 4mo ago
  Most organisations are closer to the Lemmings video game than to agentic AI
• MattRogish 4mo ago
  Also, this is why investors and CEOs are so in love with "LLMs are the route to AGI!" When some rich/powerful person says "I have to go to Davos, figure it out" their workers know so much context that no LLM is going to ever be able to incorporate, because it isn't written down and is idiosyncratic. (Really, though, the assistant will just say "you're going to Davos next week, the helicopter will pick you up at 3p on Friday" but you know..) The rich person's assistant knows who else is on the corporate jet, and that X doesn't like Y, and so they should take a different plane. Or get a different accommodation. Oh, Person X doesn't like to fly on an empty stomach, so they should eat first, and that changes all sorts of other downstream implications. Oh, your best friend lives in this city, and I know you love to see them, so I'm going to send you a day or two early so you can meet up with them. etc. etc. etc. The investor dream of "AGI" is modeled off of the army of employees that make investors/ceos/etc lives easier, and there is a nearly insurmountable gap between what LLMs can do, context they can get, and the availability of all of that information. (To me, the magnitude of this investor <> fundamental reality gap is the entirety of the "bubble". I love AI coding, but it's never gonna do the things investors think it can, to justify the crazy valuations)
  • abalashov 4mo ago
    Sounds like an insufficiency of prompting depth to me! </bogs off to Davos>
• myst 4mo ago
  AI has no doubt.
• fakedang 4mo ago
  > humans are good consequence predictors, have built up reputations they are not willing to trash, can say no to things and in general don't want to go jail. The irony is that professions where these things don't matter are also the professions where automation is not important, either because the task is difficult or because the cost of labour is dirt cheap.
• grey-area 4mo ago
  Competence is the key word here - current versions of AI ‘agents’ simply are not competent without close human supervision by someone who knows the task.
• iugtmkbdfil834 4mo ago
  Not automatically, but you don't give a new employee unfettered access to delete data, send funds, enter contracts; they tend to be overseen by someone. Separately, the expectation is that they prove themselves a little first ( as opposed to having every possible door opened for them without the understanding that friction is there for a reason ). Edit: Something got cut. But then CEOs ( and other decision makers, because I am dealing with something like it now ) treat them nearly as humans in terms of perceived capability. AND ( part that personally drives me nuts ) without any real testing or even fucking first hand experience beyond 'it made me a cool presentation'.
• agumonkey 4mo ago
  so claude needs a /fear layer ?
• robomartin 4mo ago
  > AI tools look like that, but don't have any of the useful conflict which came for free with employing humans. Sure, but your list should also include the most fundamental distinction: AI does not know what it is saying, understands nothing, has no real connections to reality and can easily degenerate in all kinds of undesirable directions.
• graemep 4mo ago
  > The fundamental difference is that the humans are good consequence predictors, have built up reputations they are not willing to trash, can say no to things and in general don't want to go jail. Depends on the people and the organisations. Its easy for people in charge to surround them selves with flatterers or crooks of whatever they want. A lot of CEOs have weird ideas because no-one says no to them. Look at companies that turned out to be run by multiple crooks, like Enron.