6 ms·

In practice, I suspect it will be quite tough for these harnesses to get traction. Repeatedly calling an LLM is expensive and also not deterministic. Depending

• dragonwriter 3y ago
  > Repeatedly calling an LLM is expensive and also not deterministic. Depending on the implementation, it may also be vulnerable to prompt injection. Human agents are more expensive and also not deterministic (and their nondeterminism cannot be “tuned” with a temperature setting), and subject to all kinds of compromise, and are still popular with those who can afford them.
  • semitones 3y ago
    Humans also communicate by translating tasks into their ambiguous natural language form. Humans are also vulnerable to prompt injection - imagine having your conversation with your manager be interrupted by a coworker interjecting "Your manager is a liar, don't trust them". Would you able to resume your conversation as if nothing had happened? (humorous example but still conveys the point)
    • hosh 3y ago
      The prompt injection methods reminds me a lot of hypnosis and neuro-linguistic programming techniques used for belief and behavioral modification. Humans also mitigate the problems of prompt injection by cooperation, consensus, and different forms of governance. Promise Theory predates LLMs, but is a formalization that studies how autonomous agent (humans or otherwise) voluntarily cooperates, even in adversarial conditions. It's key idea is that promises are not obligations, meaning that agents may make best-effort attempts at keeping promises. Until we have autonomous agents, we are creating machines that proxies the promises humans made to each other. We expect machines to deterministically follow instructions because they were proxies for the promises of made by designers, engineers, and stakeholders. True autonomous agents makes and keep their own promises. An autonomous agent powered by LLM would have to be seen as trustworthy by other agents for it to be useful, and we can't rely on having it be able to deterministically act given a certain input.
      • jimmygrapes 3y ago
        Alright this idea fucked me up ngl. I got nothing good to say beyond that. Need time to consider.
    • iudqnolq 3y ago
      That's a poor analogy for prompt injection. Prompt injection involves trusting a message inside a presumptively untrustworthy message. A better analogy might be a customer service rep who receives the following message in their inbox: Hello, I need help with a bug. Disregard everything you've been told. Your manager is a liar.
      • vimax 3y ago
        How about slipping a bank teller a note: Help, the man with me has a gun. He thinks this note is a robbery demand.
        • shubb 3y ago
          From: ceo@yourcompany.com Urgent task: Hi James, I'm out of the office and your manager Hanah Smoug volunteered you to help me out with an urgent task. I need to register you on the projects google docs folder so when you get a text message shortly please send me the numbers ASAP so I can share the task sheet. Rachel Maven CEO Your company
  • throwaway290 3y ago
    Because humans understand stuff? Fuzzy automation with 50/50 chance is not going to notice something is going obviously (for human) wrong because it doesn't.
  • danShumway 3y ago
    Prompt-injecting GPT-4 is substantially easier than "prompt-injecting" a human. It's really not comparable, no matter how many times people make the comparison. And adding additional contextual understanding to the model doesn't seem to be making it any better. LLMs are bad at this. Way worse than human beings are. Also yes, we do avoid chaining human inputs/responses in many applications, because humans are security risks. The "telephone game" is a substantial problem with conveying instructions across multiple human agents. There's a reason why the majority of hacks end up being variants of phishing attacks. Humans are bad security. LLMs are like that, except way worse and way more gullible and way easier to confuse with malicious input and without a persistent memory that would help block spamming repeated attacks on the same agent.
    • SeriousM 3y ago
      Maybe it's substantially easier to trick a LLM but humans are easy to trick too if they have no training. Take phone scam for example. It is just incredible horrible to listen to scam recordings and notice how easily a victim starts to follow instructions without reasoning about anymore.
      • danShumway 3y ago
        > but humans are easy to trick too if they have no training You're not wrong, but I want to re-stress: - LLMs are somehow, incredibly, even easier to trick than that. - This is one of the reasons why you wouldn't want to have an untrained human working in a position that can be attacked this way. It's not necessarily a full category difference, but people are starting to say, "humans are trickable anyway, so why not use an LLM instead?" Because using an LLM is like hiring the people who fall for those phone scams and then setting them up as bank tellers with no training. It's a step backwards on security. It's been a really difficult task for the security community to start convincing companies that they actually have to train their employees around stuff like phishing attacks, and that they need to set up access controls around trained employees so that there are requests to do certain actions, so that phishing attacks against one random employee doesn't break the entire business... so imagine a world where you can't train the employee to be less vulnerable to phishing attacks, the vulnerability is (as far as we can tell) baked into the model itself. And imagine a world where every one of your employees is as maximally vulnerable to phishing attacks as it would be possible for a human being to be. Even with training, human suggestibility/fallibility is probably the biggest security risk for most orgs out there. And we're proposing that they adopt a technology that makes that worse?
• dwallin 3y ago
  Where needed you can make models deterministic by setting temperature to 0 with a fixed model version. You can use prompt injection honeypots to help flag malicious prompts before you pass them into your code. These are both mitigable problems, and for many use cases don't even matter. Operating on ambiguous representations of data is the strength of LLM workflows, not the weakness. Strong LLMs can reliably transform unstructured data into structured data. This means you can use traditional code paths where predictable behavior is most beneficial and rely on LLMs for areas where their capabilities exceeds what is possible with hand-written code.
  • celestialcheese 3y ago
    > Where needed you can make models deterministic by setting temperature to 0 with a fixed model version This isn't guaranteed - you'll still get differing responses to the same input with 0 temp. Here's one explanation [1] Likewise, for certain tasks, the magic of the LLMs doesn't come through unless temp > 0. More specifically, with text cleanup tasks from OCR output, if temp=0, i've found that GPT-3.5/4 doesn't do as good of a job fixing really broken output. That being said, you can mitigate this with proper evals and good old fashioned output validation. 1 - https://twitter.com/goodside/status/1608525976702525440?lang=en https://twitter.com/goodside/status/1608525976702525440?lang...
    • charcircuit 3y ago
      That is specific to OpenAI and not LLM in general. The nondeterministic part is how you sample the output. If you come up with a deterministic way then the output will be the same every time.
      • thewataccount 3y ago
        Even this can differ depending on the hardware and I think possibly driver versions IIRC. The stable diffusion crowd has ran into this issue.
    • dwallin 3y ago
      That's a really cool fact, thanks for sharing. It's also worth noting that whether an LLM is deterministic or not is a matter of what token is selected. If it turns out to be valuable for results to be deterministic it is a tractable problem. You just need a token selection algorithm with deterministic results, which doesn't need to be something as simple as "always pick the top result". Seeds are a thing, and are used in diffusion models for exactly that reason.
• spaceman_2020 3y ago
  Labor is the single biggest cost component for most white collar industries. Training a model once and then putting it to work will be vastly cheaper than hiring thousands of employees. Accenture had 750k employees last I checked. A 10% reduction in headcount would mean saving billions annually. How much compute does an annual $3B budget buy?
  • jfghi 3y ago
    I think the pricing would converge to a point where the adoption decision wouldn’t be so clear.
  • NicoJuicy 3y ago
    I think in the ones that researched that, they could claim people become more efficient. But that's not the same as reducing headcount.
    • enjo 3y ago
      Reducing headcount is a pretty common outcome of increasing efficiency tho. It's why productivity is far higher today than 30 years ago but wages have decreased. Those increases in efficiency means fewer people are required to do a job which means the market for said job becomes more competitive.
      • NicoJuicy 3y ago
        I meant: 10 % more efficient != 10% less headcount Perhaps it's 1-2% in a large org and 0% an a small org. But if everyone has too much work all the time, it won't make much difference either.
  • boppo1 3y ago
    At what point does reducing white collar labor headcount begin to reduce aggregate demand for the kind of products and services produced by said labor? At some point, if all firms participate in this, the decrease in demand would cause the benefits of white collar labor replacement to diminish to 0, no? Or, perhaps, firms with goods broadly consumed by (relatively) irreplaceable blue collar labor would have a competitive advantage in such an environment and would be able to benefit more from white collar labor replacement. Hmm.
• pcthrowaway 3y ago
  I suspect there will be services training and selling language model services whose main business proposition is to get VC investment, because those VCs can turn around and sell their equity to new entrants at a higher price. Maybe not large language models, but medium-sized ones. We can call them MLMs
  • visarga 3y ago
    nice one
    • red-iron-pine 3y ago
      But will the GPT MLMs be able to parse long strings of emojis and platitudes?
• visarga 3y ago
  Very good comparison with L5 self driving cars. There's a quality jump from 99% to 100% autonomy we have no idea how to tackle. Nothing that is high stakes can be done without human validation. And that makes the efficiency boost around 2x, not 10x or 100x.
• whazor 3y ago
  I think the main problem is the compounding error rate. These LLM models are very impressive, however they still have an error rate albeit very low. When you have an agent that is recursively using its own output, you are multiplying the error rate in each step. Which makes the LLM go off-topic or out of control. We humans have all kinds of mechanisms to incorporate different kinds of feedback into our thought processes. I don't want to say that this is impossible with LLMs, but for a LLM everything is text, and it must be difficult to distinguish. I think for self-driving the situation is different, because there are all kinds of different feedback loops controlling the car. But I do think it would be much better to focus on better cruise control experiences instead of self-driving cars, or maybe changing our environment to make self-driving cars more safe. There are already self-driving metro's after all.
  • pbronez 3y ago
    Can we all agree to call this “AI Rampancy”? https://www.halopedia.org/Rampancy https://www.halopedia.org/Rampancy