3 ms·

An agent with full access to your codebase is able to email out without supervision in response to an untrusted messsage?

embedding-shape 1mo ago
Surely passing untrusted input to a agent with execution capabilities and also possibility to reply to the same author, couldn't possibly be used for anything negative? Parent is probably using a firewall so it's A-OK :thumbs_up:
- dormento 1mo ago
  What a wild ride huh. We're in the "fuck it we ball" era. Not even the many reports of prompt injection and takeover due to IA misfeatures can cheer me up anymore. It is all incredibly sad.
- KronisLV 1mo ago
  > Surely passing untrusted input to a agent with execution capabilities Oh hey, I know this one! It’s humans and a phishing test, they’ll click on all the links and enter information without checking the domain properly! Personally, I’d like systems that aren’t open to attack and can be depended upon. But it seems like nowadays NOTHING can be trusted - not OSes (recent Qubes OS exploit, not even mentioning others), not any software written in languages without memory safety, not even the ones with (Log4j comes to mind), not the packages in many package managers, not other humans and sure as hell not the token prediction machines. What a world. We all probably live with a 0.XX% chance of getting pwned any given day.
  - embedding-shape 1mo ago
    > But it seems like nowadays NOTHING can be trusted - not OSes (recent Qubes OS exploit, not even mentioning others), Clearly you feel alarmed, but it's important to base these "alarm" feelings on actual evidence and real concrete proof of something being bad. You clearly don't have a proper understanding of the exploit, so please take a moment to re-read what actually happened and how it would be exploited in practice, particularly the "the scope of this attack is smaller than it sounds" comment chain: https://news.ycombinator.com/item?id=49496918 https://news.ycombinator.com/item?id=49496918 Overall, I agree with you though, and it's a healthy perspective to be safer rather than sorrier, so living with the assumption that getting pwned any day is a non-zero chance/risk is probably the best approach and what I personally do too.
mike_hearn 1mo ago
Yes! We'll see how it goes, but the product in question (Conveyor) is a downloadable tool that's got deliberately unobfuscated bytecode in it, with lots of detailed logging. AI is perfectly capable of reverse engineering it and in fact this bug report contained such a reversing. So even if someone tricks it into revealing source code or similar, they won't get anything that isn't already obtainable via other methods. This isn't a SaaS where security through obscurity might conceivably help, or where the codebase might contain credentials by mistake. It's a developer tool and this level of trust helps customers debug their own problems quickly. If someone wants to break the law, they'll get a legal answer, but it's never been a problem. The bot in question cannot write to master though, only open up pull requests from its own isolated repository. It's a bet on modern models being more resistant to confusion attacks than they were before. The harness setup also makes it very clear to the model where input comes from. This might be a bad bet, but if it's not, then it's helpful for customers to get help right away.

embedding-shape 1mo ago

Surely passing untrusted input to a agent with execution capabilities and also possibility to reply to the same author, couldn't possibly be used for anything negative? Parent is probably using a firewall so it's A-OK :thumbs_up:

dormento 1mo ago
What a wild ride huh. We're in the "fuck it we ball" era. Not even the many reports of prompt injection and takeover due to IA misfeatures can cheer me up anymore. It is all incredibly sad.
KronisLV 1mo ago
> Surely passing untrusted input to a agent with execution capabilities Oh hey, I know this one! It’s humans and a phishing test, they’ll click on all the links and enter information without checking the domain properly! Personally, I’d like systems that aren’t open to attack and can be depended upon. But it seems like nowadays NOTHING can be trusted - not OSes (recent Qubes OS exploit, not even mentioning others), not any software written in languages without memory safety, not even the ones with (Log4j comes to mind), not the packages in many package managers, not other humans and sure as hell not the token prediction machines. What a world. We all probably live with a 0.XX% chance of getting pwned any given day.
- embedding-shape 1mo ago
  > But it seems like nowadays NOTHING can be trusted - not OSes (recent Qubes OS exploit, not even mentioning others), Clearly you feel alarmed, but it's important to base these "alarm" feelings on actual evidence and real concrete proof of something being bad. You clearly don't have a proper understanding of the exploit, so please take a moment to re-read what actually happened and how it would be exploited in practice, particularly the "the scope of this attack is smaller than it sounds" comment chain: https://news.ycombinator.com/item?id=49496918 https://news.ycombinator.com/item?id=49496918 Overall, I agree with you though, and it's a healthy perspective to be safer rather than sorrier, so living with the assumption that getting pwned any day is a non-zero chance/risk is probably the best approach and what I personally do too.

mike_hearn 1mo ago

Yes! We'll see how it goes, but the product in question (Conveyor) is a downloadable tool that's got deliberately unobfuscated bytecode in it, with lots of detailed logging. AI is perfectly capable of reverse engineering it and in fact this bug report contained such a reversing. So even if someone tricks it into revealing source code or similar, they won't get anything that isn't already obtainable via other methods. This isn't a SaaS where security through obscurity might conceivably help, or where the codebase might contain credentials by mistake. It's a developer tool and this level of trust helps customers debug their own problems quickly. If someone wants to break the law, they'll get a legal answer, but it's never been a problem. The bot in question cannot write to master though, only open up pull requests from its own isolated repository. It's a bet on modern models being more resistant to confusion attacks than they were before. The harness setup also makes it very clear to the model where input comes from. This might be a bad bet, but if it's not, then it's helpful for customers to get help right away.