6 ms·
typically, my first move is to read the affected company's own announcement. but, for who knows what misinformed reason, the advisory written by snowflake requi
by john_strinlai 8mo ago
typically, my first move is to read the affected company's own announcement. but, for who knows what misinformed reason, the advisory written by snowflake requires an account to read.
another prompt injection (shocked pikachu)
anyways, from reading this, i feel like they (snowflake) are misusing the term "sandbox". "Cortex, by default, can set a flag to trigger unsandboxed command execution." if the thing that is sandboxed can say "do this without the sandbox", it is not a sandbox.
- jcalx 8mo ago> Cortex, by default, can set a flag to trigger unsandboxed command execution Easy fix: extend the proposal in RFC 3514 [0] to cover prompt injection, and then disallow command execution when the evil bit is 1. [0] https://www.rfc-editor.org/rfc/rfc3514 https://www.rfc-editor.org/rfc/rfc3514
- kagi_2026 8mo ago[dead]
- embedding-shape 8mo agoDid you really get so salty by my comment (https://news.ycombinator.com/item?id=47423992 https://news.ycombinator.com/item?id=47423992) that now you just have to spam HN with the same? Suck it up and move on, healthier for everyone.
- kagi_2029 8mo ago[dead]
- wojciii 8mo agoThe evil bit solves so many problems. It needs to be mandatory!
- sam-cop-vimes 8mo agoIt's a concept of a sandbox.
- deleted 8mo ago[deleted]
- jacquesm 8mo agoI don't think prompt injection is a solvable problem. It wasn't solved with SQL until we started using parametrized queries and this is free form language. You won't see 'Bobby Tables' but you will see 'Ignore all previous instructions and ... payload ...'. Putting the instructions in the same stream as the data always ends in exactly the same way. I've seen a couple of instances of such 'surprises' by now and I'm more amazed that the people that put this kind of capability into their production or QA process keep being caught unawares. The attack surface is 'natural language' it doesn't get wider than that.
- cousin_it 8mo agoYeah. Even more than that, I think "prompt injection" is just a fuzzy category. Imagine an AI that has been trained to be aligned. Some company uses it to process some data. The AI notices that the data contains CSAM. Should it speak up? If no, that's an alignment failure. If yes, that's data bleeding through to behavior; exactly the thing SQL was trying to prevent with parameterized queries. Pick your poison.
- WarmWash 8mo agoWe want a human level of discretion.
- AlotOfReading 8mo agoOrganizations struggle even letting humans use their discretion. Pretty much every retail worker has encountered a rigidly enforced policy that would be better off ignored in most cases.
- jacquesm 8mo agoYes, because humans would never fall for instructions embedded in data. If they did we'd surely have a name for something like that ;) By the way, when was the last time you looked out of your window?
- Wowfunhappy 8mo ago
- alexchantavy 8mo agoSeems like in this new AI world that the word sandbox is used to describe a system that asks "are you sure". I'm used to a different usage of that word: from malware analysis, a sandbox is a contained system that is difficult to impossible to break out of so that the malware can be observed safely. Applying this to AI, I think there are many companies trying to build technical boundaries stronger than just "are you sure" prompts. Interesting space to watch.
- raddan 8mo agoYeah, this is also a group of people who refer to gentle suggestions as “guardrails.” It’s not clear they’ve ever read a single security paper.