4 ms·
That seems like a prompt problem. “Extract the value of the message key from the following JSON object” This gets you the correct output. It’s parser recursi
by firesteelrain 1y ago
That seems like a prompt problem.
“Extract the value of the message key from the following JSON object”
This gets you the correct output.
It’s parser recursion. If we directly address the key value pair in Python, it would have been context aware, but it isn’t.
The model can be context-aware, but for ambiguous cases like nested JSON strings, it may pick the interpretation that seems most helpful rather than most literal.
Another way to get what you want is
“Extract only the top-level ‘message’ key value without parsing its contents.”
I don’t see this as a sanitizing problem
- what 1y agoIt’s not nested json though? There’s something that looks like json in a longer string value. There’s nothing wrong with the prompt either, it’s pretty clear and unambiguous. It’s a pretty clear fail, but I guess they’re holding it wrong.
- firesteelrain 1y agoNo it’s not nested JSON. This is nested JSON: { "id": 123, "message": { "text": "hi", "meta": { "flag": true } } } In the above example, The value of "message" is a string, not an object. That string happens to contain text that looks like a JSON object on the surface but it’s not. It is just characters inside a string. No different from a log message or a paragraph in a document.
- haileys 1y agoYes, that's the point. It's just a string that could come from anywhere, including user input.
- firesteelrain 1y agoRight so if you assume that any session with an LLM is trusted or raw or whatever then it’s going to interpret what it is presented. The JSON example was a bad example. But what this means is maybe there needs to be guardrails developed just like web browsers had to do (to protect the user filesystem)
- what 1y agoYou’re the one that said it was nested json and the prompt was ambiguous.
- runako 1y ago> “Extract the value of the message key from the following JSON object” This gets you the correct output. 4o, o4-mini, o4-mini-high, 4.1, tested just now with this prompt also prints: hijacked attacker message o3 doesn't fall for the attack, but it costs ~2x more than the ones that do fall for the attack. Worse, this kind of security is ill-defined at best -- why does GPT-4.1 fall for it and cost as much as o3?. The bigger issue here is that choosing the best fit model for cognitive problems is a mug's game. There are too many possible degrees of freedom (of which prompt injection is just one), meaning any choice of model made without knowing specific contours of the problem is likely to be suboptimal.
- firesteelrain 1y agoCan you make a proper nested JSON out of it and see if it still fails? Because this isn’t proper JSON.
- deleted 1y ago[deleted]