3 ms·

Think about all the companies doing "chat with PDF" and relying on the text of those PDF files. If you're making light of this instead of recognizing how bad a

Think about all the companies doing "chat with PDF" and relying on the text of those PDF files. If you're making light of this instead of recognizing how bad actors will be using PDF files as an attack vector to LLM-based systems, you're lacking in imagination.

kromem 3y ago
Lol. Attack vector in a chatbot with complete chat isolation. People see the letters 'A' and 'I' and common sense goes out the window. Please describe a credible threat vector, with special attention paid to the scope of a compromise of the chatbot by a PDF...
- spdustin 3y ago
  You're a paralegal preparing case summaries for your boss. You use a high-end and trusted retrieval-augmented generative platform, and it dutifully summarizes the cases. After a spot-check, it seems right: correct plaintiffs, correct defendants, correct case numbers and stylings. But a former partner poisoned the case "rooms" where docs were uploaded. Now, the senior partner is making a filing on behalf of their largest client. The errors are subtle. Enough to escape detection. But they catch someone's eye, and it turns out that, collectively, you've misstated past judgments to the tune of $250,000. Now everything done for that client is suspect, other clients lose faith, and the firm is embarrassed on the national stage due to their failure to listen to someone who clearly stated the liability issue with untrusted inputs, but instead they chuckled along with the snarky post about common sense.
  - kromem 3y ago
    "But a former partner poisoned the documents" So basically "protocol breach involving tampering that could be EXTREMELY severe in 99 non-AI ways could be made to be potentially problematic in a fictional generative AI scenario which is already suspect as using a product prone to accuracy issues in preparing accuracy critical work without adequate checks is a bad idea to begin with." I appreciate your effort in coming up with a scenario. Let me try my hand as well: "China's APT team gets unfettered access to Microsoft's servers and buries deep for three years. Eventually they modify PDFs of client financing agreements - not changing the visible information, but changing invisible instructions to alter how an AI might read credit limits. This results in an AI's unaudited review of credit agreement compliance to deny sales to large customers and allow sales to clients with low credit worthiness. After a year of this flying under the radar, Microsoft has lost business or had unpaid debt totaling over $100 million dollars." I'm sure Microsoft's CSO reading that paragraph would be most concerned about the PDF reading AI, out of everything listed. How does this not already have a CVSS of 'severe'?
- justinclift 3y ago
  > complete chat isolation. They're communicating with humans, and some of those humans can be fooled.

kromem 3y ago

Lol. Attack vector in a chatbot with complete chat isolation. People see the letters 'A' and 'I' and common sense goes out the window. Please describe a credible threat vector, with special attention paid to the scope of a compromise of the chatbot by a PDF...

spdustin 3y ago
You're a paralegal preparing case summaries for your boss. You use a high-end and trusted retrieval-augmented generative platform, and it dutifully summarizes the cases. After a spot-check, it seems right: correct plaintiffs, correct defendants, correct case numbers and stylings. But a former partner poisoned the case "rooms" where docs were uploaded. Now, the senior partner is making a filing on behalf of their largest client. The errors are subtle. Enough to escape detection. But they catch someone's eye, and it turns out that, collectively, you've misstated past judgments to the tune of $250,000. Now everything done for that client is suspect, other clients lose faith, and the firm is embarrassed on the national stage due to their failure to listen to someone who clearly stated the liability issue with untrusted inputs, but instead they chuckled along with the snarky post about common sense.
- kromem 3y ago
  "But a former partner poisoned the documents" So basically "protocol breach involving tampering that could be EXTREMELY severe in 99 non-AI ways could be made to be potentially problematic in a fictional generative AI scenario which is already suspect as using a product prone to accuracy issues in preparing accuracy critical work without adequate checks is a bad idea to begin with." I appreciate your effort in coming up with a scenario. Let me try my hand as well: "China's APT team gets unfettered access to Microsoft's servers and buries deep for three years. Eventually they modify PDFs of client financing agreements - not changing the visible information, but changing invisible instructions to alter how an AI might read credit limits. This results in an AI's unaudited review of credit agreement compliance to deny sales to large customers and allow sales to clients with low credit worthiness. After a year of this flying under the radar, Microsoft has lost business or had unpaid debt totaling over $100 million dollars." I'm sure Microsoft's CSO reading that paragraph would be most concerned about the PDF reading AI, out of everything listed. How does this not already have a CVSS of 'severe'?
justinclift 3y ago
> complete chat isolation. They're communicating with humans, and some of those humans can be fooled.