3 ms·

You're a paralegal preparing case summaries for your boss. You use a high-end and trusted retrieval-augmented generative platform, and it dutifully summarizes t

You're a paralegal preparing case summaries for your boss. You use a high-end and trusted retrieval-augmented generative platform, and it dutifully summarizes the cases. After a spot-check, it seems right: correct plaintiffs, correct defendants, correct case numbers and stylings. But a former partner poisoned the case "rooms" where docs were uploaded. Now, the senior partner is making a filing on behalf of their largest client. The errors are subtle. Enough to escape detection. But they catch someone's eye, and it turns out that, collectively, you've misstated past judgments to the tune of $250,000. Now everything done for that client is suspect, other clients lose faith, and the firm is embarrassed on the national stage due to their failure to listen to someone who clearly stated the liability issue with untrusted inputs, but instead they chuckled along with the snarky post about common sense.

kromem 3y ago
"But a former partner poisoned the documents" So basically "protocol breach involving tampering that could be EXTREMELY severe in 99 non-AI ways could be made to be potentially problematic in a fictional generative AI scenario which is already suspect as using a product prone to accuracy issues in preparing accuracy critical work without adequate checks is a bad idea to begin with." I appreciate your effort in coming up with a scenario. Let me try my hand as well: "China's APT team gets unfettered access to Microsoft's servers and buries deep for three years. Eventually they modify PDFs of client financing agreements - not changing the visible information, but changing invisible instructions to alter how an AI might read credit limits. This results in an AI's unaudited review of credit agreement compliance to deny sales to large customers and allow sales to clients with low credit worthiness. After a year of this flying under the radar, Microsoft has lost business or had unpaid debt totaling over $100 million dollars." I'm sure Microsoft's CSO reading that paragraph would be most concerned about the PDF reading AI, out of everything listed. How does this not already have a CVSS of 'severe'?

kromem 3y ago

"But a former partner poisoned the documents" So basically "protocol breach involving tampering that could be EXTREMELY severe in 99 non-AI ways could be made to be potentially problematic in a fictional generative AI scenario which is already suspect as using a product prone to accuracy issues in preparing accuracy critical work without adequate checks is a bad idea to begin with." I appreciate your effort in coming up with a scenario. Let me try my hand as well: "China's APT team gets unfettered access to Microsoft's servers and buries deep for three years. Eventually they modify PDFs of client financing agreements - not changing the visible information, but changing invisible instructions to alter how an AI might read credit limits. This results in an AI's unaudited review of credit agreement compliance to deny sales to large customers and allow sales to clients with low credit worthiness. After a year of this flying under the radar, Microsoft has lost business or had unpaid debt totaling over $100 million dollars." I'm sure Microsoft's CSO reading that paragraph would be most concerned about the PDF reading AI, out of everything listed. How does this not already have a CVSS of 'severe'?