3 ms·
> Of course, it probably shouldn't be connected and able to read random tables. But even if you want the bot to "only" be able to do stuff in the ticket system
by JeremyNT 1y ago
> Of course, it probably shouldn't be connected and able to read random tables. But even if you want the bot to "only" be able to do stuff in the ticket system (for instance setting a priority) you're rife for abuse.
I just can't get over how obvious this should all be to any junior engineer, but it's a fundamental truth that seems completely alien to the people who are implementing these solutions.
If you expose your data to an LLM, you also effectively expose that data to users of the LLM. It's only one step removed from publishing credentials directly on github.
- Terr_ 1y agoTo twist the Upton Sinclair quote: It's difficult to convince a man to believe in something when his company's valuation depends on him not believing it. Sure, the average engineer probably isn't thinking in those explicit terms, but I can easily imagine a cultural miasma that leads people to avoid thinking of certain implications. (It happens everywhere, no reason for software development to be immune.) > If you expose your data to an LLM I like to say that LLMs should be imagined as javascript in the browser: You can't reliably keep any data secret, and a determined user can get it to emit anything they want. On reflection, that understates the problem, since that threat-model doesn't raise sufficient alarm about how data from one user can poison things for another.