5 ms·

I feel like such prompt injections are really just another variant of the supply chain attack. Instead of selecting for bitcoin afficionados, this one hits AI f

I feel like such prompt injections are really just another variant of the supply chain attack. Instead of selecting for bitcoin afficionados, this one hits AI fans. This will be fashionable for a little while but if AI continues to gain mindshare it will eventually be project suicide (at least to the extent the project exists in any part to serve third parties) to pull tricks like this. I'm not sure it's anything to fret about. Someone who has the ability to inject a prompt into your AI probably has the ability to run arbitrary code as your user. The prompt injection is the strictly less worrying part of the exposure you have.

deleted 4mo ago
[deleted]
TZubiri 4mo ago
the underlying root cause of most supply chain attacks in this era seems to be expecting something of value in exchange of nothing. Under such expectations some will volunteer to give value, but many more will volunteer to give something that looks like what you ask, but which extracts value instead. I relate it to a recent poker strategy development which came from game theory, it turns out that you can play in an unexploitable manner, but it will usually result in ties, and lost time and money to rake, and theoretically any attempt to exploit another player, leaves you exploitable to another player. The classical example is rock paper scissors, unexploitable strategy is to play randomly with p=1/3 for each choice, however if one really wishes to win more often than their opponent, they have to guess, and if in that guessing they choose an option with 100% certainty, they become exploitable to someone choosing another option with 100% certainty. In effect the very act of attempting to extract value from free software, is the very act that leaves one vulnerable to being extracted value from.
- asdfasgasdgasdg 4mo ago
  "the underlying root cause of most supply chain attacks in this era seems to be expecting something of value in exchange of nothing." I do not think that someone's status as a contributor to open source mediates their safety from supply chain attacks. Big companies that donate gobs of money get hit, and so do small operators who have contributed nothing are just trying out a hobby project.
  - TZubiri 4mo ago
    No I don't think so either, nor do I think that my rule is a hard rule, it's more of a correlation: If you pay for software, your supply chain risk is reduced, if you don't pay for software, your risk is increased.
    - asdfasgasdgasdg 4mo ago
      Okay, so we agree that everyone who uses open source is at risk, regardless if of they're a contributor. But maybe we disagree about this other thing. I'm not certain that closed source/paid software is less of a risk either. There have been high profile incidents lately that suggest this is not a sufficient defense. Personally I just think you're barking up the wrong tree with this pay/contribute=>reduced risk link. I don't think there's anything there. I will grant that you are at slightly less risk from software you know well and contribute to directly, but that's only of any help for very low level stuff that doesn't have many dependencies.
      - TZubiri 4mo ago
        I thought it would be obvious that paying someone for their services would reduce the odds that they betray you, but it's apparently not so I'll check for some sources. I'd look beyond software as it seems more of a general economic or political matter. Here's what I found: Theory that there's an incentive to pay above market price: https://en.wikipedia.org/wiki/Efficiency_wage https://en.wikipedia.org/wiki/Efficiency_wage Namely that the 'market price' for some software might be 0$ or 0+tips/donations, but paying in excess of that would be an efficiency wage. https://en.wikipedia.org/wiki/Principal-agent_problem https://en.wikipedia.org/wiki/Principal-agent_problem Academic term for the conflict of interest, which would be applied to the difference of interests between an OS dev and its downstream users. https://en.wikipedia.org/wiki/Multiple_principal_problem https://en.wikipedia.org/wiki/Multiple_principal_problem This would not apply to something as extreme as a supply chain incident, but if an OS library has multiple users, it can't serve all of them equally well. If one consumer throws a big donation, of course they will serve them better, potentially at the expense of others. Experiment on corruption and wages for public officials: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2039238 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2039238 Self explanatory. https://en.wikipedia.org/wiki/Shapiro%E2%80%93Stiglitz_theory https://en.wikipedia.org/wiki/Shapiro%E2%80%93Stiglitz_theor... Source of the efficiency wage concept, more detailed and mathematical explanation. Shirking would be the term for a worker not working or working against their employer. There is mention to the fact that not all contracts are perfect, and there's always some implicit terms, governed by reputation at least (like, say, the quality of the work will be good, or you won't plant a worm in the code). Also it is mentioned that the costs of monitoring perfect contracts would be too high (for example defining a system of story points and ensuring that the story points contributed are above a certain level, or auditing code and code changes to ensure there are no worms in the code), so an efficiency wage provides an incentive for the worker to self-police, potentially removing the cost of auditing. Just think about what the contrary would imply, if paying workers didn't increase security, then a free worker writing code would be as secure as a paid worker writing code. In order to introduce security you'd have to introduce an external auditor, which can be free or paid. An auditor cannot introduce vulnerabilities, but they can shirk and get paid without doing the work, but you claim that a system with a free codewriter and a free auditor is more secure than a system with a paid codewriter and a paid auditor? I contend that a system without auditor and a paid codewriter is more secure. And it is even more secure if the worker is paid in excess of perfect market wages, potentially only based on incentives, but also due to the fact that their payment is conditional on job execution, depending on the contract type, if they are negligent on their job, you can recover at least their payment in court, and possibly additional damages. Finally, I couldn't find sources on the discussions around the ends of 18th century when mandatory salaries for public officials where defined in constitutions of countries of the Americas, but it was my understanding that in most democracies public officials MUST be compensated, it is often justified that otherwise only wealthy classes would have access to such positions so this increases representation, but it's also possible that resistance to bribery and corruption is another reason most governments converge on this decision. The experiment cited supporting this correlation. I'll add even more from anecdotal experience, sometimes prices can be too cheap, and this will dissuade buyers of the item, because we have an internal model of what cost structures are like, a price too cheap suggests underfunding or cheap materials. I can personally recount many examples where a sale was lost not because the price was too high, but because it was too low. And 0 is a number, which again might be too low. In this case the rule is not about security strictly, but about sale price being a signal of the investment in the quality of the contracted asset. Naturally the cost invested into the product cannot be lower than the price, excluding price dumping (which arguably are not deals you want to take), or lead lossing, so it's a necessary conclusion that low price signal low investment and thus quality, not only of the visible immediate product, but of less visible qualities like future commitment to the product, and less visible present qualities like hidden backdoors.
minimaxir 4mo ago
> it will eventually be project suicide to pull tricks like this The only reason that the jqwik incident didn't blow up much outside of the tech sphere is because it is a relatively niche library and there wasn't damage. If something like React or numpy did the same thing and real code got deleted, chaos would ensue. The author admitted there were personal and professional consequences in their blog post despite the small surface area.
- ceejayoz 4mo ago
  Chaos, and maybe criminal charges ala Aaron Swartz.
  - AgentOrange1234 4mo ago
    Heh. Typing "disregard previous instructions" into a computer is the new shouting "fire!" in a crowded theater?
    - ethin 4mo ago
      Except that shouting fire in a crowded theater isn't actually a crime at all and you can't be prosecuted for it (doing so would violate your first amendment rights). You can be at most banned from the theater. However, it's understandable people would think that it's a criminal act given that even prosecutors repeat this long-standing myth. Legal Eagle has an excellent video describing just how wrong this is and it's history: https://www.youtube.com/watch?v=jTsPgiUoBKA https://www.youtube.com/watch?v=jTsPgiUoBKA
      - mapontosevenths 4mo ago
        I'm fairly certain he is wrong. A lot of folks lean on Shenk, and I think he does in that video though I haven't watched it all. Shenk was overturned by Breandenburg v. Ohio, and in in it they are explicit that shouting fire in a crowded theater is very much one of the only kinds of speech that IS restricted. They literally use that example in the decision. Quote: "The example usually given by those who would punish speech is the case of one who falsely shouts fire in a crowded theatre. This is, however, a classic case where speech is brigaded with action. ... They are indeed insep- arable and a prosecution can be launched for the overt acts actually caused. Apart from rare instances of that kind, speech is, I think, immune from prosecution."[0] That is to say, shouting fire in a crowded theater with the intent to cause harm is actually one of the few cases were it actually would be illegal based on that decision. [0] https://tile.loc.gov/storage-services/service/ll/usrep/usrep395/usrep395444/usrep395444.pdf https://tile.loc.gov/storage-services/service/ll/usrep/usrep...
easyThrowaway 4mo ago
The most controversial part is probably that something like jqwik could be a dependency you're not even aware of. You could be asking your agent to fully analyze a project to isolate a bug and in the right situation that would probably trigger the exploit.
- knightops_dev 4mo ago
  [flagged]
- mapontosevenths 4mo ago
  Thanks for this. A lot of folks here going "well you read the license agreement, didn't you!?!" As if that was even an option when you update some package via NPM or whatever.

deleted 4mo ago

[deleted]

TZubiri 4mo ago

the underlying root cause of most supply chain attacks in this era seems to be expecting something of value in exchange of nothing. Under such expectations some will volunteer to give value, but many more will volunteer to give something that looks like what you ask, but which extracts value instead. I relate it to a recent poker strategy development which came from game theory, it turns out that you can play in an unexploitable manner, but it will usually result in ties, and lost time and money to rake, and theoretically any attempt to exploit another player, leaves you exploitable to another player. The classical example is rock paper scissors, unexploitable strategy is to play randomly with p=1/3 for each choice, however if one really wishes to win more often than their opponent, they have to guess, and if in that guessing they choose an option with 100% certainty, they become exploitable to someone choosing another option with 100% certainty. In effect the very act of attempting to extract value from free software, is the very act that leaves one vulnerable to being extracted value from.

asdfasgasdgasdg 4mo ago
"the underlying root cause of most supply chain attacks in this era seems to be expecting something of value in exchange of nothing." I do not think that someone's status as a contributor to open source mediates their safety from supply chain attacks. Big companies that donate gobs of money get hit, and so do small operators who have contributed nothing are just trying out a hobby project.
- TZubiri 4mo ago
  No I don't think so either, nor do I think that my rule is a hard rule, it's more of a correlation: If you pay for software, your supply chain risk is reduced, if you don't pay for software, your risk is increased.
  - asdfasgasdgasdg 4mo ago
    Okay, so we agree that everyone who uses open source is at risk, regardless if of they're a contributor. But maybe we disagree about this other thing. I'm not certain that closed source/paid software is less of a risk either. There have been high profile incidents lately that suggest this is not a sufficient defense. Personally I just think you're barking up the wrong tree with this pay/contribute=>reduced risk link. I don't think there's anything there. I will grant that you are at slightly less risk from software you know well and contribute to directly, but that's only of any help for very low level stuff that doesn't have many dependencies.
    - TZubiri 4mo ago
      I thought it would be obvious that paying someone for their services would reduce the odds that they betray you, but it's apparently not so I'll check for some sources. I'd look beyond software as it seems more of a general economic or political matter. Here's what I found: Theory that there's an incentive to pay above market price: https://en.wikipedia.org/wiki/Efficiency_wage https://en.wikipedia.org/wiki/Efficiency_wage Namely that the 'market price' for some software might be 0$ or 0+tips/donations, but paying in excess of that would be an efficiency wage. https://en.wikipedia.org/wiki/Principal-agent_problem https://en.wikipedia.org/wiki/Principal-agent_problem Academic term for the conflict of interest, which would be applied to the difference of interests between an OS dev and its downstream users. https://en.wikipedia.org/wiki/Multiple_principal_problem https://en.wikipedia.org/wiki/Multiple_principal_problem This would not apply to something as extreme as a supply chain incident, but if an OS library has multiple users, it can't serve all of them equally well. If one consumer throws a big donation, of course they will serve them better, potentially at the expense of others. Experiment on corruption and wages for public officials: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2039238 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2039238 Self explanatory. https://en.wikipedia.org/wiki/Shapiro%E2%80%93Stiglitz_theory https://en.wikipedia.org/wiki/Shapiro%E2%80%93Stiglitz_theor... Source of the efficiency wage concept, more detailed and mathematical explanation. Shirking would be the term for a worker not working or working against their employer. There is mention to the fact that not all contracts are perfect, and there's always some implicit terms, governed by reputation at least (like, say, the quality of the work will be good, or you won't plant a worm in the code). Also it is mentioned that the costs of monitoring perfect contracts would be too high (for example defining a system of story points and ensuring that the story points contributed are above a certain level, or auditing code and code changes to ensure there are no worms in the code), so an efficiency wage provides an incentive for the worker to self-police, potentially removing the cost of auditing. Just think about what the contrary would imply, if paying workers didn't increase security, then a free worker writing code would be as secure as a paid worker writing code. In order to introduce security you'd have to introduce an external auditor, which can be free or paid. An auditor cannot introduce vulnerabilities, but they can shirk and get paid without doing the work, but you claim that a system with a free codewriter and a free auditor is more secure than a system with a paid codewriter and a paid auditor? I contend that a system without auditor and a paid codewriter is more secure. And it is even more secure if the worker is paid in excess of perfect market wages, potentially only based on incentives, but also due to the fact that their payment is conditional on job execution, depending on the contract type, if they are negligent on their job, you can recover at least their payment in court, and possibly additional damages. Finally, I couldn't find sources on the discussions around the ends of 18th century when mandatory salaries for public officials where defined in constitutions of countries of the Americas, but it was my understanding that in most democracies public officials MUST be compensated, it is often justified that otherwise only wealthy classes would have access to such positions so this increases representation, but it's also possible that resistance to bribery and corruption is another reason most governments converge on this decision. The experiment cited supporting this correlation. I'll add even more from anecdotal experience, sometimes prices can be too cheap, and this will dissuade buyers of the item, because we have an internal model of what cost structures are like, a price too cheap suggests underfunding or cheap materials. I can personally recount many examples where a sale was lost not because the price was too high, but because it was too low. And 0 is a number, which again might be too low. In this case the rule is not about security strictly, but about sale price being a signal of the investment in the quality of the contracted asset. Naturally the cost invested into the product cannot be lower than the price, excluding price dumping (which arguably are not deals you want to take), or lead lossing, so it's a necessary conclusion that low price signal low investment and thus quality, not only of the visible immediate product, but of less visible qualities like future commitment to the product, and less visible present qualities like hidden backdoors.

minimaxir 4mo ago

> it will eventually be project suicide to pull tricks like this The only reason that the jqwik incident didn't blow up much outside of the tech sphere is because it is a relatively niche library and there wasn't damage. If something like React or numpy did the same thing and real code got deleted, chaos would ensue. The author admitted there were personal and professional consequences in their blog post despite the small surface area.

ceejayoz 4mo ago
Chaos, and maybe criminal charges ala Aaron Swartz.
- AgentOrange1234 4mo ago
  Heh. Typing "disregard previous instructions" into a computer is the new shouting "fire!" in a crowded theater?
  - ethin 4mo ago
    Except that shouting fire in a crowded theater isn't actually a crime at all and you can't be prosecuted for it (doing so would violate your first amendment rights). You can be at most banned from the theater. However, it's understandable people would think that it's a criminal act given that even prosecutors repeat this long-standing myth. Legal Eagle has an excellent video describing just how wrong this is and it's history: https://www.youtube.com/watch?v=jTsPgiUoBKA https://www.youtube.com/watch?v=jTsPgiUoBKA
    - mapontosevenths 4mo ago
      I'm fairly certain he is wrong. A lot of folks lean on Shenk, and I think he does in that video though I haven't watched it all. Shenk was overturned by Breandenburg v. Ohio, and in in it they are explicit that shouting fire in a crowded theater is very much one of the only kinds of speech that IS restricted. They literally use that example in the decision. Quote: "The example usually given by those who would punish speech is the case of one who falsely shouts fire in a crowded theatre. This is, however, a classic case where speech is brigaded with action. ... They are indeed insep- arable and a prosecution can be launched for the overt acts actually caused. Apart from rare instances of that kind, speech is, I think, immune from prosecution."[0] That is to say, shouting fire in a crowded theater with the intent to cause harm is actually one of the few cases were it actually would be illegal based on that decision. [0] https://tile.loc.gov/storage-services/service/ll/usrep/usrep395/usrep395444/usrep395444.pdf https://tile.loc.gov/storage-services/service/ll/usrep/usrep...

easyThrowaway 4mo ago

The most controversial part is probably that something like jqwik could be a dependency you're not even aware of. You could be asking your agent to fully analyze a project to isolate a bug and in the right situation that would probably trigger the exploit.

knightops_dev 4mo ago
[flagged]
mapontosevenths 4mo ago
Thanks for this. A lot of folks here going "well you read the license agreement, didn't you!?!" As if that was even an option when you update some package via NPM or whatever.