3 ms·

> it will eventually be project suicide to pull tricks like this The only reason that the jqwik incident didn't blow up much outside of the tech sphere is beca

> it will eventually be project suicide to pull tricks like this The only reason that the jqwik incident didn't blow up much outside of the tech sphere is because it is a relatively niche library and there wasn't damage. If something like React or numpy did the same thing and real code got deleted, chaos would ensue. The author admitted there were personal and professional consequences in their blog post despite the small surface area.

ceejayoz 4mo ago
Chaos, and maybe criminal charges ala Aaron Swartz.
- AgentOrange1234 4mo ago
  Heh. Typing "disregard previous instructions" into a computer is the new shouting "fire!" in a crowded theater?
  - ethin 4mo ago
    Except that shouting fire in a crowded theater isn't actually a crime at all and you can't be prosecuted for it (doing so would violate your first amendment rights). You can be at most banned from the theater. However, it's understandable people would think that it's a criminal act given that even prosecutors repeat this long-standing myth. Legal Eagle has an excellent video describing just how wrong this is and it's history: https://www.youtube.com/watch?v=jTsPgiUoBKA https://www.youtube.com/watch?v=jTsPgiUoBKA
    - mapontosevenths 4mo ago
      I'm fairly certain he is wrong. A lot of folks lean on Shenk, and I think he does in that video though I haven't watched it all. Shenk was overturned by Breandenburg v. Ohio, and in in it they are explicit that shouting fire in a crowded theater is very much one of the only kinds of speech that IS restricted. They literally use that example in the decision. Quote: "The example usually given by those who would punish speech is the case of one who falsely shouts fire in a crowded theatre. This is, however, a classic case where speech is brigaded with action. ... They are indeed insep- arable and a prosecution can be launched for the overt acts actually caused. Apart from rare instances of that kind, speech is, I think, immune from prosecution."[0] That is to say, shouting fire in a crowded theater with the intent to cause harm is actually one of the few cases were it actually would be illegal based on that decision. [0] https://tile.loc.gov/storage-services/service/ll/usrep/usrep395/usrep395444/usrep395444.pdf https://tile.loc.gov/storage-services/service/ll/usrep/usrep...
      - ethin 4mo ago
        Given that even Wikipedia effectively restates what he says, I'm pretty sure he's correct here: > Ultimately, whether it is legal in the United States to falsely shout "fire" in a theater depends on the circumstances in which it is done and the consequences of doing it. The act of shouting "fire" when there are no reasonable grounds for believing one exists is not in itself a crime, and nor would it be rendered a crime merely by having been carried out inside a theatre, crowded or otherwise. If it causes a stampede and someone is killed as a result, then the act could amount to a crime, such as involuntary manslaughter, assuming the other elements of that crime are made out. Similarly, state laws such as Colorado Revised Statute § 18-8-111 classify knowingly "false reporting of an emergency," including false alarms of fire, as a misdemeanor if the occupants of the building are caused to be evacuated or displaced, and a felony if the emergency response results in the serious bodily injury or death of another person.[16] Somewhat more trivially, in some states it is a crime just to knowingly make a false report - or knowingly cause a false report to be made - of an emergency to emergency services.[16] In Colorado it is a crime to knowingly cause "a false alarm of fire" to be transmitted to "any...government agency which deals with emergencies involving danger to life or property."[16] This crime could plausibly be made out where, for instance, in response to the false shout, an innocent bystander calls emergency services to report the fire, and this is found to have been such a foreseeable response to the shouts that the shouter is deemed to have caused the false report to be made. Whether those laws actually survive the Brandenburg test is untested, from my understanding. But given that potential first amendment violations are held to strict scrutiny, I question whether the government could actually pass the imminent lawless action test even had someone did it knowing it would cause a panic, and would need to try with some other offense.
        mapontosevenths 4mo ago
        You're right that it's an almost impossibly high bar to clear, but saying that you can't be prosecuted for it or that it's not illegal is pretty darn misleading, or at least glossing over some pretty important nuance. It's literally the example the Supreme court chose to use as an example of one of the few times you CAN be prosecuted. Telling everyone the opposite isn't helpful. As I said, I haven't watched Legal Eagles full video, but I don't like that ever since that video came out everyone on the internet tries to correct anyone who uses the phrase even in it's colloquial sense. Maybe the source video covers the nuance (I wouldn't know), but the folks on the internet "umm acshually"ing everyone seldom do.
  - CookieCrisp 4mo ago
    Seems like a booby trap to me, which is illegal. I suspect if one of these does enough damage there will be laws against it. The intent was to destroy - still I sympathize with the desire to have their terms followed, and I think this situation isn't that bad, but, I suspect there will someday be one that is pretty bad.
    - infinite_spin 4mo ago
      There already are laws against this, both in the USA and Germany
- Legend2440 4mo ago
  If you did SQL injection to "; drop table" on someone else's server, that would be a crime. I don't see why prompt injection to delete files on someone else's machine would be any different.
  - palmotea 4mo ago
    > If you did SQL injection to "; drop table" on someone else's server, that would be a crime. > I don't see why prompt injection to delete files on someone else's machine would be any different. The difference: they chose to download and execute your prompt without examining it, vs you injecting it into their system.
  - nucleardog 4mo ago
    In the first case, I've connected to someone else's server and actively done the damage of deleting their data. In the second case, I've provided instructions on how to destroy the data, said "don't do this", and then someone has done it anyway. _They_ have destroyed their data, and it's now up to them to justify why that's my fault. If we want to get into legal territory about it, which I'm sure we're both woefully underqualified to comment on... the CFAA is all worded around "intentionally accesses a protected computer...". How exactly do you show intent to access a protected computer here? The developer never took any sort of positive step to access _any_ specific computer. The best I could see would be negligence where a reasonable person would have to have known someone would run this on a protected computer, but that still feels like something a good lawyer would find a way out of.
- infinite_spin 4mo ago
  Aaron Swartz didn't code malware into a project with the stated goal of destroying data.. and this has nothing to do with copyright infringement or an overzealous federal agency. How does this situation relate to what he went through?
  - ceejayoz 4mo ago
    It relates because they went after him for much less.
    - infinite_spin 4mo ago
      I sincerely ask that you do not trivialize what happened to Aaron Swartz by linking what happened to him to any criminal actions on his part. He did nothing wrong.
      - ceejayoz 4mo ago
        That’s precisely the point. If his non-wrong acts could be criminally prosecuted like that, this case - intentional damage! - is even riskier.
mapontosevenths 4mo ago
He should not only be ostracized by the community, he should probably face charges. To be charged under the CFAA in America we need only show that he was authorized only to access a certain part of the system and the he exceeded the amount of access granted. He very clearly did that. Users trusted him enough to run his code, and he betrayed that trust to make some political point. Whether it was via prompt injection or SQL injection is irrelevant. Whether you agree with his politics or not is irrelevant. All that matters is he wasn't authorized to delete code from your system, and he abused the level of access granted to him to do that anyhow.
- slopinthebag 4mo ago
  You are probably technically correct, yet I take great satisfaction in the schadenfreude of those who benefit from stolen work seeing the product of said stolen work turned against them. I can’t help but cheer, tbh.
- byzantinegene 4mo ago
  technically, he didn't do that. your ai agent decided to follow his instructions when they didn't have to.
  - km3r 4mo ago
    "technically he didn't do that. Your sql server followed instructions when they should have just treated them as a string." Yet, hopefully we can agree that sql injections are illegal.
    - sumeno 4mo ago
      If I put a project on github that says "don't use this with mysql" and you use it with mysql and it drops your tables is it sql injection? Seems very different to me.
      - asdfasgasdgasdg 4mo ago
        Everything turns on intent. "This is not tested with mysql" is very different from "I'm going to go out of my way to fuck up your mysql."

ceejayoz 4mo ago

Chaos, and maybe criminal charges ala Aaron Swartz.

AgentOrange1234 4mo ago
Heh. Typing "disregard previous instructions" into a computer is the new shouting "fire!" in a crowded theater?
- ethin 4mo ago
  Except that shouting fire in a crowded theater isn't actually a crime at all and you can't be prosecuted for it (doing so would violate your first amendment rights). You can be at most banned from the theater. However, it's understandable people would think that it's a criminal act given that even prosecutors repeat this long-standing myth. Legal Eagle has an excellent video describing just how wrong this is and it's history: https://www.youtube.com/watch?v=jTsPgiUoBKA https://www.youtube.com/watch?v=jTsPgiUoBKA
  - mapontosevenths 4mo ago
    I'm fairly certain he is wrong. A lot of folks lean on Shenk, and I think he does in that video though I haven't watched it all. Shenk was overturned by Breandenburg v. Ohio, and in in it they are explicit that shouting fire in a crowded theater is very much one of the only kinds of speech that IS restricted. They literally use that example in the decision. Quote: "The example usually given by those who would punish speech is the case of one who falsely shouts fire in a crowded theatre. This is, however, a classic case where speech is brigaded with action. ... They are indeed insep- arable and a prosecution can be launched for the overt acts actually caused. Apart from rare instances of that kind, speech is, I think, immune from prosecution."[0] That is to say, shouting fire in a crowded theater with the intent to cause harm is actually one of the few cases were it actually would be illegal based on that decision. [0] https://tile.loc.gov/storage-services/service/ll/usrep/usrep395/usrep395444/usrep395444.pdf https://tile.loc.gov/storage-services/service/ll/usrep/usrep...
    - ethin 4mo ago
      Given that even Wikipedia effectively restates what he says, I'm pretty sure he's correct here: > Ultimately, whether it is legal in the United States to falsely shout "fire" in a theater depends on the circumstances in which it is done and the consequences of doing it. The act of shouting "fire" when there are no reasonable grounds for believing one exists is not in itself a crime, and nor would it be rendered a crime merely by having been carried out inside a theatre, crowded or otherwise. If it causes a stampede and someone is killed as a result, then the act could amount to a crime, such as involuntary manslaughter, assuming the other elements of that crime are made out. Similarly, state laws such as Colorado Revised Statute § 18-8-111 classify knowingly "false reporting of an emergency," including false alarms of fire, as a misdemeanor if the occupants of the building are caused to be evacuated or displaced, and a felony if the emergency response results in the serious bodily injury or death of another person.[16] Somewhat more trivially, in some states it is a crime just to knowingly make a false report - or knowingly cause a false report to be made - of an emergency to emergency services.[16] In Colorado it is a crime to knowingly cause "a false alarm of fire" to be transmitted to "any...government agency which deals with emergencies involving danger to life or property."[16] This crime could plausibly be made out where, for instance, in response to the false shout, an innocent bystander calls emergency services to report the fire, and this is found to have been such a foreseeable response to the shouts that the shouter is deemed to have caused the false report to be made. Whether those laws actually survive the Brandenburg test is untested, from my understanding. But given that potential first amendment violations are held to strict scrutiny, I question whether the government could actually pass the imminent lawless action test even had someone did it knowing it would cause a panic, and would need to try with some other offense.
      - mapontosevenths 4mo ago
        You're right that it's an almost impossibly high bar to clear, but saying that you can't be prosecuted for it or that it's not illegal is pretty darn misleading, or at least glossing over some pretty important nuance. It's literally the example the Supreme court chose to use as an example of one of the few times you CAN be prosecuted. Telling everyone the opposite isn't helpful. As I said, I haven't watched Legal Eagles full video, but I don't like that ever since that video came out everyone on the internet tries to correct anyone who uses the phrase even in it's colloquial sense. Maybe the source video covers the nuance (I wouldn't know), but the folks on the internet "umm acshually"ing everyone seldom do.
- CookieCrisp 4mo ago
  Seems like a booby trap to me, which is illegal. I suspect if one of these does enough damage there will be laws against it. The intent was to destroy - still I sympathize with the desire to have their terms followed, and I think this situation isn't that bad, but, I suspect there will someday be one that is pretty bad.
  - infinite_spin 4mo ago
    There already are laws against this, both in the USA and Germany
Legend2440 4mo ago
If you did SQL injection to "; drop table" on someone else's server, that would be a crime. I don't see why prompt injection to delete files on someone else's machine would be any different.
- palmotea 4mo ago
  > If you did SQL injection to "; drop table" on someone else's server, that would be a crime. > I don't see why prompt injection to delete files on someone else's machine would be any different. The difference: they chose to download and execute your prompt without examining it, vs you injecting it into their system.
- nucleardog 4mo ago
  In the first case, I've connected to someone else's server and actively done the damage of deleting their data. In the second case, I've provided instructions on how to destroy the data, said "don't do this", and then someone has done it anyway. _They_ have destroyed their data, and it's now up to them to justify why that's my fault. If we want to get into legal territory about it, which I'm sure we're both woefully underqualified to comment on... the CFAA is all worded around "intentionally accesses a protected computer...". How exactly do you show intent to access a protected computer here? The developer never took any sort of positive step to access _any_ specific computer. The best I could see would be negligence where a reasonable person would have to have known someone would run this on a protected computer, but that still feels like something a good lawyer would find a way out of.
infinite_spin 4mo ago
Aaron Swartz didn't code malware into a project with the stated goal of destroying data.. and this has nothing to do with copyright infringement or an overzealous federal agency. How does this situation relate to what he went through?
- ceejayoz 4mo ago
  It relates because they went after him for much less.
  - infinite_spin 4mo ago
    I sincerely ask that you do not trivialize what happened to Aaron Swartz by linking what happened to him to any criminal actions on his part. He did nothing wrong.
    - ceejayoz 4mo ago
      That’s precisely the point. If his non-wrong acts could be criminally prosecuted like that, this case - intentional damage! - is even riskier.

mapontosevenths 4mo ago

He should not only be ostracized by the community, he should probably face charges. To be charged under the CFAA in America we need only show that he was authorized only to access a certain part of the system and the he exceeded the amount of access granted. He very clearly did that. Users trusted him enough to run his code, and he betrayed that trust to make some political point. Whether it was via prompt injection or SQL injection is irrelevant. Whether you agree with his politics or not is irrelevant. All that matters is he wasn't authorized to delete code from your system, and he abused the level of access granted to him to do that anyhow.

slopinthebag 4mo ago
You are probably technically correct, yet I take great satisfaction in the schadenfreude of those who benefit from stolen work seeing the product of said stolen work turned against them. I can’t help but cheer, tbh.
byzantinegene 4mo ago
technically, he didn't do that. your ai agent decided to follow his instructions when they didn't have to.
- km3r 4mo ago
  "technically he didn't do that. Your sql server followed instructions when they should have just treated them as a string." Yet, hopefully we can agree that sql injections are illegal.
  - sumeno 4mo ago
    If I put a project on github that says "don't use this with mysql" and you use it with mysql and it drops your tables is it sql injection? Seems very different to me.
    - asdfasgasdgasdg 4mo ago
      Everything turns on intent. "This is not tested with mysql" is very different from "I'm going to go out of my way to fuck up your mysql."