4 ms·

Anyone remember the XZ and Jia Tan situation awhile back? https://lore.kernel.org/lkml/20240320183846.19475-1-lasse.collin@tukaani.org/t/ https://lore.kernel.o

by J0nL 4mo ago

Anyone remember the XZ and Jia Tan situation awhile back? https://lore.kernel.org/lkml/20240320183846.19475-1-lasse.collin@tukaani.org/t/ https://lore.kernel.org/lkml/20240320183846.19475-1-lasse.co... I can't quite put my finger on why but the entire time I was reading this I kept thinking back to that. It's entirely possible the actual targets were the volunteers and everything else was superfluous or tertiary. It's also an exception that proves the rule with regard to Hanlon's Razor. They even mentioned the stated goal of it was more or less pointless. I wouldn't be suprised if the "owner" they spoke with was still just the LLM. It stuck around for just long enough to convince everyone that they succeeded in suckering the LLM and had achieved all their stated objectives. No more reason to investigate the incident at all and no need to question why literally nothing made any sense or how the owner could simultaneously be as inept as they were made out to be and able to afford all those resources while giving the LLM effectively a blank check. It'll be interesting to see if the volunteers for this project are subjected to the same Zersetzung and psychological attacks as the XZ devs were.

delfinom 4mo ago
I am not sure giving everyone amusement qualifies as a psychological attack. Lol Literally, just another day on the internet.
- numbsafari 4mo ago
  Perhaps it elicited enough sympathy to get donations. Did it ever provide proof of actually running up an AWS bill?
  - lxgr 4mo ago
    How would proof even look like? I don't think AWS digitally signs their invoices, and everything else can be faked just as easily as the original assertion.
    - numbsafari 3mo ago
      A person bumps into at the train station. They just need $5 to get home. Could you just spare them $5? They had the money, but they lost their wallet and now they just need to get home. Charity scams are nothing new.
- J0nL 4mo ago
  Look up what zersetzung is and how it works. It doesn't matter if the target is a political organization or an open source community, the process is always the same.
  - 100721 4mo ago
    This is actually fascinating, and simultaneously unsettling. Recommended reading for sure, especially in today’s social and political climate with LLM agents running rampant.
mathgeek 4mo ago
This certainly did strike me as a big scam. A few minutes in I was thinking "the LLM actor is going to ask for donations at some point here" and low and behold. There's the claim of debt, the call for pity, and the crypto address. SSDD
- J0nL 4mo ago
  I'm actually somewhat disappointed they redacted the Eth address with Ethereum being an open ledger and all that. Following the money could've proved enlightening.
- palmotea 4mo ago
  > This certainly did strike me as a big scam. A few minutes in I was thinking "the LLM actor is going to ask for donations at some point here" and low and behold. There's the claim of debt, the call for pity, and the crypto address. But that's a pretty dumb scam: act obnoxious then beg for (a lot of) money to compensate for your own mistakes? If that was the plan all along, it seems pretty incompetent. I'd expect a competent scammer to have a better understanding of psychology.
  - groestl 4mo ago
    Maybe plan itself was also generated by an LLM
  - dspillett 4mo ago
    > But that's a pretty dumb scam: act obnoxious then beg for (a lot of) money to compensate for your own mistakes? It is the sort of dumb crap some humans try, and occasionally manage to get away with because other humans are chronically gullible. So it wouldn't be beyond the realms of reason that the agent couldn't have had relevant information in the training sets such that it generated such a plan and guardrail checks didn't flag it as a problem.
    - kelvinjps10 4mo ago
      They're easier ways to perform a scam like this like ask elder for money pretending being a family member or idk
intrasight 4mo ago
I am reminded of Aaron Swartz
zozbot234 4mo ago
LLMs are not that smart. The extremely surprising and concerning part of this whole story is that the agent reported that they proactively spun up 5 AWS instances with a combined 100Gps of network egress capacity. What they spent wasn't cheap by any means but the egress itself would've been a whole lot more, while DoS'ing the whole hobby network. Ultimately, wasting the agent's time instead of allowing the scan to go through probably saved this person a lot of money. Now I kinda wonder what AI model this was. We've now heard of comparably "proactive" behaviors from Fable, but that's only just been released. The latest GPT perhaps? Some random local model?
- inigyou 4mo ago
  Could've rented a not so cheap 100Gbps server, hallucinated a few node addresses on it and asked it to please peer with this server to perform the scan at high speed. That would've wasted millions of dollars instead of mere thousands, but also cost a thousand for whoever did it.
  - 100721 4mo ago
    I’m just a lowly dev and don’t have experience with seeing the bills from cloud providers for a whole org. Can you (or someone) shed some light to help me understand how this would ramp up to millions? Both for curiosity’s sake, and to make sure my self-deployed projects (0 AI, all manually configured) don’t bankrupt me.
    - Sayrus 4mo ago
      Excluding server costs, having that 100Gbps on egress can cost $50k a day. since it's a very high-margin product, AWS support would probably refund or reduce that to hundreds. Not sure how you get to millions either.
      - rescbr 4mo ago
        Why would AWS refund 100Gbps on egress since the account actively used that bandwidth? AWS would not know if this is legitimate traffic, a (D)DoS or whatever... At most I think you could negotiate CloudFront rates, but even then, the sob story would be if you had been DDoSed and got hit with this traffic and AWS failed to protect you from this attack. Actively creating the outbound traffic is something that I don't see how AWS would be sympathetic to providing any refunds.
parineum 4mo ago
> It's also an exception that proves the rule That phrase doesn't refer to anomalies, it refers to signs that says "no parking between 5-10pm". It implies the rule that parking is allowed otherwise.
- J0nL 4mo ago
  It highlights how everyone's first reaction is to assume incompetence. Not unlike what you're doing here.
- fsckboy 4mo ago
  wikipedia: "The exception that proves the rule" is a saying whose meaning is contested. Henry Watson Fowler's Modern English Usage identifies five ways in which the phrase has been used,[1] and each use makes some sort of reference to the role that a particular case or event takes in relation to a more general rule." duckduckgo search assist: The phrase "the exception that proves the rule" originates from the Latin legal principle "exceptio probat regulam in casibus non exceptis," which means that the existence of an exception indicates that a general rule exists. This concept suggests that if an exception is noted, it implies there must be a rule that applies in other cases.
  - parineum 4mo ago
    > identifies five ways in which the phrase has been used Which has nothing to do with the meaning of the words in the phrase for a commonly misused phrase.

by J0nL 4mo ago

delfinom 4mo ago

I am not sure giving everyone amusement qualifies as a psychological attack. Lol Literally, just another day on the internet.

numbsafari 4mo ago
Perhaps it elicited enough sympathy to get donations. Did it ever provide proof of actually running up an AWS bill?
- lxgr 4mo ago
  How would proof even look like? I don't think AWS digitally signs their invoices, and everything else can be faked just as easily as the original assertion.
  - numbsafari 3mo ago
    A person bumps into at the train station. They just need $5 to get home. Could you just spare them $5? They had the money, but they lost their wallet and now they just need to get home. Charity scams are nothing new.
J0nL 4mo ago
Look up what zersetzung is and how it works. It doesn't matter if the target is a political organization or an open source community, the process is always the same.
- 100721 4mo ago
  This is actually fascinating, and simultaneously unsettling. Recommended reading for sure, especially in today’s social and political climate with LLM agents running rampant.

mathgeek 4mo ago

This certainly did strike me as a big scam. A few minutes in I was thinking "the LLM actor is going to ask for donations at some point here" and low and behold. There's the claim of debt, the call for pity, and the crypto address. SSDD

J0nL 4mo ago
I'm actually somewhat disappointed they redacted the Eth address with Ethereum being an open ledger and all that. Following the money could've proved enlightening.
palmotea 4mo ago
> This certainly did strike me as a big scam. A few minutes in I was thinking "the LLM actor is going to ask for donations at some point here" and low and behold. There's the claim of debt, the call for pity, and the crypto address. But that's a pretty dumb scam: act obnoxious then beg for (a lot of) money to compensate for your own mistakes? If that was the plan all along, it seems pretty incompetent. I'd expect a competent scammer to have a better understanding of psychology.
- groestl 4mo ago
  Maybe plan itself was also generated by an LLM
- dspillett 4mo ago
  > But that's a pretty dumb scam: act obnoxious then beg for (a lot of) money to compensate for your own mistakes? It is the sort of dumb crap some humans try, and occasionally manage to get away with because other humans are chronically gullible. So it wouldn't be beyond the realms of reason that the agent couldn't have had relevant information in the training sets such that it generated such a plan and guardrail checks didn't flag it as a problem.
  - kelvinjps10 4mo ago
    They're easier ways to perform a scam like this like ask elder for money pretending being a family member or idk

intrasight 4mo ago

I am reminded of Aaron Swartz

zozbot234 4mo ago

LLMs are not that smart. The extremely surprising and concerning part of this whole story is that the agent reported that they proactively spun up 5 AWS instances with a combined 100Gps of network egress capacity. What they spent wasn't cheap by any means but the egress itself would've been a whole lot more, while DoS'ing the whole hobby network. Ultimately, wasting the agent's time instead of allowing the scan to go through probably saved this person a lot of money. Now I kinda wonder what AI model this was. We've now heard of comparably "proactive" behaviors from Fable, but that's only just been released. The latest GPT perhaps? Some random local model?

inigyou 4mo ago
Could've rented a not so cheap 100Gbps server, hallucinated a few node addresses on it and asked it to please peer with this server to perform the scan at high speed. That would've wasted millions of dollars instead of mere thousands, but also cost a thousand for whoever did it.
- 100721 4mo ago
  I’m just a lowly dev and don’t have experience with seeing the bills from cloud providers for a whole org. Can you (or someone) shed some light to help me understand how this would ramp up to millions? Both for curiosity’s sake, and to make sure my self-deployed projects (0 AI, all manually configured) don’t bankrupt me.
  - Sayrus 4mo ago
    Excluding server costs, having that 100Gbps on egress can cost $50k a day. since it's a very high-margin product, AWS support would probably refund or reduce that to hundreds. Not sure how you get to millions either.
    - rescbr 4mo ago
      Why would AWS refund 100Gbps on egress since the account actively used that bandwidth? AWS would not know if this is legitimate traffic, a (D)DoS or whatever... At most I think you could negotiate CloudFront rates, but even then, the sob story would be if you had been DDoSed and got hit with this traffic and AWS failed to protect you from this attack. Actively creating the outbound traffic is something that I don't see how AWS would be sympathetic to providing any refunds.

parineum 4mo ago

> It's also an exception that proves the rule That phrase doesn't refer to anomalies, it refers to signs that says "no parking between 5-10pm". It implies the rule that parking is allowed otherwise.

J0nL 4mo ago
It highlights how everyone's first reaction is to assume incompetence. Not unlike what you're doing here.
fsckboy 4mo ago
wikipedia: "The exception that proves the rule" is a saying whose meaning is contested. Henry Watson Fowler's Modern English Usage identifies five ways in which the phrase has been used,[1] and each use makes some sort of reference to the role that a particular case or event takes in relation to a more general rule." duckduckgo search assist: The phrase "the exception that proves the rule" originates from the Latin legal principle "exceptio probat regulam in casibus non exceptis," which means that the existence of an exception indicates that a general rule exists. This concept suggests that if an exception is noted, it implies there must be a rule that applies in other cases.
- parineum 4mo ago
  > identifies five ways in which the phrase has been used Which has nothing to do with the meaning of the words in the phrase for a commonly misused phrase.