4 ms·

I’m surprised to see no mention of the attack vectors that – for lack of a better term – I call “drowning in crap”: - send plausible phishing messages personal

by klabb3 3y ago

I’m surprised to see no mention of the attack vectors that – for lack of a better term – I call “drowning in crap”: - send plausible phishing messages personalized to you (from data leaks, brokers) goes from expensive to cheap and automatable - evade content based filtering such as spam filters to flood people with more crap - generate so much plausible but regurgitated crap content for ranking systems (search, product reviews, friend requests) - etc, etc These things are already before LLMs one of the biggest problems across the board online. I would expect LLMs in particular to disrupt that power balance significantly in favor of the spammers and scammers. Is the implicit lack of attention to this issue that it’s not that bad? Won’t get worse? Or are we just supposed to learn how to swim in a sea of crap?

fxd123 3y ago
I assume that would be under "Poisoning"
- wizardforhire 3y ago
  In the spirit of endless hn pedantics… Poisoning implies a certain amount of finesse. All poisons of course being enzyme inhibitors in themselves and enzymes being a fancy biological term for a catalyst… but digress and no. I think drowning is very much the apt descriptor of this phenomenon.
  - sjfjsjdjwvwvc 3y ago
    [dead]
  - winwang 3y ago
    "poison scales with dex, drowning is a weapon passive"
- klabb3 3y ago
  No, double checked: > Poisoning attacks occur in the training phase by introducing corrupted data. The distinction matters because poisoning (and all other listed attacks) impact the quality of the product and their users: I’d argue that AI providers are already incentivized to fix those. Whereas if say a hypothetical company ClosedAI is providing scammers/spammers with daily fresh loads of crap to use outside of the platform they simply make money from that. In other words, they have a first-order-incentive to be suppliers for these grey/bad actors. I guess this would be similar to the relationship between ad-platforms and scammy/shady advertisers.
throwaway789394 3y ago
[dead]
golly_ned 3y ago
The link pertains to attacks on AI systems, not attacks using AI systems.
nik_seetharaman 3y ago
As one of the posters below mentioned, this focuses on attacks on AI systems, not attacks by (or utilizing) AI systems. In general there is a concerning apathy in the infosec community about the implications of generative AI on offensive tooling and what it means for already overwhelmed defenders. Seems to be much more of a focus on protecting against prompt injection, etc which doesn’t make much sense when you consider there are many, many models on HuggingFace that can easily be fine tuned to perform whatever nefarious task you may desire - no prompt injection required whatsoever. We wrote up more thoughts here: https://www.wraithwatch.com/post/adapt-or-die-generative-ai-the-revolution-of-american-cyber-defense https://www.wraithwatch.com/post/adapt-or-die-generative-ai-...
cyanydeez 3y ago
during Trump's tenure, there was likely a whole bunch of bots betting on market movement and smearing the shit to make it happen. maybe the political shit is fearful, but if capitalism requires a market and that market requires clear signals, well, here now is the greatshitstorm of our times.

by klabb3 3y ago

fxd123 3y ago

I assume that would be under "Poisoning"

wizardforhire 3y ago
In the spirit of endless hn pedantics… Poisoning implies a certain amount of finesse. All poisons of course being enzyme inhibitors in themselves and enzymes being a fancy biological term for a catalyst… but digress and no. I think drowning is very much the apt descriptor of this phenomenon.
- sjfjsjdjwvwvc 3y ago
  [dead]
- winwang 3y ago
  "poison scales with dex, drowning is a weapon passive"
klabb3 3y ago
No, double checked: > Poisoning attacks occur in the training phase by introducing corrupted data. The distinction matters because poisoning (and all other listed attacks) impact the quality of the product and their users: I’d argue that AI providers are already incentivized to fix those. Whereas if say a hypothetical company ClosedAI is providing scammers/spammers with daily fresh loads of crap to use outside of the platform they simply make money from that. In other words, they have a first-order-incentive to be suppliers for these grey/bad actors. I guess this would be similar to the relationship between ad-platforms and scammy/shady advertisers.

throwaway789394 3y ago

[dead]

golly_ned 3y ago

The link pertains to attacks on AI systems, not attacks using AI systems.

nik_seetharaman 3y ago

As one of the posters below mentioned, this focuses on attacks on AI systems, not attacks by (or utilizing) AI systems. In general there is a concerning apathy in the infosec community about the implications of generative AI on offensive tooling and what it means for already overwhelmed defenders. Seems to be much more of a focus on protecting against prompt injection, etc which doesn’t make much sense when you consider there are many, many models on HuggingFace that can easily be fine tuned to perform whatever nefarious task you may desire - no prompt injection required whatsoever. We wrote up more thoughts here: https://www.wraithwatch.com/post/adapt-or-die-generative-ai-the-revolution-of-american-cyber-defense https://www.wraithwatch.com/post/adapt-or-die-generative-ai-...

cyanydeez 3y ago

during Trump's tenure, there was likely a whole bunch of bots betting on market movement and smearing the shit to make it happen. maybe the political shit is fearful, but if capitalism requires a market and that market requires clear signals, well, here now is the greatshitstorm of our times.