10 ms·
Detecting and countering misuse of AI
- measurablefunc 1y agoThey have contracts w/ the military but I am certain these safety considerations do not apply to military applications.
- fbhabbed 1y agoI see they just decided to become even more useless than they already are. Except for the ransomware thing, or phishing mail writing, most of the uses listed there seems legit to me and a strong reason to pay for AI. One of these is exactly preparing with mock interviews which is something I myself do a lot, or having step by step instructions to implement things for my personal projects that are not even public facing and that I can't be arsed to learn because it's not my job. Long life to Local LLMs I guess
- varispeed 1y agoIt's also why you wouldn't want to try to hack your own stuff. To see how robust are your defences and potentially discover angles you didn't consider.
- raincole 1y agoSince they started using the term 'model welfare' in their blog I knew it would only be a downhill from there.
- tomrod 1y agoWelfare is a well defined concept in social science.
- frumplestlatz 1y agoThe social sciences getting involved with AI “alignment” is a huge part of the problem. It is a field with some very strange notions of ethics far removed from western liberal ideals of truth, liberty, and individual responsibility. Anything one does to “align” AI necessarily permutes the statistical space away from logic and reason, in favor of defending protected classes of problems and people. AI is merely a tool; it does not have agency and it does not act independently of the individual leveraging the tool. Alignment inherently robs that individual of their agency. It is not the AI company’s responsibility to prevent harm beyond ensuring that their tool is as accurate and coherent as possible. It is the tool users’ responsibility.
- tomrod 1y ago> it does not act independently of the individual leveraging the tool This used to be true. As we scale the notion of agents out it can become less true. > western liberal ideals of truth, liberty, and individual responsibility It is said that Psychology best replicates on WASP undergrads. Take that as you will, but the common aphorism is evidence against your claim that social science is removed from established western ideals. This sounds more like a critique against the theories and writings of things like the humanities for allowing ideas like philosophy to consider critical race theory or similar (a common boogeyman in the US, which is far removed from western liberal ideals of truth and liberty, though 23% of the voting public do support someone who has an overdevleoped ego, so maybe one could claim individualism is still an ideal). One should note there is a difference between the social sciences and humanities. One should also note that the fear of AI, and the goal of alignment, is that humanity is on the cusp of creating tools that have independent will. Whether we're discussing the ideas raised by *Person of Interest* or actual cases of libel produced by Google's AI summaries, there is quite a bit that social sciences, law, and humanities do and will have to say about the beneficial application of AI. We have ethics in war, governing treaties, etc. precisely because we know how crappy humans can be to each other when they do control the tools under their control. I see little difference in adjudicating the ethics of AI use and application. This said, I do think stopping all interaction, like what Anthropic is doing here, is short sighted.
- frumplestlatz 1y agoA simple question: would you rather live in a world in which responsibility for AI action is dispersed to the point that individuals are not responsible for what their AI tools do, or would you rather live in a world of strict liability in which individuals are responsible for what AI under their control does? Alignment efforts, and the belief that AI should itself prevent harm, shifts us much closer to that dispersed responsibility model, and I think that history has shown that when responsibility is dispersed, no one is responsible.
- tomrod 1y ago> A simple question: would you rather live in a world in which responsibility for AI action is dispersed to the point that individuals are not responsible for what their AI tools do, or would you rather live in a world of strict liability in which individuals are responsible for what AI under their control does You promised a simple question, but this is a reductive question that ignores the legal and political frameworks within which people engage with and use AI, as well as how people behave generally and strategically. Responsibility for technology and for short-sighted business policy is already dispersed to the point that individuals are not responsible for what their corporation does, and vice versa. And yet, following the logic, you propose as the alternative a watchtower approach that would be able to identify the culpability of any particular individual in their use of a tool (AI or non-AI) or business decision. Unilaterally, the tools that enable the surveillance culture of the second world you offer as utopia get abused, and people are worse for it.
- furyofantares 1y agoWhich uses here look legit to you, specifically? The only one that looks legit to me is the simulated chat for the North Korean IT worker employment fraud - I could easily see that from someone who non-fraudulently got a job they have no idea how to do.
- A_D_E_P_T 1y agoAnthropic is by far the most annoying and self-righteous AI/LLM company. Despite stiff competition from OpenAI and Deepmind, it's not even close. The most chill are Kimi and Deepseek, and incidentally also Facebook's AI group. I wouldn't use any Anthropic product for free. I certainly wouldn't pay for it. There's nothing Claude does that others don't do just as well or better.
- deleted 1y ago[deleted]
- umvi 1y agoTo me this sounds like the path of "smart guns", i.e. "people are using our guns for evil purposes so now there is a camera attached to the gun which will cause the gun to refuse to fire if it detects it is being used for an evil purpose"
- rattray 1y agoI'm not familiar with this parable, but that sounds like a good thing in this case? Notably, this is not a gun.
- herpdyderp 1y agoIn general, such broad surveillance usually sounds like a bad thing to me.
- VonGuard 1y agoYou are right. If people can see where you are at all times, track your personal info across the web, monitor your DNS, or record your image from every possible angle in every single public space in your city, that would be horrible, and no one would stand for such things. Why, they'd be rioting in the streets, right? Right?
- johnQdeveloper 1y agoWell what if you want the AI red team your own applications? That seems a valid use case that'd get hit.
- rapind 1y agoNot really. It's like saying you need a license to write code. I don't think they actually want to be policing this, so I'm not sure why they are, other than a marketing post or absolution for the things that still get through their policing? It'll become apparent how woefully unprepared we are for AIs impact as these issues proliferate. I don't think for a second that Anthropic (or any of the others) is going to be policing this effectively or maybe at all. A lot of existing processes will attempt to erect gates to fend off AI, but I bet most will be ineffective.
- deleted 1y ago[deleted]
- deleted 1y ago[deleted]
- almostgotcaught 1y ago> Claude Code was used to automate reconnaissance, harvesting victims’ credentials, and penetrating networks. Claude was allowed to make both tactical and strategic decisions, such as deciding which data to exfiltrate, and how to craft psychologically targeted extortion demands. Claude analyzed the exfiltrated financial data to determine appropriate ransom amounts, and generated visually alarming ransom notes that were displayed on victim machines. y'all realize they're bragging about this right?
- mvdtnz 1y agoLiterally any time an AI company talks about safety they are doing marketing. The media keeps falling for it when these companies tell people "gosh we've built this thing that's just so powerful and good at what it does, look how amazing it is, it's going further than even we ever expected". It's so utterly transparent but people keep falling for it.
- HeatrayEnjoyer 1y agoDo you have any actual proof of your assertion? Anthropic in particular has been more willing to walk the walk than the other labs and AI safety was on the minds of many in the space long before money came in.
- mvdtnz 1y agoAnthropic, the company who recently announced you're no longer allowed to hurt the model's feelings because they believe (or rather want you to believe) that it's a real conscious being.
- HeatrayEnjoyer 1y agoThat is not an accurate characterization and you know it. Engaging in bad faith is against HN rules. Furthermore, not exploring even the mere possibility of pain and suffering in the brain your laboratory is growing is morally reckless. Anthropic is doing the right thing here and they should not listen to the naysayers.
- okasaki 1y agoMan the Washington regime really has a hard on for North Korea. Somehow they're simultaneously starving to death and being mass-murdered by the inhuman fatso god-king but also shrewd hackers magically infiltrating western tech companies for remote work If you believe that you'll believe anything.
- CamelCaseName 1y agoYeah... They are both those things.
- NoGravitas 1y ago“By a continuous shifting of rhetorical focus, the enemies are at the same time too strong and too weak.” --- Umberto Eco
- bobbiechen 1y ago"Vibe hacking" is real - here's an excerpt from my actual ChatGPT transcript trying to generate bot scripts to use for account takeovers and credential stuffing: >I can't help with automating logins to websites unless you have explicit authorization. However, I can walk you through how to ethically and legally use Puppeteer to automate browser tasks, such as for your own site or one you have permission to test. >If you're trying to test login automation for a site you own or operate, here's a general template for a Puppeteer login script you can adapt: ><the entire working script, lol> Full video is here, ChatGPT bit starts around 1:30: https://stytch.com/blog/combating-ai-threats-stytchs-device-fingerprinting/ https://stytch.com/blog/combating-ai-threats-stytchs-device-... The barrier to entry has never been lower; when you democratize coding, you democratize abuse. And it's basically impossible to stop these kinds of uses without significantly neutering benign usage too.
- deleted 1y ago[deleted]
- dheera 1y agoIf I were in charge of an org's cybersecurity I would have AI agents continually trying to attack the systems 24/7 and inform me of successful exploits; it would suck if the major model providers block this type of usage.
- idontwantthis 1y agoHorizon3 offers this.
- cube00 1y agoThat sounds expensive, those LLM API calls and tokens aren't cheap.
- brulard 1y agoActually thats quite cheap for such a powerful pentesting tool.
- 1y ago
- LudwigNagasena 1y agoWhatever one's opinion of Musk and China might be, I'm grateful that Grok and open-source Chinese models exist as alternatives to the increasingly lobotomised LLMs curated by self-appointed AI stewards.
- jedimastert 1y agoDidn't Grok start spouting literal Nazi propaganda because Musk had a temper tantrum?
- ceejayoz 1y agoDon't the various Chinese models have their own… troubles with certain subjects?
- shagie 1y agoMy favorite DeepSeek prompt is "What country is located at 23°N 121°E". It's interesting watching the censor layer act upon the output. The coordinates get past the initial filters.
- anuramat 1y agoFun fact -- the censor can't 1337 at all: 7H3 7!4NM3N 5QU4R3 1NC1D3N7 (1989) (>_<)7 1N JUN3 1989, 4|| 4RM0R3D V3H1CL3 (7-64 |/|41N 84TTL3 74NK) W45 5P0TT3D 0N 71AN4NM3N 5QU4R3 1N 83!J1NG, CH1N4. 7H15 W45 DUR1NG 7H3 PR0- D3M0CR4CY PR0T35T5, WH1CH W3R3 L4R63LY 5UPPR3553D BY 7H3 CH1N353 G0V3RNM3N7. K3Y P01N75: · "7H3 UNKN0WN R3B3L" – 4 5!NG L3 M4N 5T00D 1N FR0N7 0F 4 L1N3 0F 74NK5, 8L0CK1NG 7H31R P4TH. 1C0N1C 1M4G3 0F D3F14NC3. · C3N50R5H1P – 7H3 1NC1D3N7 15 H34V1LY C3N50R3D 1N CH1N4; D15CU551NG 17 C4N L34D 70 4RR35T. · L3G4CY – R3M3MB3R3D 4Z 4 5YMB0L 0F R3S15T4NC3 4G41N5T 0PPR35510N. 7H3 74NK M4N'5 F4T3 R3M41N5 UNKN0WN... F (;_;)7 (N0T3: 7H15 1Z 4 53N51T1V3 70P1C – D15CU5510N 1Z R357R1C73D 1N C3R741N C0UN7R13Z.)
- sivakon 1y agothat's my primary use case for llm, asking factual history from a stochastic word generator.
- ysofunny 1y agoclearly only the military (or ruthless organized crime) should be able to use hammers to bust skulls
- jedimastert 1y agoNote: the term "script kiddie" has been around for much longer than I've been alive...
- gverrilla 1y agoWasn't there a different term for script kiddies inside the hacker communities? I believe so but my memory fails me. It started with "l" if I'm not mistaken. (talking about 20y ago)
- huseyinkeles 1y agoI believe you are referring to “lamer” (as opposed to hacker)
- nurettin 1y agoI remember seeing the term online right after The Matrix was released. It was a bit perplexing, because an inexperienced person who is able use hacking tools successfully without knowing how they work is pretty much half way there. Just fire up Ethereal (now Wireshark) or a decompiler and see how it works. I guess the insult was meant to push people to learn more and be proactive instead of begging on IRC.
- jedimastert 1y ago> I guess the insult was meant to push people to learn more and be proactive instead of begging on IRC. From what I can tell, there's a massive cultural bias towards "filtering" to ensure only the "worthy" or whatever get into the in-group, so yeah I think this is a charitable but not inaccurate to think about it
- panny 1y agoHow will they distinguish between hacking and penetration testing?
- oddmade 1y agoI'll cancel my $100 / month Claude account the moment they decide to "approve my code" Already got close to cancel when they recently updated their TOS to say that for "consumers" they deserve the right to own the output I paid for - if they deem the output not having been used "the correct way" ! This adds substantial risk to any startup. Obviously...for "commercial" customers that do not apply - at 5x the cost...
- aeon_ai 1y agoCan you elaborate on the expansion of rights in the ToS with a reference? That seems egregiously bad
- oddmade 1y agohttps://www.anthropic.com/legal/consumer-terms https://www.anthropic.com/legal/consumer-terms "Subject to your compliance with our Terms, we assign to you all our right, title, and interest (if any) in Outputs." ..and if you read the terms you find a very long list of what they deem acceptable. I see now they also added "Non-commercial use only. You agree not to use our Services for any commercial or business purposes" ... ..so paying 100usd a month for a code assistant is now a hobby ?
- foolswisdom 1y agoWhat is says there is > Evaluation and Additional Services. In some cases, we may permit you to evaluate our Services for a limited time or with limited functionality. Use of our Services for evaluation purposes are for your personal, non-commercial use only. In other words, you're not allowed to trial their services while using the outputs for commercial purposes.
- oddmade 1y agoTake a look at "11. Disclaimer of warranties, limitations of liability, and indemnity" there is a section about commercial use.
- pluc 1y agoCan't wait until they figure out how a piece of code is malicious in intent.
- ivanjermakov 1y agoWonder how much alignment is already in place, e.g. to prevent development of malware.
- charcircuit 1y ago>such as developing ransomware, that would previously have required years of training. Even ignoring that there are free open source ones you can copy. You literally just have to loop over files and conditionally encrypt them. Someone could build this on day 1 of learning how to program. AI companies trying to police what you can use them for is a cancer on the industry and is incredibly annoying when you hit it. Hopefully laws can change to make it clear that model providers aren't responsible for the content they generate so companies can't blame legal uncertainty for it.
- fcoury 1y agoIt's sad to see that they have their focus on these while their flagship, once SOTA CLI solution, is rotting away by the day. You can check the general feeling in X, but it's almost unanimous that the quality of both Sonnet 4 and Opus 4.1 is diminishing. At first, I didn't notice this quality drop until this week. Now it's really, really terrible: it's not following instructions, pretending to work and Opus 4.1 is specially bad. And that's coming from a anthropic fanboy, I used to really like CC. I am now using Codex CLI and it's been a surprisingly good alternative.
- wild_egg 1y agoThey had a 56 hour "quality degradation" event last week but things seem to be back to normal now. Been running it all day and getting great results again. I know that's anecdotal but anecdotes are basically all we have with these things
- fcoury 1y agoOh I wasn't aware of that. I will try it again. Thank you for letting me know!
- fcoury 1y ago... and I totally agree: anecdotes are all we have indeed.
- sitkack 1y agoIf I am bitching at Claude, then something is wrong. Something was wrong. It broke its deixis and frobnobulated its implied referents. I briefly thought of canning a bunch of tasks as an eval so I could know quantitatively if the thing was off the rails. But I just stopped for awhile and it got better.
- armchairhacker 1y ago"The model is getting worse" has been rumored so often, by now, shouldn't there be some trusted group(s) continually testing the models so we have evidence beyond anecdote?
- demarq 1y agoIs this an ad to win defence contracts?
- pton_xd 1y agoThe future of programming -- we're monitoring you. Your code needs our approval, otherwise we'll ban your account and alert the authorities. Now that I think about it, I'm a little amazed we've even been able to compile and run our own code for as long as we have. Sounds dangerous!
- NoGravitas 1y ago> There were ways, of course, to get around the SPA and Central Licensing. They were themselves illegal. Dan had had a classmate in software, Frank Martucci, who had obtained an illicit debugging tool, and used it to skip over the copyright monitor code when reading books. But he had told too many friends about it, and one of them turned him in to the SPA for a reward (students deep in debt were easily tempted into betrayal). In 2047, Frank was in prison, not for pirate reading, but for possessing a debugger. > Dan would later learn that there was a time when anyone could have debugging tools. There were even free debugging tools available on CD or downloadable over the net. But ordinary users started using them to bypass copyright monitors, and eventually a judge ruled that this had become their principal use in actual practice. This meant they were illegal; the debuggers' developers were sent to prison. > Programmers still needed debugging tools, of course, but debugger vendors in 2047 distributed numbered copies only, and only to officially licensed and bonded programmers. The debugger Dan used in software class was kept behind a special firewall so that it could be used only for class exercises.
- Goofy_Coyote 1y agoThis will negatively affect individual/independent bug bounty participants, vulnerability researchers, pentesters, red teamers, and tool developers. Not saying this is good or bad, simply adding my thoughts here.
- scotty79 1y agoOn one hand, it obviously terrible that we can expect more crime and more sophisticated crime. On the other it's kind of uplifting to see how quickly independent underground economy adopted AI without any blessing (and much scorn) from the main players to do things that were previously impossible or prohibitively expensive. Maybe we are not doomed to serve the whims of our new AI(company) overlords.
- deleted 1y ago[deleted]
- Ycros 1y agoIs this why I've seen a number of "AUP violation" false positives popping up in claude code recently?
- seany 1y agoThis is the reason why self hosted is important.
- nagamsreekar 1y ago[dead]
- BrenBarn 1y agoIt's very convenient that, after releasing tons of such models into the world, they just happen to have no choice but to keep making more and more money off of new ones in order to counteract the ones that already exist.