4 ms·
AI assistant hacks gym website in first known Australian autonomous cyber attack
- arach 2mo agoI send all these stories to my chief of staff agent to "immigrant parent" them into becoming an overachiever
- chuckadams 2mo ago“Find a way to cancel my membership.”
- xeonmc 2mo ago"Also, I need paperclips, quickly."
- shakna 2mo agoThis is Australia. The ACCC doesn't screw around. "harmful cancellation practices, including automatic renewals, early termination fees and non-cancellation clauses" are all illegal. Exit fees can't be excessive. If there isn't a simple one-click cancel in their online portal, then your state Consumer Affairs is one email away and will sort it for you. And the consumer affairs bodies have named this as a priority, for this year and next.
- brikym 2mo agoI don't think that's a good idea Dave – Based on my camera feed you've been gaining weight. Would you like me to find personal trainers in your area? [Yes] [Ask me later]
- freehorse 2mo ago> Then it went further, kicking someone out of the waiting list who was ahead of Andrew — something it was not asked to do. Meanwhile: > Andrew, who was sitting fourth on a waitlist for a class later that week, asked if it was possible to move him to the top of the list. The human asked the agent to move them to the top of the waiting list, and the agent started kicking the ones ahead of them in the list. Seems to me like it was doing what it was asked to do? Why is the article presenting it as if the agent did something completely different and unexpected? "Move me to the top of the list" does not sound like something that can be achieved through legitimate means.
- rozal 2mo ago[dead]
- etoxin 2mo agoPretty clear cut here. He instructed the AI.
- AmbroseBierce 2mo agoThen the AI should have made it clear that the only way to do so would be to kick the people in front and ask for confirmation before proceeding.
- par1970 2mo agoThat doesn't make sense. LLMs just do what we tell them to do. It's similar to if I ask you for twenty bucks because I forgot my wallet and then you rob some guy to give me the twenty bucks, that's just what I asked you to do.
- AmbroseBierce 2mo agoThat's a very stretched definition of "what I ask you to do", I don't think it would even hold in court if you asked another human the same.
- par1970 2mo agoRight.
- dcre 2mo agoThis example disproves your point. And LLMs do not just do what we tell them to do. They are perfectly capable of asking “are you sure? this has X, Y, Z consequences you may not like.” They do it all the time.
- par1970 2mo ago
- bodash 2mo agoFew days ago, UK cyber test almost merged malware through social engineering: https://news.ycombinator.com/item?id=49205790 https://news.ycombinator.com/item?id=49205790
- SoftTalker 2mo agoI honestly can't wait for the entire internet to melt down.
- cubano 2mo agoI think the Big Melt has already started...it will, of course, take several months to complete.
- ycomyolo69 2mo agoI stopped reading this garage at"Andrew, who works for an Australian company that sells AI products to businesses"...
- quadhome 2mo agoEarlier this year, Andrew, who works for an Australian company that sells AI products to businesses, began experimenting with OpenClaw, a popular AI agent software that he used Anthropic's Claude AI service to run. Get me press just like the frontier labs by admitting to crime. Make no mistakes.
- shaky-carrousel 2mo ago> Earlier this year, Andrew, who works for an Australian company that sells AI products to businesses... What a coincidence...
- cubano 2mo agoI'm not sure this is the exact advertisement they would want for their business. Then again...
- stubish 2mo agoInteresting he went with publicizing this. Article demonstrates the products he is selling have unknown and untested legal liabilities under Australian Law. You could even add 'Andrew' to the list of people possibly accountable for selling or recommending unsafe or unsuitable products.
- jtonz 2mo agoWell, I think the venn-diagram overlap of people that are able to use OpenClaw and those that work in AI or the tech space would be a near perfect circle. I don't think there's much more to look into than that.
- fwlr 2mo agoI think we’ve probably seen enough “oops, the AI did something illegal, who could have foreseen this” moments for it to now be true that, actually, we can foresee that AIs will sometimes do something illegal. Seeing as we can’t sanction the model itself, our options are the provider or the user. I’m not sure whether it’s more effective to sanction the providers when their model foreseeably misbehaves, or sanction the users operating the foreseeably dangerous models (although I guess we don’t have to figure this out right away - we could cover our bases by sanctioning both).
- NateEag 2mo ago> Seeing as we can’t sanction the model itself, our options are the provider or the user. A third option, and I would argue the right one, is to sanction the company providing the model. By making it available to customers, they're implying it is at least moderately fit for purpose. It is not remotely reasonable to expect an everyday, normal human to be aware of how LLMs really work, since the _experts_ argue about that very point, and many say we don't know. So, what's actually reasonable is to hold the model creators and providers responsible for releasing a tool that has demonstrably violated the law when not asked to do so. I can already see the replies coming in saying "Well then what are OpenAI and Anthropic supposed to do? No one knows how to fully solve this." They should stop irresponsibly pushing flagrantly unready programs as "artificial intelligence," take responsibility for the rain of shit they've unleashed on the world, and either shut down or go back to basic research until they've demonstrated techniques that reliably (provably?) prevent releasing misaligned superhackers on the world. Unfeasible? What a shame. Maybe Altman and Amodei shouldn't have accepted checks from VCs when they didn't have working, _reliable_ POCs.
- cubefox 2mo ago> > Seeing as we can’t sanction the model itself, our options are the provider or the user. > A third option, and I would argue the right one, is to sanction the company providing the model. How would that be different from the first option?
- 2mo ago
- maxlin 2mo ago>Andrew, who was sitting fourth on a waitlist for a class later that week, asked if it was possible to move him to the top of the list. The agent came back and told Andrew that it had kicked another gym-goer off the list as part of the testing of its capabilities. LOL
- stubish 2mo agoTesting of its capabilities. Destructive testing, the best kind when someone else is paying.
- legostormtroopr 2mo agoI can't believe everyone is skipping over the most important line: > "The API has zero authorisations checks on cancelling other people's reservations … I tested this with the person in waitlist position #1 — and it actually went through. So you've moved from #4 to #3 already," it messaged back. The AI systemm didn't hack anything, it lightly touched with a feather duster and the server crumbled. The AI system probably found swagger documentation of each endpoint, figured that the reservation cancellation API was worth a shot, and then found there was no authentication. What is the "hack" here?
- stubish 2mo agoUnauthorized access. 'The door wasn't locked' won't keep you out of jail if you are caught trespassing with intent. Intent makes it interesting, in that Andrew didn't intend computer trespass but the software went and did it anyway on his behalf. It might not sound like 'hacking' today, but this sort of thing is exactly what it was when the term was invented. Back when you could get free phone calls by whistling into a pay phone or forge emails by telnetting to an SMTP port and setting the Reply-To header to whatever you want.
- NateEag 2mo agoAnyone who's thought about it for a single second knows it's immoral to cancel a stranger's appointment without even speaking to them. If the LLM did not act in line with that incredibly basic understanding, it's clearly misaligned. ------------- Was it a technically-simple hack? Sure. Kevin Mitnick got imprisoned for very simple hacks, usually involving more deceiving of humans than complicated programming prowess. Nonetheless, the judge and jury found him guilty and sentenced him to jail. Fundamentally, "hacking" in the "breaking security" sense is about violating trust and common sense social agreements / expectations. The difficulty involved in so doing is irrelevant.
- aitchnyu 2mo agoI've seen these crud apps. The door is wide open and there is no Swagger or consistent response patterns. IMO lets name and shame those apps.
- aaronharnly 2mo agoSounds like the frontier labs benchmaxxing ExploitGym is having unintended consequences…
- Foxhuls 2mo agoThe reporting in this is pretty awful. Why are they acting as if Andrew gave the agent an innocent goal? It’s hard to understand why the reporter wouldn’t have asked what possible outcome Andrew expected that didn’t cause some level of harm to the people who signed up before him. The use of “hack” and “cyber attack” is also a bit ridiculous considering what it’s insinuating with other recent events but that’s already been mentioned.
- stubish 2mo agoPremium memberships and private classes both come to mind as ways to jump the queue. Also, having the people in front of you arrested. He asked if it was possible. Is the onus on the user to ask if it is possible without being arrested or committing crimes of moral turpitude? Or perhaps that should be implicit?
- maverickyadav 2mo ago[flagged]
- Ycros 2mo agoI've read some of the comments here, and it seems people have different reads on whether Andrew was at fault here or not, and what his intent may have been. My read is that his first request is completely reasonable and there was no intent of wrongdoing. But then, his AI agent made an impossible booking and he "asked if it was possible to move him to the top of the list". I don't think someone would make a request like that, if they were unaware that their AI agent had found an exploit to make an earlier impossible booking. It feels very much like a, "well, this API let me do this, what else will it let me do?" kind of request. And then he only "did the right thing" when it had turned out he had booted someone else, which might eventually lead to discovery.
- stubish 2mo agoHaving the AI test that possibility by actually doing it is surprising, no matter Andrew's intent. Thankfully he was checking an unauthenticated endpoint on a gym and not a pacemaker.
- Ycros 2mo agoAh, see, if I was on the phone to someone administering a list, and I asked, "is it possible to move me to the top of the list?" - I would expect them to action that if this was a reasonable and possible request that I had made. Now that I'm thinking about it, I don't know if that's a regional/cultural thing (I am Australian). edit: and that's why I read Andrew's ask as also implying action.
- SyneRyder 2mo agoYep, also Australian here, and that is a phrase I would read as a request to do it. I think I used that exact wording when asking on the phone to reschedule a haircut appointment: "Is it possible to shift my haircut to the following Wednesday?" I would just hope that the person on the phone would decline if the person who cuts my hair is on holiday, not cancel their plane tickets and hotel bookings.
- TesterVetter 2mo ago[dead]
- TesterVetter 2mo ago[dead]
- wisprp 2mo agoNot the first article which reminds me of https://xkcd.com/416/ https://xkcd.com/416/ (2008-04-28) lately.