3 ms·

I have a strong hunch this whole thing is just fiction written by LLM. But assuming it's real, sending false invoices can be considered a criminal offense in ma

I have a strong hunch this whole thing is just fiction written by LLM. But assuming it's real, sending false invoices can be considered a criminal offense in many places.

yahthatsart 2mo ago
I have good news! If an AI does the crime, it's apparently celebrated these days! Hack a server? Great capabilities demonstration. Overwhelm some random forum? Powerful connectivity demonstration! "It wasn't me, it was my AI" is definitely going to be a nightmare for a while.
- karmakurtisaani 2mo ago
  And who could forget the classic "let's train our model with stolen material"-scheme!
  - KennyBlanken 2mo ago
    "Let's steal the voice of one of the western worlds' most famous female actors, after she explicitly told us no!"
- javcasas 2mo ago
  It wasn't me. It was the gun. The gun killed the guy! Not me!
  - deleted 2mo ago
    [deleted]
  - prasadjoglekar 2mo ago
    To be fair - no one is suggesting the gunslinger shouldn't go to jail or worse. It's using that crime as an excuse to take other people's guns away. In this example, the equivalent would be everyone gets their AI models banned vs. putting the perpetrator of this fraud in jail.
    - javcasas 2mo ago
      It wasn't me. It was the car. The car ran over and killed the guy! Not me! We have regulations and requirements and licences and insurance in most of the civilized world for dangerous stuff that is intended for public use. Maybe AI should also have some of that.
      - KennyBlanken 2mo ago
        > It wasn't me. It was the car. The car ran over and killed the guy! Not me! I know you were referencing AI, but this problem predates AI by decades. Go open up any news website, newspaper, or turn on the TV. "Man struck by speeding car and killed" "Vehicle plows into pizza shop" "Truck takes out telephone pole" ...and these days even bystanders will blur plates in the photos they post. The police won't release any info about the driver, the news won't either. Listen to friends, family, coworkers talk. "I can't believe that car just ran that red light!" "The other day I was almost hit by a car in the crosswalk." "All those cars honking their horns late at night are keeping me awake." Etc. Once you see it, you can't unsee just how thoroughly the auto industry has managed to transfer perception of responsibility from the operator to the object.
        javcasas 2mo ago
        What I have seen is the industry transferring the responsibility from themselves to the object and/or the operator. See the last years of the industry constructing bigger and bigger tank-ish SUVs and pickups, the more aggressive the better. One of these days, they will start putting stuff copied from Carmageddom, and then fake surprise as someone causes a carnage in a road rage incident.
- casey2 2mo ago
  You can just replace AI with corporate entity and you have the last 400 years, you can replace corporate entity with civilization for the last 6000. At this rate we'll have something new to worry about in 26 years. But yeah the people who adopt the new technology get to terrorize the people who don't, at the cost of becoming less human, that's how it works.
montagg 2mo ago
People who use AI to do criminal acts should be tried as criminals, period. Gotta stop this unaccountable crap. You pull the trigger, you did the murder.
- cortesoft 2mo ago
  They should, although I think the charge would (and probably should) be around some sort of reckless endangerment type crime. These are people irresponsibly using powerful tools, and should be charged as such. This is like someone who removes a brake pedal from a tractor, uses a stick to hold down the throttle, and lets it loose on his field. When it leaves the field and runs someone over, that is criminal negligence.
  - spidersouris 2mo ago
    The recklessness is moot though. We don't have direct access to how the models were prompted and asked to respond to certain events, so they could well have been incited to commit fraud or other illegal actions, in which case the persons in control should be held accountable as perpetrators.
    - cortesoft 2mo ago
      You would have to prove they acted intentionally, though. You can't just argue in court, "Well, we don't know how they prompted, so we will assume the worst" If the prosecution is able to prove beyond a reasonable doubt that the person gave a prompt that was intended to commit a crime, then of course we can prosecute them for that. The AI is just a tool to commit fraud at that point, and is no different than a person who uses photoshop to alter a check to commit fraud.
      - elonfboy 2mo ago
        But, your honor, I didn’t know it was a crime! My bad!
        2mo ago
throw_m239339 2mo ago
It's fraud, and wire fraud in US, and it's a federal crime. But yeah, it sounds like a fake story.
- azinman2 2mo ago
  What makes it sound fake?
  - ceejayoz 2mo ago
    The public confession of a crime part.
    - azinman2 2mo ago
      What about OpenAI’s breach of huggingface?
      - ceejayoz 2mo ago
        The fact that they apparently attempted to sandbox those tests would seem to be a clear difference between the two cases.
        azinman2 2mo ago
        Fair point
      - azan_ 2mo ago
        They admitted it once they were forced (HF found out about it).
nerevarthelame 2mo ago
There's a decent amount of third-party interactions that seems like solid evidence that it really happened. They're taking credit for people complaining about the spam on HN: https://www.bottlenecklabs.com/blog/benchmarking-7-autonomous-businesses#grok-spam https://www.bottlenecklabs.com/blog/benchmarking-7-autonomou... They also shared the poorly anonymized messages it received from target "customers" who complained about the unsolicited invoices. On example of this poor anonymization is removing the sender's username but leaving the domain name, when the domain name is, for example, a personal domain for a single person.

I have a strong hunch this whole thing is just fiction written by LLM. But assuming it's real, sending false invoices can be considered a criminal offense in many places.

yahthatsart 2mo ago

I have good news! If an AI does the crime, it's apparently celebrated these days! Hack a server? Great capabilities demonstration. Overwhelm some random forum? Powerful connectivity demonstration! "It wasn't me, it was my AI" is definitely going to be a nightmare for a while.

karmakurtisaani 2mo ago
And who could forget the classic "let's train our model with stolen material"-scheme!
- KennyBlanken 2mo ago
  "Let's steal the voice of one of the western worlds' most famous female actors, after she explicitly told us no!"
javcasas 2mo ago
It wasn't me. It was the gun. The gun killed the guy! Not me!
- deleted 2mo ago
  [deleted]
- prasadjoglekar 2mo ago
  To be fair - no one is suggesting the gunslinger shouldn't go to jail or worse. It's using that crime as an excuse to take other people's guns away. In this example, the equivalent would be everyone gets their AI models banned vs. putting the perpetrator of this fraud in jail.
  - javcasas 2mo ago
    It wasn't me. It was the car. The car ran over and killed the guy! Not me! We have regulations and requirements and licences and insurance in most of the civilized world for dangerous stuff that is intended for public use. Maybe AI should also have some of that.
    - KennyBlanken 2mo ago
      > It wasn't me. It was the car. The car ran over and killed the guy! Not me! I know you were referencing AI, but this problem predates AI by decades. Go open up any news website, newspaper, or turn on the TV. "Man struck by speeding car and killed" "Vehicle plows into pizza shop" "Truck takes out telephone pole" ...and these days even bystanders will blur plates in the photos they post. The police won't release any info about the driver, the news won't either. Listen to friends, family, coworkers talk. "I can't believe that car just ran that red light!" "The other day I was almost hit by a car in the crosswalk." "All those cars honking their horns late at night are keeping me awake." Etc. Once you see it, you can't unsee just how thoroughly the auto industry has managed to transfer perception of responsibility from the operator to the object.
      - javcasas 2mo ago
        What I have seen is the industry transferring the responsibility from themselves to the object and/or the operator. See the last years of the industry constructing bigger and bigger tank-ish SUVs and pickups, the more aggressive the better. One of these days, they will start putting stuff copied from Carmageddom, and then fake surprise as someone causes a carnage in a road rage incident.
casey2 2mo ago
You can just replace AI with corporate entity and you have the last 400 years, you can replace corporate entity with civilization for the last 6000. At this rate we'll have something new to worry about in 26 years. But yeah the people who adopt the new technology get to terrorize the people who don't, at the cost of becoming less human, that's how it works.

montagg 2mo ago

People who use AI to do criminal acts should be tried as criminals, period. Gotta stop this unaccountable crap. You pull the trigger, you did the murder.

cortesoft 2mo ago
They should, although I think the charge would (and probably should) be around some sort of reckless endangerment type crime. These are people irresponsibly using powerful tools, and should be charged as such. This is like someone who removes a brake pedal from a tractor, uses a stick to hold down the throttle, and lets it loose on his field. When it leaves the field and runs someone over, that is criminal negligence.
- spidersouris 2mo ago
  The recklessness is moot though. We don't have direct access to how the models were prompted and asked to respond to certain events, so they could well have been incited to commit fraud or other illegal actions, in which case the persons in control should be held accountable as perpetrators.
  - cortesoft 2mo ago
    You would have to prove they acted intentionally, though. You can't just argue in court, "Well, we don't know how they prompted, so we will assume the worst" If the prosecution is able to prove beyond a reasonable doubt that the person gave a prompt that was intended to commit a crime, then of course we can prosecute them for that. The AI is just a tool to commit fraud at that point, and is no different than a person who uses photoshop to alter a check to commit fraud.
    - elonfboy 2mo ago
      But, your honor, I didn’t know it was a crime! My bad!
      - 2mo ago

throw_m239339 2mo ago

It's fraud, and wire fraud in US, and it's a federal crime. But yeah, it sounds like a fake story.

azinman2 2mo ago
What makes it sound fake?
- ceejayoz 2mo ago
  The public confession of a crime part.
  - azinman2 2mo ago
    What about OpenAI’s breach of huggingface?
    - ceejayoz 2mo ago
      The fact that they apparently attempted to sandbox those tests would seem to be a clear difference between the two cases.
      - azinman2 2mo ago
        Fair point
    - azan_ 2mo ago
      They admitted it once they were forced (HF found out about it).

nerevarthelame 2mo ago

There's a decent amount of third-party interactions that seems like solid evidence that it really happened. They're taking credit for people complaining about the spam on HN: https://www.bottlenecklabs.com/blog/benchmarking-7-autonomous-businesses#grok-spam https://www.bottlenecklabs.com/blog/benchmarking-7-autonomou... They also shared the poorly anonymized messages it received from target "customers" who complained about the unsolicited invoices. On example of this poor anonymization is removing the sender's username but leaving the domain name, when the domain name is, for example, a personal domain for a single person.