6 ms·

What this incident really shows is the growing gap between how easy it is to create a convincing warning and how costly it is to verify what's actually happenin

What this incident really shows is the growing gap between how easy it is to create a convincing warning and how costly it is to verify what's actually happening. Hoaxes aren't new, but generative tools make fabrication almost free and massively increase the volume. The rail operator didn't do anything wrong. After an earthquake and a realistic-looking image, the only responsible action is to treat it as potentially real and inspect the track. This wasn't catastrophic, but it's a preview of a world where a single person can cheaply trigger high-cost responses. The systems we build will have to adapt, not by ignoring social media reports, but by developing faster, more resilient ways to distinguish signal from noise.

bncndn0956 10mo ago
Sir, this is AI prose. Wendy's doesn't allow AI prose.
- mr_windfrog 10mo ago
  Thanks for the heads-up! I actually wrote this based on my own thoughts about the incident, but I understand the concern. I'll make sure to keep my posts in line with the community guidelines.
foxglacier 10mo ago
You don't need AI for this kind of disruption. People have been making fake bomb threats for years. You just have to say it, either directly to the railway/etc. or publicly enough that somebody else will believe it and forward it to them. The difference might be of intent - if you say you planted a bomb on the bridge, you're probably committing a crime, but if you just post a piece of art without context, it's more plausibly deniable. It's also pretty common in the UK for trains to be delayed just because some passenger accidentally left their bag on the platform. Not even any malicious intent. I was on a train that stopped in a tunnel for that reason once. They're just very vulnerable to any hint of danger.
- array_key_first 10mo ago
  AI definitely makes it easier and it will happen more often. You don't need anything for anything. You can do war with long sticks. Turns out guns, planes, and firebombs work better.
  - j_maffe 10mo ago
    Exactly. More is different.
  - foxglacier 10mo ago
    AI image generation is already freely available to everyone. Why is it not already happening very often? This is the first case I've even heard of. Seems like you're already proven wrong, unless you're counting on some future change that isn't here yet?
    - array_key_first 10mo ago
      > Why is it not already happening very often? It is, and it will increase as it becomes more accessible. > This is the first case I've even heard of. That's because you're both not paying much attention and this is under-reported. I would wager maybe a quarter of all content on the internet is bot generated. I'm not the first to propose this. > Seems like you're already proven wrong, unless you're counting on some future change that isn't here yet? I kind of am, notably AI both becoming better and more accessible. You're right, it might not.
hurturue 10mo ago
an AI sees the image on social media, deploys a drone to quickly go there, looks at the live video feed, and declares all is good
intended 10mo ago
Not a hope. Most economic value arises from distinguishing signal from noise. All of science is distinguishing signal from noise. Its valuable, because it is hard. It is also slow - the only way to verify something is often to have reports from someone who IS there. The conflict arises not from verifying the easy things - searching under the illumination of street lights. Its verifying if you have a weird disease, or if people are alive in a disaster, or what is actually going on in a distant zone. Verification is laborious. In essence, the universe is not going to open up its secrets to us, unless the effort is put in. Content generation on the other hand, is story telling. It serves other utility functions to consumers - fulfilling emotional needs for example. As the ratio of content to information keeps growing, or the ratio of content to verification capacity grows - we will grow increasingly overwhelmed by the situation.
mytailorisrich 10mo ago
It is cheap to have live monitoring of key infrastructure these days, and in the case of rail infrastructure it would also save time and money in general. Perhaps this hoax will push this higher up the todo list.
- usr1106 10mo ago
  It may be cheap to monitor a single spot. It is extremely expensive to monitor everything.
  - mytailorisrich 10mo ago
    There is a balance like always. It seems odd that they have zero cameras on bridges and other main infrastructure, although I believe that level crossings tend to have them (perhaps more to avoid liability in case of accidents, though...) Main point is that there aren't technical difficulties in verifying the state of main infrastructure in real time (contrary to the claim of the commenter I was initially replying to), and it's more a question of priority and will than doability or cost. It will happen but the usual way is that "it's not possible", "it's too expensive", etc until something bad enough happens and then suddenly it is doable and done.
belorn 10mo ago
Given the number of cctv cameras that operate in the UK, and their continued growth, I am surprised that the rail operator did not have access to a direct view of the bridge. I am also a bit surprised that there isn't technology to detect rail damage, especially the power lines that runs over the track. Where I live it is not uncommon for rail to have detection for people walking on the rail, and bridges to have extra protection against jumpers. I wouldn't be that surprised if the same system can be used to verify damage.
- _jzlw 10mo ago
  This is the part that I find insane. What if the bridge had collapsed, and no one had bothered to post a picture of it to social media?
  - pmyteh 10mo ago
    I mean, you're supposed to call the police or Network Rail: there are placards on the (remains of the) bridge with the telephone number. But yes, it's not uncommon to have to send a train to examine the line (at slow speed, able to stop within line-of-sight) after extreme weather.
- pmyteh 10mo ago
  There is technology that could detect rail breaks, in the form of track circuits: feed a current into a rail, detect whether it gets to the other end (or bridge the two rails at the other end of the circuit and see if it gets back to the start of the other rail). A variation of this is commonly used in signalling systems to verify that the track is clear: if a pair of wheels is in the track section then the signal will short across the rails and make the circuit show 'occupied'. Ultimately, though, this kind of stuff is expensive (semi-bespoke safety-critical equipment every few miles across an enormous network) and doesn't reduce all risks. Landslides don't necessarily break rails (but can cause derailments), embankments and bridges can get washed out but the track remains hanging, and lots of other failure modes. There are definitely also systems to confirm that the power lines aren't down, but unfortunately the wires can stay up and the track be damaged or vice versa, so proving one doesn't prove the other. CCTV is probably a better bet, but that's still a truly enormous number of cameras, plus running power supplies all along the railway and ensuring a data link, plus monitoring.
tenthirtyam 10mo ago
I also think not a hope, check Brandolini's law[1]: The amount of energy needed to refute bullshit is an order of magnitude bigger than that needed to produce it. [1] https://en.wikipedia.org/wiki/Brandolini%27s_law https://en.wikipedia.org/wiki/Brandolini%27s_law
- ImHereToVote 10mo ago
  There should be a countervailing law that the more bullshit is produced the more skeptical the populace becomes. The amount of conspiracy theorists has remained constant even with the advent of the Internet this hasn't changed.
soerxpso 10mo ago
Would calling and saying, "Hey, the bridge is destroyed!" without an image not have also triggered a delay? I question the safety standards of the railway if they would just ignore such a call after an earthquake. Generative AI doesn't change the situation at all. An image shouldn't be treated as carrying more weight than a statement, but the statement without the image would be the same in this situation. This has really been an issue since the popularization of the telephone, which made it sufficiently easy to communicate a lie from far away that someone might choose to do so for fun.
- pixl97 10mo ago
  This in itself is not a big deal... but there very much scenarios that could mean life or death. Take a fast moving wildfire with one of the paths of escape blocked. There may be other lines of escape but fake images of one of those open roads showing its blocked by fire could lead to traffic jams and eventual danger on the remaining routes of escape.
- tremon 10mo ago
  Calling identifies one person by name/number, and makes that person liable for any damages from the hoax, similar to how calling in a fake bomb threat is a crime. Publicly posting a fake comment and waiting for the rail operator to react of their own volition removes liability from that individual. That's where the AI footage comes in: it makes it more likely for the hoax to be taken serious.
oars 10mo ago
Great comment and very true in this AI world. In 2030 it will be even easier to make even more realistic images much quicker... Reminds me of the attacker vs defender dilemma in cybersecurity - attackers just need one attack to succeed while a defender must spend resources considering and defending against all the different possibilities.

bncndn0956 10mo ago

Sir, this is AI prose. Wendy's doesn't allow AI prose.

mr_windfrog 10mo ago
Thanks for the heads-up! I actually wrote this based on my own thoughts about the incident, but I understand the concern. I'll make sure to keep my posts in line with the community guidelines.

foxglacier 10mo ago

You don't need AI for this kind of disruption. People have been making fake bomb threats for years. You just have to say it, either directly to the railway/etc. or publicly enough that somebody else will believe it and forward it to them. The difference might be of intent - if you say you planted a bomb on the bridge, you're probably committing a crime, but if you just post a piece of art without context, it's more plausibly deniable. It's also pretty common in the UK for trains to be delayed just because some passenger accidentally left their bag on the platform. Not even any malicious intent. I was on a train that stopped in a tunnel for that reason once. They're just very vulnerable to any hint of danger.

array_key_first 10mo ago
AI definitely makes it easier and it will happen more often. You don't need anything for anything. You can do war with long sticks. Turns out guns, planes, and firebombs work better.
- j_maffe 10mo ago
  Exactly. More is different.
- foxglacier 10mo ago
  AI image generation is already freely available to everyone. Why is it not already happening very often? This is the first case I've even heard of. Seems like you're already proven wrong, unless you're counting on some future change that isn't here yet?
  - array_key_first 10mo ago
    > Why is it not already happening very often? It is, and it will increase as it becomes more accessible. > This is the first case I've even heard of. That's because you're both not paying much attention and this is under-reported. I would wager maybe a quarter of all content on the internet is bot generated. I'm not the first to propose this. > Seems like you're already proven wrong, unless you're counting on some future change that isn't here yet? I kind of am, notably AI both becoming better and more accessible. You're right, it might not.

hurturue 10mo ago

an AI sees the image on social media, deploys a drone to quickly go there, looks at the live video feed, and declares all is good

intended 10mo ago

Not a hope. Most economic value arises from distinguishing signal from noise. All of science is distinguishing signal from noise. Its valuable, because it is hard. It is also slow - the only way to verify something is often to have reports from someone who IS there. The conflict arises not from verifying the easy things - searching under the illumination of street lights. Its verifying if you have a weird disease, or if people are alive in a disaster, or what is actually going on in a distant zone. Verification is laborious. In essence, the universe is not going to open up its secrets to us, unless the effort is put in. Content generation on the other hand, is story telling. It serves other utility functions to consumers - fulfilling emotional needs for example. As the ratio of content to information keeps growing, or the ratio of content to verification capacity grows - we will grow increasingly overwhelmed by the situation.

mytailorisrich 10mo ago

It is cheap to have live monitoring of key infrastructure these days, and in the case of rail infrastructure it would also save time and money in general. Perhaps this hoax will push this higher up the todo list.

usr1106 10mo ago
It may be cheap to monitor a single spot. It is extremely expensive to monitor everything.
- mytailorisrich 10mo ago
  There is a balance like always. It seems odd that they have zero cameras on bridges and other main infrastructure, although I believe that level crossings tend to have them (perhaps more to avoid liability in case of accidents, though...) Main point is that there aren't technical difficulties in verifying the state of main infrastructure in real time (contrary to the claim of the commenter I was initially replying to), and it's more a question of priority and will than doability or cost. It will happen but the usual way is that "it's not possible", "it's too expensive", etc until something bad enough happens and then suddenly it is doable and done.

belorn 10mo ago

Given the number of cctv cameras that operate in the UK, and their continued growth, I am surprised that the rail operator did not have access to a direct view of the bridge. I am also a bit surprised that there isn't technology to detect rail damage, especially the power lines that runs over the track. Where I live it is not uncommon for rail to have detection for people walking on the rail, and bridges to have extra protection against jumpers. I wouldn't be that surprised if the same system can be used to verify damage.

_jzlw 10mo ago
This is the part that I find insane. What if the bridge had collapsed, and no one had bothered to post a picture of it to social media?
- pmyteh 10mo ago
  I mean, you're supposed to call the police or Network Rail: there are placards on the (remains of the) bridge with the telephone number. But yes, it's not uncommon to have to send a train to examine the line (at slow speed, able to stop within line-of-sight) after extreme weather.
pmyteh 10mo ago
There is technology that could detect rail breaks, in the form of track circuits: feed a current into a rail, detect whether it gets to the other end (or bridge the two rails at the other end of the circuit and see if it gets back to the start of the other rail). A variation of this is commonly used in signalling systems to verify that the track is clear: if a pair of wheels is in the track section then the signal will short across the rails and make the circuit show 'occupied'. Ultimately, though, this kind of stuff is expensive (semi-bespoke safety-critical equipment every few miles across an enormous network) and doesn't reduce all risks. Landslides don't necessarily break rails (but can cause derailments), embankments and bridges can get washed out but the track remains hanging, and lots of other failure modes. There are definitely also systems to confirm that the power lines aren't down, but unfortunately the wires can stay up and the track be damaged or vice versa, so proving one doesn't prove the other. CCTV is probably a better bet, but that's still a truly enormous number of cameras, plus running power supplies all along the railway and ensuring a data link, plus monitoring.

tenthirtyam 10mo ago

I also think not a hope, check Brandolini's law[1]: The amount of energy needed to refute bullshit is an order of magnitude bigger than that needed to produce it. [1] https://en.wikipedia.org/wiki/Brandolini%27s_law https://en.wikipedia.org/wiki/Brandolini%27s_law

ImHereToVote 10mo ago
There should be a countervailing law that the more bullshit is produced the more skeptical the populace becomes. The amount of conspiracy theorists has remained constant even with the advent of the Internet this hasn't changed.

soerxpso 10mo ago

Would calling and saying, "Hey, the bridge is destroyed!" without an image not have also triggered a delay? I question the safety standards of the railway if they would just ignore such a call after an earthquake. Generative AI doesn't change the situation at all. An image shouldn't be treated as carrying more weight than a statement, but the statement without the image would be the same in this situation. This has really been an issue since the popularization of the telephone, which made it sufficiently easy to communicate a lie from far away that someone might choose to do so for fun.

pixl97 10mo ago
This in itself is not a big deal... but there very much scenarios that could mean life or death. Take a fast moving wildfire with one of the paths of escape blocked. There may be other lines of escape but fake images of one of those open roads showing its blocked by fire could lead to traffic jams and eventual danger on the remaining routes of escape.
tremon 10mo ago
Calling identifies one person by name/number, and makes that person liable for any damages from the hoax, similar to how calling in a fake bomb threat is a crime. Publicly posting a fake comment and waiting for the rail operator to react of their own volition removes liability from that individual. That's where the AI footage comes in: it makes it more likely for the hoax to be taken serious.

oars 10mo ago

Great comment and very true in this AI world. In 2030 it will be even easier to make even more realistic images much quicker... Reminds me of the attacker vs defender dilemma in cybersecurity - attackers just need one attack to succeed while a defender must spend resources considering and defending against all the different possibilities.