5 ms·

UK unveils extremism blocking tool

45 points by dberhane 9y ago

slivym 9y ago
What's really worrying about this to me, is how deep this goes. Hacker News is a relatiely well informed audience about this topic. We understand: * that the false positive rate is going to result in more legal material being taken down than ISIS propaganda by an order of magnitude even if what they claim is true. * That the success rate on the data set you've been training on is highly likely to completely mis-represent your true success rate. * That the processing power required for this is likely prohibitive. Not capable of being run real-time, and any actual implementation will have much worse results due to performance optimization. That's laying aside the deep and obvious problems with the government forcing private companies to censor legal speech with no over-sight or even a human in the loop. Let alone law enforcement in the loop. What this press release is, is a calculated attack on free speech. Deliberately misleading the public about the capabilities of technology to attack the technology companies they claim to want to work with. To apply public pressure to private companies to do police enforcement jobs. The only response to this is to state the obvious: If the government wants something censored they can apply to a court injunction as is due process, and in the mean time, let's get rid of this abhorrent stream of Home Secretaries.
- npr11 9y ago
  * False positive rate probably depends on the cut-off for when something is flagged up for a humnan to review, but yeah, for all we know could be high * No one quotes metrics based on the training set, so I'm sure these guys haven't done that. * Most trained ML models don't require much computing at all to make predictions. * The idea seems to be to flag content for review by a human, so there is a human in the loop * No one is forcing anyone to use this... so there's no straight forward censorship issue * maybe this does embarass bigger companies into explaining why they don't develop there own system to do this (removing IS content seems uncontroversial) - I presume this is why the Home Sec is really interested
  - Sionyn 9y ago
    Not so. They specifically stated they'd force this out technology companies if they didn't volunteer. Typical right-wing fascism.
huffmsa 9y ago
Blocking porn => blocking "Xtreme" Content => Oceania has always been at war with Eastasia, never at war with Eurasia
finknotal 9y ago
Does anyone believe this will work as well as they imply it will?
- madaxe_again 9y ago
  Not for a moment. They give no numbers on false positives, and I imagine it’s astronomical. If anything, I can see human moderators having to do more work to restore falsely flagged content than they currently have to do in removing content - but humans are lazy, so instead we’ll just see content going in the “extremism” bin, along with a mandatory report to the government, and people being taken to court for spreading extremist content, which is actually cat videos. It’s a Kafkaesque nightmare in the making.
- EnderMB 9y ago
  The key point for me is that Amber Rudd wouldn't rule out forcing tech companies to use it by law. If the UK government are going to throw their weight behind a tool, they'd better hope that it's of sufficient quality to not be torn to shreds by some of the most technically competent people in the world. The UK government has always had a hostile approach to technology. Now they want to give a tech company the chance to be hostile back, and if this tool doesn't work I can see a very public response to the legitimacy of this tool.
  - pjc50 9y ago
    Whether it works or not has nothing to do with whether people will be forced to use it. Tech companies who fight it will get "ENEMY OF THE PEOPLE" headlines against them in the Mail.
    - EnderMB 9y ago
      On that subject, I'm surprised that we haven't seen a clash between journalists and the "tech elite" yet. Sure, there's a lot of crap written about Google, Facebook, Twitter, and the like on printed media, but given how reliant UK broadcasters are on social media I'm surprised a negative reaction hasn't led to Twitter banning journalists, or Google de-listing a publication for hate-speech.
rlpb 9y ago
"ASI Data Science said the software is capable of detecting 94% of IS's online activity, with an accuracy of 99.995%." I'm not sure what this means. It may be a 0.005% false positive rate. If they're scanning, say, Youtube videos, then the resulting false positive number would be huge.
- m0nty 9y ago
  I would think it unlikely that they have a comprehensive corpus of "offensive content", meaning they can't train/test their system on all of it. So really they're claiming "99.995% accuracy against what we already have". The real world experience will surely be different. You're right that the number is essentially meaningless.
mosselman 9y ago
Censorship is never something to get excited about. It is a slippery slope towards the event in Spain last year when the government misused censorship allowances to silence political opposition[1]. [1]: https://www.eff.org/deeplinks/2017/10/no-justification-spanish-internet-censorship-during-catalonian-referendum https://www.eff.org/deeplinks/2017/10/no-justification-spani...
- userbinator 9y ago
  If you don't want things like this to continue to happen, refuse to work for companies who develop such technology --- and advocate for others to do the same. "Stop making the nooses on which to hang yourselves," as the saying goes.
- anvandare 9y ago
  "As the Americans learned so painfully in Earth's final century, free flow of information is the only safeguard against tyranny. The once-chained people whose leaders at last lose their grip on information flow will soon burst with freedom and vitality, but the free nation gradually constricting its grip on public discourse has begun its rapid slide into despotism. Beware of he who would deny you access to information, for in his heart he deems himself your master." — Sid Meier's Alpha Centauri
  - baud147258 9y ago
    I've always found that SMAC is an example of an horror game, with just the tech descriptions.
    - pvdebbe 9y ago
      SMAC is just wonderful for all lovers of technoporn
  - mosselman 9y ago
    Very cool quote and this really hits the spot: "Beware of he who would deny you access to information, for in his heart he deems himself your master."
tankenmate 9y ago
And what about academics that study these phenomenon? It will encourage research to go underground.
- realusername 9y ago
  And journalists as well... And that's implying the video detection works perfectly 100% of the time (which of course it won't).
SideburnsOfDoom 9y ago
Will it block comments like "If you believe you’re a citizen of the world, you’re a citizen of nowhere." ? http://www.independent.co.uk/news/uk/politics/theresa-may-mein-kampf-adolf-hitler-nazi-vince-cable-liberal-democrat-conservatives-a7825381.html http://www.independent.co.uk/news/uk/politics/theresa-may-me...
toomanybeersies 9y ago
Why does the UK seem to take 1984 as a guide book, rather than a cautionary tale? As much as I applaud efforts to stop extremism, censorship in this form is concerning. Who is the arbiter of what counts as extremism? Obviously any website urging people to join Daesh should be blocked by their standard, but what about websites promoting the PLO, or the PKK? How about websites about the Rohingya? The Burmese government certainly seems to think they are terrorists.
- pjc50 9y ago
  > Why does the UK seem to take 1984 as a guide book, rather than a cautionary tale? Answering a rhetorical question here, but the UK is an unreconstructed colonial government left over from the Empire. In some ways it hasn't really adapted to the independence of Ireland, despite that being nearly 100 years ago. Instead it runs a system designed for control of "distant savages" from London on an absurdly reduced scale, like someone trying to use Kubernetes for their "hello world". Three of the four home nations have colonial office structures. There's a reason there's no Secretary of State for England. Even within England, places like Stoke-on-Trent and Great Yarmouth get treated as if they were in remote inaccessible jungle because you can't get there on an Oyster card. Brexit is where we find out which of the bits of the British constitution are load-bearing. (The PLO are not a proscribed organisation, but Hamas and the PKK are: https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/670599/20171222_Proscription.pdf https://www.gov.uk/government/uploads/system/uploads/attachm... )
- erikrothoff 9y ago
  Isn't that the difference between conservative and liberal politics? With the conservative Tories in power this is right up their alley. Disclaimer: Am Swedish so have absolutely no idea
  - white-flame 9y ago
    There's no difference. Look to Facebook and Google in the types of speech that they're working to suppress and how they want to go about it: It's the same thing from the liberal side. Neither "side"'s stereotype actually wants free speech. That's turning into more of a centrist thing, simply by attrition via tribalism of the "two sides" (which itself is a false dichotomy).
crdoconnor 9y ago
If the government were actually serious about cracking down on extremism, they wouldn't block the content, they'd intercept it and subtly change it to make it ridiculous and humiliating for ISIS. This really wouldn't be that hard. Turning the notion of ISIS recruits into a joke (which, sadly, they actually kind of are) would potentially stem their flow. Censorship, however, will be interpreted by potential recruits as "we are afraid of the truth", will harden their resolve and they'll figure out a way to get around it anyway. Amber Rudd and Theresa May are simply trying to create the infrastructure for a police state, though, using ISIS as a pretext. They're authoritarians at heart and always will be.
jlebrech 9y ago
as long as they acknowledge a communist can also be extremist.
- klez 9y ago
  But that's exactly the point, isn't it? To me, people trying to censor media are extremists, because of how I view the world. In the same way, they may flag me as an extremist because of this. In the end, the problem with censorship is that it gets defined by someone who decides what is extremist and what's not.
- pjc50 9y ago
  Nobody's worried about commies any more, not since the 90s.
moomin 9y ago
If it doesn’t spot The Telegraph, I think it needs better training data.
- seanwasere 9y ago
  And the bbc. Everything I know about Isis, I got from the bbc. Even What it’s flag looks like. imagine if you could use the bbc world broadcast to tell the whole world, what Isis is, and what Isis does. Every day, and behind the ‘breaking news’ banner and panick sound effect
mavdi 9y ago
We all ultimately follow the Chinese when it comes to tech. Just after much drama and a couple terror attacks.
- deleted 9y ago
  [deleted]
white-flame 9y ago
As always this whole thing is pointless and stupid. As things get blocked, the posters of this material will tweak it until it passes the AI-driven censors. It'd all be a laughable trainwreck... if it didn't set further precedent of censorship and potentially destroy people who fall into false positives.
anonytrary 9y ago
What a sad state. It starts with "protecting" people from terrorism, but it will quickly devolve into humanity-stunting censorship. Instead, educate people about terrorism. We cannot defer education in the hopes of protecting people. This never, ever ends the way it is intended. One day I fear I will open my eyes to find parts of the world blurry, because of state-mandated image filtering device, embedded in my eye.
sequence7 9y ago
So it looks like one of two things has happened: 1) The British government have built AI tech capable of identifying extremist views in any video at the incredibly low price of £600000, doing what no other technology company in the world is capable of. 2) A smart consulting company has convinced the British government to waste £600000 on building a model that can flag certain videos from a very specific training set but in real world use is horribly innacurate and completely useless.
- raverbashing 9y ago
  The most surprising part of it is that it cost "only" 600k I'm sure some other companies could have done it for 100x the cost using "proven technology" (that doesn't work for the problem at hand but sounds nice on paper)
- BenedictEggs 9y ago
  The politician pushing this is the Home Secretary Amber Rudd, who is notorious even among UK politicians for her shaky grasp of technical matters. Anyone unfamiliar with her previous work is advised to Google "Amber Rudd encryption". My guess: someone enterprising has seen her on TV and spotted a business opportunity. We should be grateful that they only took us for £600k.
- andy_ppp 9y ago
  Alternatively, this technology will ban loads of things that aren't extremist material. To be honest I actually think if you just look at metadata (IPs, locations, services, usernames, previous uploads, titles, etc.) you can probably get the percentages they are talking about, at least until people start marking up the extremist material in different ways. The article even says they don't want to say how the technology works which implies it'll be trivial to get around.
seanwasere 9y ago
‘The Fear Media Governer’
pjc50 9y ago
Hands up those who remember the previous incarnation of this, the broadcast ban on Sinn Fein: https://www.youtube.com/watch?v=Xr8bsOgmGhI https://www.youtube.com/watch?v=Xr8bsOgmGhI / http://news.bbc.co.uk/1/hi/4409447.stm http://news.bbc.co.uk/1/hi/4409447.stm

45 points by dberhane 9y ago

What's really worrying about this to me, is how deep this goes. Hacker News is a relatiely well informed audience about this topic. We understand: * that the false positive rate is going to result in more legal material being taken down than ISIS propaganda by an order of magnitude even if what they claim is true. * That the success rate on the data set you've been training on is highly likely to completely mis-represent your true success rate. * That the processing power required for this is likely prohibitive. Not capable of being run real-time, and any actual implementation will have much worse results due to performance optimization. That's laying aside the deep and obvious problems with the government forcing private companies to censor legal speech with no over-sight or even a human in the loop. Let alone law enforcement in the loop. What this press release is, is a calculated attack on free speech. Deliberately misleading the public about the capabilities of technology to attack the technology companies they claim to want to work with. To apply public pressure to private companies to do police enforcement jobs. The only response to this is to state the obvious: If the government wants something censored they can apply to a court injunction as is due process, and in the mean time, let's get rid of this abhorrent stream of Home Secretaries.

npr11 9y ago
* False positive rate probably depends on the cut-off for when something is flagged up for a humnan to review, but yeah, for all we know could be high * No one quotes metrics based on the training set, so I'm sure these guys haven't done that. * Most trained ML models don't require much computing at all to make predictions. * The idea seems to be to flag content for review by a human, so there is a human in the loop * No one is forcing anyone to use this... so there's no straight forward censorship issue * maybe this does embarass bigger companies into explaining why they don't develop there own system to do this (removing IS content seems uncontroversial) - I presume this is why the Home Sec is really interested
- Sionyn 9y ago
  Not so. They specifically stated they'd force this out technology companies if they didn't volunteer. Typical right-wing fascism.

huffmsa 9y ago

Blocking porn => blocking "Xtreme" Content => Oceania has always been at war with Eastasia, never at war with Eurasia

finknotal 9y ago

Does anyone believe this will work as well as they imply it will?

madaxe_again 9y ago
Not for a moment. They give no numbers on false positives, and I imagine it’s astronomical. If anything, I can see human moderators having to do more work to restore falsely flagged content than they currently have to do in removing content - but humans are lazy, so instead we’ll just see content going in the “extremism” bin, along with a mandatory report to the government, and people being taken to court for spreading extremist content, which is actually cat videos. It’s a Kafkaesque nightmare in the making.
EnderMB 9y ago
The key point for me is that Amber Rudd wouldn't rule out forcing tech companies to use it by law. If the UK government are going to throw their weight behind a tool, they'd better hope that it's of sufficient quality to not be torn to shreds by some of the most technically competent people in the world. The UK government has always had a hostile approach to technology. Now they want to give a tech company the chance to be hostile back, and if this tool doesn't work I can see a very public response to the legitimacy of this tool.
- pjc50 9y ago
  Whether it works or not has nothing to do with whether people will be forced to use it. Tech companies who fight it will get "ENEMY OF THE PEOPLE" headlines against them in the Mail.
  - EnderMB 9y ago
    On that subject, I'm surprised that we haven't seen a clash between journalists and the "tech elite" yet. Sure, there's a lot of crap written about Google, Facebook, Twitter, and the like on printed media, but given how reliant UK broadcasters are on social media I'm surprised a negative reaction hasn't led to Twitter banning journalists, or Google de-listing a publication for hate-speech.

rlpb 9y ago

"ASI Data Science said the software is capable of detecting 94% of IS's online activity, with an accuracy of 99.995%." I'm not sure what this means. It may be a 0.005% false positive rate. If they're scanning, say, Youtube videos, then the resulting false positive number would be huge.

m0nty 9y ago
I would think it unlikely that they have a comprehensive corpus of "offensive content", meaning they can't train/test their system on all of it. So really they're claiming "99.995% accuracy against what we already have". The real world experience will surely be different. You're right that the number is essentially meaningless.

mosselman 9y ago

Censorship is never something to get excited about. It is a slippery slope towards the event in Spain last year when the government misused censorship allowances to silence political opposition[1]. [1]: https://www.eff.org/deeplinks/2017/10/no-justification-spanish-internet-censorship-during-catalonian-referendum https://www.eff.org/deeplinks/2017/10/no-justification-spani...

userbinator 9y ago
If you don't want things like this to continue to happen, refuse to work for companies who develop such technology --- and advocate for others to do the same. "Stop making the nooses on which to hang yourselves," as the saying goes.
anvandare 9y ago
"As the Americans learned so painfully in Earth's final century, free flow of information is the only safeguard against tyranny. The once-chained people whose leaders at last lose their grip on information flow will soon burst with freedom and vitality, but the free nation gradually constricting its grip on public discourse has begun its rapid slide into despotism. Beware of he who would deny you access to information, for in his heart he deems himself your master." — Sid Meier's Alpha Centauri
- baud147258 9y ago
  I've always found that SMAC is an example of an horror game, with just the tech descriptions.
  - pvdebbe 9y ago
    SMAC is just wonderful for all lovers of technoporn
- mosselman 9y ago
  Very cool quote and this really hits the spot: "Beware of he who would deny you access to information, for in his heart he deems himself your master."

tankenmate 9y ago

And what about academics that study these phenomenon? It will encourage research to go underground.

realusername 9y ago
And journalists as well... And that's implying the video detection works perfectly 100% of the time (which of course it won't).

SideburnsOfDoom 9y ago

Will it block comments like "If you believe you’re a citizen of the world, you’re a citizen of nowhere." ? http://www.independent.co.uk/news/uk/politics/theresa-may-mein-kampf-adolf-hitler-nazi-vince-cable-liberal-democrat-conservatives-a7825381.html http://www.independent.co.uk/news/uk/politics/theresa-may-me...

toomanybeersies 9y ago

Why does the UK seem to take 1984 as a guide book, rather than a cautionary tale? As much as I applaud efforts to stop extremism, censorship in this form is concerning. Who is the arbiter of what counts as extremism? Obviously any website urging people to join Daesh should be blocked by their standard, but what about websites promoting the PLO, or the PKK? How about websites about the Rohingya? The Burmese government certainly seems to think they are terrorists.

pjc50 9y ago
> Why does the UK seem to take 1984 as a guide book, rather than a cautionary tale? Answering a rhetorical question here, but the UK is an unreconstructed colonial government left over from the Empire. In some ways it hasn't really adapted to the independence of Ireland, despite that being nearly 100 years ago. Instead it runs a system designed for control of "distant savages" from London on an absurdly reduced scale, like someone trying to use Kubernetes for their "hello world". Three of the four home nations have colonial office structures. There's a reason there's no Secretary of State for England. Even within England, places like Stoke-on-Trent and Great Yarmouth get treated as if they were in remote inaccessible jungle because you can't get there on an Oyster card. Brexit is where we find out which of the bits of the British constitution are load-bearing. (The PLO are not a proscribed organisation, but Hamas and the PKK are: https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/670599/20171222_Proscription.pdf https://www.gov.uk/government/uploads/system/uploads/attachm... )
erikrothoff 9y ago
Isn't that the difference between conservative and liberal politics? With the conservative Tories in power this is right up their alley. Disclaimer: Am Swedish so have absolutely no idea
- white-flame 9y ago
  There's no difference. Look to Facebook and Google in the types of speech that they're working to suppress and how they want to go about it: It's the same thing from the liberal side. Neither "side"'s stereotype actually wants free speech. That's turning into more of a centrist thing, simply by attrition via tribalism of the "two sides" (which itself is a false dichotomy).

crdoconnor 9y ago

If the government were actually serious about cracking down on extremism, they wouldn't block the content, they'd intercept it and subtly change it to make it ridiculous and humiliating for ISIS. This really wouldn't be that hard. Turning the notion of ISIS recruits into a joke (which, sadly, they actually kind of are) would potentially stem their flow. Censorship, however, will be interpreted by potential recruits as "we are afraid of the truth", will harden their resolve and they'll figure out a way to get around it anyway. Amber Rudd and Theresa May are simply trying to create the infrastructure for a police state, though, using ISIS as a pretext. They're authoritarians at heart and always will be.

jlebrech 9y ago

as long as they acknowledge a communist can also be extremist.

klez 9y ago
But that's exactly the point, isn't it? To me, people trying to censor media are extremists, because of how I view the world. In the same way, they may flag me as an extremist because of this. In the end, the problem with censorship is that it gets defined by someone who decides what is extremist and what's not.
pjc50 9y ago
Nobody's worried about commies any more, not since the 90s.

moomin 9y ago

If it doesn’t spot The Telegraph, I think it needs better training data.

seanwasere 9y ago
And the bbc. Everything I know about Isis, I got from the bbc. Even What it’s flag looks like. imagine if you could use the bbc world broadcast to tell the whole world, what Isis is, and what Isis does. Every day, and behind the ‘breaking news’ banner and panick sound effect

mavdi 9y ago

We all ultimately follow the Chinese when it comes to tech. Just after much drama and a couple terror attacks.

deleted 9y ago
[deleted]

white-flame 9y ago

As always this whole thing is pointless and stupid. As things get blocked, the posters of this material will tweak it until it passes the AI-driven censors. It'd all be a laughable trainwreck... if it didn't set further precedent of censorship and potentially destroy people who fall into false positives.

anonytrary 9y ago

What a sad state. It starts with "protecting" people from terrorism, but it will quickly devolve into humanity-stunting censorship. Instead, educate people about terrorism. We cannot defer education in the hopes of protecting people. This never, ever ends the way it is intended. One day I fear I will open my eyes to find parts of the world blurry, because of state-mandated image filtering device, embedded in my eye.

sequence7 9y ago

So it looks like one of two things has happened: 1) The British government have built AI tech capable of identifying extremist views in any video at the incredibly low price of £600000, doing what no other technology company in the world is capable of. 2) A smart consulting company has convinced the British government to waste £600000 on building a model that can flag certain videos from a very specific training set but in real world use is horribly innacurate and completely useless.

raverbashing 9y ago
The most surprising part of it is that it cost "only" 600k I'm sure some other companies could have done it for 100x the cost using "proven technology" (that doesn't work for the problem at hand but sounds nice on paper)
BenedictEggs 9y ago
The politician pushing this is the Home Secretary Amber Rudd, who is notorious even among UK politicians for her shaky grasp of technical matters. Anyone unfamiliar with her previous work is advised to Google "Amber Rudd encryption". My guess: someone enterprising has seen her on TV and spotted a business opportunity. We should be grateful that they only took us for £600k.
andy_ppp 9y ago
Alternatively, this technology will ban loads of things that aren't extremist material. To be honest I actually think if you just look at metadata (IPs, locations, services, usernames, previous uploads, titles, etc.) you can probably get the percentages they are talking about, at least until people start marking up the extremist material in different ways. The article even says they don't want to say how the technology works which implies it'll be trivial to get around.

seanwasere 9y ago

‘The Fear Media Governer’

pjc50 9y ago

Hands up those who remember the previous incarnation of this, the broadcast ban on Sinn Fein: https://www.youtube.com/watch?v=Xr8bsOgmGhI https://www.youtube.com/watch?v=Xr8bsOgmGhI / http://news.bbc.co.uk/1/hi/4409447.stm http://news.bbc.co.uk/1/hi/4409447.stm