7 ms·

Open Rights Group has launched a UK Censorship Monitoring API and Botnet

82 points by namos 12y ago

namos 12y ago
For the past 8 months we've been working on building a series of probes; - Android https://play.google.com/store/apps/details?id=uk.org.blocked.app https://play.google.com/store/apps/details?id=uk.org.blocked... - Python https://github.com/openrightsgroup/OrgProbe https://github.com/openrightsgroup/OrgProbe - Raspberry PI images https://github.com/openrightsgroup/lepidopter https://github.com/openrightsgroup/lepidopter) And an open API to power them: https://wiki.openrightsgroup.org/wiki/Censorship_Monitoring_Project_API https://wiki.openrightsgroup.org/wiki/Censorship_Monitoring_... This is helping to build a picture of what the UK Governments Internet Blocking/Filtering/Censorship actually looks like. You can submit URLs to be tested by visiting https://www.Blocked.org.uk https://www.Blocked.org.uk by installing the Android app or by tweeting a URL with the hashtag #IsBlocked Once we receive a URL it is dispatched to all probes over a period of time where it is checked against a know list of blocking methods. The results are returned back to the API for public evaluation. For example: https://www.blocked.org.uk/results?url=https%3A%2F%2Fnews.ycombinator.com https://www.blocked.org.uk/results?url=https%3A%2F%2Fnews.yc... Please help us build a picture of UK Internet Censorship by adding URLs to be tested, by helping develop new probes (browser extensions, an iPhone client etc etc) or just by spreading the word that https://Blocked.org.uk https://Blocked.org.uk is available. Please also see https://www.blocked.org.uk/faq https://www.blocked.org.uk/faq which will explain the level of filtering enabled on some of the probes which will explain why some sites may appear blocked even if you, being on the same ISP, can reach said URL.
- vidarh 12y ago
  It would seem like some classification of the blocked URLs plus a suitable extension to use willing proxies would be quite useful as an exercise in making the blocklist completely pointless. I'd imagine a lot of people that have issues with e.g. running a Tor node would be perfectly happy to offer up proxies offering up various subsets of the blocked sites whitelisted by category.
- mike-cardwell 12y ago
  Please consider freeing the data and make it downloadable as a simple CSV. I'd like a list of all of the sites blocked by particular ISPs...
  - namos 12y ago
    Snapshots are made available: https://www.blocked.org.uk/assets/data/alexa_100k_isp_latest_status_2014_06_29.ods https://www.blocked.org.uk/assets/data/alexa_100k_isp_latest...
- gone35 12y ago
  Whoaa why the hell is The Brights' Net being blocked by BT??!? Since when is atheism considered "censorable" content in the UK?? This is madness. https://www.blocked.org.uk/results?url=http://www.the-brights.net/ https://www.blocked.org.uk/results?url=http://www.the-bright...
- finnn 12y ago
  Have you considered setting a user agent other than Python Requests default? Probably not necessary, but good practice. Also, ironically, the app is not available on any of my devices.
justincormack 12y ago
Ah I knew something was in the works as Andrews & Arnold were writing about obtaining connections from ISPs to test what was blocked. Great to know what is being censored (if you are not on A&A).
- dfkvldfmv 12y ago
  "(if you are not on A&A)." Or if you're on any of the ISPs who do not have filters, including but not limited to A&A.
  - SideburnsOfDoom 12y ago
    Who are the other ISPs who do not have filters?
    - dspillett 12y ago
      There is no handy list that I know of. The big six definitely do have the filters. Some smaller players do too, but a fair few don't. The only people I know for sure are A&A because they put their heads above the parapet and made a stand. Some simply haven't implemented it because it would be work and they therefore won't until forced to (rather than because of any moral objection) - they'll not make any statement about not doing it as it will look bad if they eventually do. There may of course be others in A&A's position who haven't done as good a job getting the message out.
spingsprong 12y ago
I've been checking various websites I go on that are completely fine for kids. It's amazing what's getting blocked. One example according to that website is, TalkTalk blocks "http://www.writingexcuses.com/" http://www.writingexcuses.com/" It's a podcast that teaches story writing. Blocking that is insane! And the blocked website itself is blocked on two ISPs https://www.blocked.org.uk/results?url=http%3A%2F%2Fblocked.org.uk https://www.blocked.org.uk/results?url=http%3A%2F%2Fblocked....
- TazeTSchnitzel 12y ago
  I think TalkTalk is either using a whitelist or an oversensitive keyword filter.
  - namos 12y ago
    The TalkTalk filter is the default "KidSafe" level
  - vidarh 12y ago
    My personal website is blocked by Talk Talk apparently. I guess compiler writing must be harmful for children.
    - keithpeter 12y ago
      Here we go. If a lot of harmless or potentially even useful material gets blocked, people who want to use that material will simply opt out or route round the blocking, hence making it pointless.
      - vidarh 12y ago
        I kind of hope they overreach drastically for exactly that reason. It'll be a lot easier to get the filters dropped entirely if we can attack them with "useful website X is blocked for no good reason" over and over and over again, than if we have to argue using examples that many people would actually have a problem with.
blueskin_ 12y ago
I just learned that one of my personal sites is blocked on BT and TalkTalk... there is zero adult or other inappropriate content on it. Time to complain, I guess, but I'm sure they will just go straight to /dev/null :(
- dspillett 12y ago
  Make sure you update the site to state that it is currently blocked by these ISPs, you don't know why, you have asked them to investigate, but you have not heard nothing back and until they respond it is completely out of your control. That way anyone who can access it from work but not home (for example) or recommends the site to a friend who finds it doesn't work, will have a clue as to what the problem is (and it isn't your fault). If your site is of significant general interest then this may also increase the chance of the ISPs acting quickly.
- AlyssaRowan 12y ago
  Have you considered legal action?
  - blueskin_ 12y ago
    Nope. It's a medium sized site I don't get any money out of, not linked to any business. I suppose I could theoretically small claims court them, but I'm not sure if I'd win as they can probably piss about with their customers' connections in whatever way they please due to their T&Cs :(
lucb1e 12y ago
Wow, I had not expected this. My personal blog (which is about technology) is being blocked by one of the ISPs. I don't host porn nor have I ever, so why is it blocked behind their porn filter? Interesting.... https://www.blocked.org.uk/results?url=http://lucb1e.com https://www.blocked.org.uk/results?url=http://lucb1e.com Edit: TalkTalk live chat support reports that I am not being blocked. Citation: > I can confirm that is not blocked by our Adult Filters as your website does not have any adult content. So it seems blocked.org.uk has a false-positive...
- blueskin_ 12y ago
  TalkTalk are a completely awful ISP. Not sure what the US equivalent would be, but they're the kind who do promotions of "Broadband for £3/month"-level. You get what you pay for service-wise, but they are also by far the most enthusiastic ISP around when it comes to censorship; they have a spider that follows their users to all URLs they visit to categorise the sites[1] (giving them the nickname StalkStalk), and have been doing this long before the current set of idiots in Parliament forced censorship in (at least since 2011). I'm in a similar situation, one site I run is blocked both by StalkStalk and BT. No porn, no other 'adult' content; it's deliberately kept PG-13 to use the US ratings. [1]Check your logs for HuaweiSymantecSpider. It obeys robots.txt though, so a quick "Disallow: /" in robots.txt stops it, although they still grab content from unencrypted pages via DPI.
  - lucb1e 12y ago
    No HuaweiSymantecSpider in my logs in the past 8 months (grepped access.log), but the site is available over both http and https so they might have classified it using DPI (talk about evil...).
- namos 12y ago
  TalkTalk has more than one filter setting. Ask them if your site is blocked at the "Kid Safe" level (the default) which is what the probes run on.
- endersshadow 12y ago
  BT is blocking my site about my homebrewed beer. That's fun.
jamesbrownuhh 12y ago
Sad to see that The Best Page In The Universe (http://maddox.xmission.com http://maddox.xmission.com) seems to be blocked by a non-zero number of larger ISPs. Obviously all those children need to be protected from harsh reviews of their artwork.

82 points by namos 12y ago

namos 12y ago

For the past 8 months we've been working on building a series of probes; - Android https://play.google.com/store/apps/details?id=uk.org.blocked.app https://play.google.com/store/apps/details?id=uk.org.blocked... - Python https://github.com/openrightsgroup/OrgProbe https://github.com/openrightsgroup/OrgProbe - Raspberry PI images https://github.com/openrightsgroup/lepidopter https://github.com/openrightsgroup/lepidopter) And an open API to power them: https://wiki.openrightsgroup.org/wiki/Censorship_Monitoring_Project_API https://wiki.openrightsgroup.org/wiki/Censorship_Monitoring_... This is helping to build a picture of what the UK Governments Internet Blocking/Filtering/Censorship actually looks like. You can submit URLs to be tested by visiting https://www.Blocked.org.uk https://www.Blocked.org.uk by installing the Android app or by tweeting a URL with the hashtag #IsBlocked Once we receive a URL it is dispatched to all probes over a period of time where it is checked against a know list of blocking methods. The results are returned back to the API for public evaluation. For example: https://www.blocked.org.uk/results?url=https%3A%2F%2Fnews.ycombinator.com https://www.blocked.org.uk/results?url=https%3A%2F%2Fnews.yc... Please help us build a picture of UK Internet Censorship by adding URLs to be tested, by helping develop new probes (browser extensions, an iPhone client etc etc) or just by spreading the word that https://Blocked.org.uk https://Blocked.org.uk is available. Please also see https://www.blocked.org.uk/faq https://www.blocked.org.uk/faq which will explain the level of filtering enabled on some of the probes which will explain why some sites may appear blocked even if you, being on the same ISP, can reach said URL.

vidarh 12y ago
It would seem like some classification of the blocked URLs plus a suitable extension to use willing proxies would be quite useful as an exercise in making the blocklist completely pointless. I'd imagine a lot of people that have issues with e.g. running a Tor node would be perfectly happy to offer up proxies offering up various subsets of the blocked sites whitelisted by category.
mike-cardwell 12y ago
Please consider freeing the data and make it downloadable as a simple CSV. I'd like a list of all of the sites blocked by particular ISPs...
- namos 12y ago
  Snapshots are made available: https://www.blocked.org.uk/assets/data/alexa_100k_isp_latest_status_2014_06_29.ods https://www.blocked.org.uk/assets/data/alexa_100k_isp_latest...
gone35 12y ago
Whoaa why the hell is The Brights' Net being blocked by BT??!? Since when is atheism considered "censorable" content in the UK?? This is madness. https://www.blocked.org.uk/results?url=http://www.the-brights.net/ https://www.blocked.org.uk/results?url=http://www.the-bright...
finnn 12y ago
Have you considered setting a user agent other than Python Requests default? Probably not necessary, but good practice. Also, ironically, the app is not available on any of my devices.

justincormack 12y ago

Ah I knew something was in the works as Andrews & Arnold were writing about obtaining connections from ISPs to test what was blocked. Great to know what is being censored (if you are not on A&A).

dfkvldfmv 12y ago
"(if you are not on A&A)." Or if you're on any of the ISPs who do not have filters, including but not limited to A&A.
- SideburnsOfDoom 12y ago
  Who are the other ISPs who do not have filters?
  - dspillett 12y ago
    There is no handy list that I know of. The big six definitely do have the filters. Some smaller players do too, but a fair few don't. The only people I know for sure are A&A because they put their heads above the parapet and made a stand. Some simply haven't implemented it because it would be work and they therefore won't until forced to (rather than because of any moral objection) - they'll not make any statement about not doing it as it will look bad if they eventually do. There may of course be others in A&A's position who haven't done as good a job getting the message out.

spingsprong 12y ago

I've been checking various websites I go on that are completely fine for kids. It's amazing what's getting blocked. One example according to that website is, TalkTalk blocks "http://www.writingexcuses.com/" http://www.writingexcuses.com/" It's a podcast that teaches story writing. Blocking that is insane! And the blocked website itself is blocked on two ISPs https://www.blocked.org.uk/results?url=http%3A%2F%2Fblocked.org.uk https://www.blocked.org.uk/results?url=http%3A%2F%2Fblocked....

TazeTSchnitzel 12y ago
I think TalkTalk is either using a whitelist or an oversensitive keyword filter.
- namos 12y ago
  The TalkTalk filter is the default "KidSafe" level
- vidarh 12y ago
  My personal website is blocked by Talk Talk apparently. I guess compiler writing must be harmful for children.
  - keithpeter 12y ago
    Here we go. If a lot of harmless or potentially even useful material gets blocked, people who want to use that material will simply opt out or route round the blocking, hence making it pointless.
    - vidarh 12y ago
      I kind of hope they overreach drastically for exactly that reason. It'll be a lot easier to get the filters dropped entirely if we can attack them with "useful website X is blocked for no good reason" over and over and over again, than if we have to argue using examples that many people would actually have a problem with.

blueskin_ 12y ago

I just learned that one of my personal sites is blocked on BT and TalkTalk... there is zero adult or other inappropriate content on it. Time to complain, I guess, but I'm sure they will just go straight to /dev/null :(

dspillett 12y ago
Make sure you update the site to state that it is currently blocked by these ISPs, you don't know why, you have asked them to investigate, but you have not heard nothing back and until they respond it is completely out of your control. That way anyone who can access it from work but not home (for example) or recommends the site to a friend who finds it doesn't work, will have a clue as to what the problem is (and it isn't your fault). If your site is of significant general interest then this may also increase the chance of the ISPs acting quickly.
AlyssaRowan 12y ago
Have you considered legal action?
- blueskin_ 12y ago
  Nope. It's a medium sized site I don't get any money out of, not linked to any business. I suppose I could theoretically small claims court them, but I'm not sure if I'd win as they can probably piss about with their customers' connections in whatever way they please due to their T&Cs :(

lucb1e 12y ago

Wow, I had not expected this. My personal blog (which is about technology) is being blocked by one of the ISPs. I don't host porn nor have I ever, so why is it blocked behind their porn filter? Interesting.... https://www.blocked.org.uk/results?url=http://lucb1e.com https://www.blocked.org.uk/results?url=http://lucb1e.com Edit: TalkTalk live chat support reports that I am not being blocked. Citation: > I can confirm that is not blocked by our Adult Filters as your website does not have any adult content. So it seems blocked.org.uk has a false-positive...

blueskin_ 12y ago
TalkTalk are a completely awful ISP. Not sure what the US equivalent would be, but they're the kind who do promotions of "Broadband for £3/month"-level. You get what you pay for service-wise, but they are also by far the most enthusiastic ISP around when it comes to censorship; they have a spider that follows their users to all URLs they visit to categorise the sites[1] (giving them the nickname StalkStalk), and have been doing this long before the current set of idiots in Parliament forced censorship in (at least since 2011). I'm in a similar situation, one site I run is blocked both by StalkStalk and BT. No porn, no other 'adult' content; it's deliberately kept PG-13 to use the US ratings. [1]Check your logs for HuaweiSymantecSpider. It obeys robots.txt though, so a quick "Disallow: /" in robots.txt stops it, although they still grab content from unencrypted pages via DPI.
- lucb1e 12y ago
  No HuaweiSymantecSpider in my logs in the past 8 months (grepped access.log), but the site is available over both http and https so they might have classified it using DPI (talk about evil...).
namos 12y ago
TalkTalk has more than one filter setting. Ask them if your site is blocked at the "Kid Safe" level (the default) which is what the probes run on.
endersshadow 12y ago
BT is blocking my site about my homebrewed beer. That's fun.

jamesbrownuhh 12y ago

Sad to see that The Best Page In The Universe (http://maddox.xmission.com http://maddox.xmission.com) seems to be blocked by a non-zero number of larger ISPs. Obviously all those children need to be protected from harsh reviews of their artwork.