11 ms·

>What makes Google's legitimate over OP's? Respecting robots.txt, using a well-known ua, coming from publicly declared, google owned IP block.

by avip 7y ago

AznHisoka 7y ago
"well-known ua" So if you're a small business that wants to build the next Google, you can't? Because obviously, your User Agent won't be well known when you start.
- cft 7y ago
  I think IPO empowered Matthew Prince with his benevolent extrajuditial internet court is becoming a real threat for the open web.
  - AznHisoka 7y ago
    I don't have a problem with Cloudflare's reach and power, but would love for them to be transparent as to what they constitute a bad bot. What's the rules? Follow robots.txt, Scrape just X pages a minute, what else?
    - Crosseye_Jack 7y ago
      The problem I see is they won't explicitly say what would trigger being flagged as bad (nor what would be considered allowable) as they would claim (as almost everyone in the anti cheat field does) it would give too much info to the bad guys to avoid detection. I would love for them to prove me wrong though and be open about such things.
      - danShumway 7y ago
        > it would give too much info to the bad guys to avoid detection. I'm pretty sympathetic to this line of thought, so don't take this as 'you're wrong', but I notice that we would never apply this logic to laws, or environmental standards, or contracts. You'd never hear someone say, "if we have a clearly defined tax code, that will just make it easier for people to find loopholes." Where I do hear this argument come up is explicitly in contexts of moderation and abuse policies. And it's something that sounds very reasonable, but it's hard for me to get away from the fact that in most other contexts it sounds problematic to me. Maybe part of the problem is Cloudflare's scale? Maybe the reason it feels bad to have a police officer pull me over because, "we think you're going too fast" instead of "you went over a posted limit", is because that's a critical infrastructure that I can't avoid. Cloudflare is a private company, and even if it wasn't, I don't know if it would be big enough presence that I would worry about their policies. But when I hear things like this: > Our goal is nothing short of making it no longer viable to run a malicious bot on the Internet. And we think, with our scale, we can do exactly that. That maybe shifts the situation a tiny bit farther away from "moderation policy on a personal blog" towards "policeman pulling me over because I broke a law I didn't know existed." I dunno. I'm not 100% sure how to feel about it. I do think that Cloudflare should be able to filter traffic however they see fit, but that doesn't mean that every strategy they choose is inherently good, or that it might not be problematic for them to lack transparency about their standards.
    - avip 7y ago
      I know this is a cliché, but a fraud detection system cannot work transparently. Also for fairness - Does Akamai publicize the internals of BotManager? Imperva gives you much transparency into how DistillNetworks works?
      - yjftsjthsd-h 7y ago
        > I know this is a cliché, but a fraud detection system cannot work transparently. Then fraud detection system cannot work ethically.
- mfontani 7y ago
  If you're doing that using Googlebot's UA, you shouldn't. Pick your own.
  - Crosseye_Jack 7y ago
    The way I took what they were saying is they wouldn't be using Googlebot's UA, so they wouldn't be a "well-known UA".
Crosseye_Jack 7y ago
Who says OP's crawler doesn't respect robots.txt? Can't become a well-known UA if your shutdown before you can make a name for yourself. And a Google IP block doesn't mean legitimate. I'm not trying to say that Google is bad, but why can't OP's crawler be legitimate?
- avip 7y ago
  >Google IP block doesn't mean legitimate It does. 2-way DNS lookup is how you verify legit indexers [0] (as ua itself is meaningless). [0] https://support.google.com/webmasters/answer/80553 https://support.google.com/webmasters/answer/80553
  - Crosseye_Jack 7y ago
    Ok my wording was slightly off. What I was trying to say is, just because a crawl doesn't come from a Google IP block doesn't mean its not a legit crawl from someone outside of Google.
    - mfontani 7y ago
      The way I take that to mean is: if you're using a Googlebot UA, and your request doesn't come from a Google-owned IP range, you're very unlikely to be a legit crawl from Google, and you'll be detected and dealt with.
      - Crosseye_Jack 7y ago
        Which would be fine imo. But as the person I was replying to says you can already detect a bad bot pretending to be google and deal with them how you wish (my fav is just feeding them random values). But what are CF (and others like akamai) using to determine you are a bad bots? My objection is more the threat they will use their position to kick bad bots offline. Now don't get me wrong, I had bad bots as much as the next person. But what about misfires? They will happen. I hope CF have a plan in place to deal with such cases. Personally, due to family issues I've been working remote-remote (I'm a remote worker anyways, but I've been working remotely away from my normal office). As such I've been using a 4G Mobile broadband connection and the amount of CloudFlare "We are just checking you are not a bot" hoops I've had to jump though recently has opened my eyes on how much of the net is actually behind CF's network because CF still associate "IP as Identity" and my 4G connection is CG-NAT'ed. I just fear at times we forget that there is the rest of the world out there, not just our little bubble and the choices we make have a huge impact on the usability of the net for others. EDIT: LOL.. Just checked your profile. The Reg is one of the sites that doesn't like my 4G connection and as such I visit less. What are the chances. (Though you only make me sit there and wait to access the content....)

>What makes Google's legitimate over OP's? Respecting robots.txt, using a well-known ua, coming from publicly declared, google owned IP block.

by avip 7y ago

>What makes Google's legitimate over OP's? Respecting robots.txt, using a well-known ua, coming from publicly declared, google owned IP block.

AznHisoka 7y ago

"well-known ua" So if you're a small business that wants to build the next Google, you can't? Because obviously, your User Agent won't be well known when you start.

cft 7y ago
I think IPO empowered Matthew Prince with his benevolent extrajuditial internet court is becoming a real threat for the open web.
- AznHisoka 7y ago
  I don't have a problem with Cloudflare's reach and power, but would love for them to be transparent as to what they constitute a bad bot. What's the rules? Follow robots.txt, Scrape just X pages a minute, what else?
  - Crosseye_Jack 7y ago
    The problem I see is they won't explicitly say what would trigger being flagged as bad (nor what would be considered allowable) as they would claim (as almost everyone in the anti cheat field does) it would give too much info to the bad guys to avoid detection. I would love for them to prove me wrong though and be open about such things.
    - danShumway 7y ago
      > it would give too much info to the bad guys to avoid detection. I'm pretty sympathetic to this line of thought, so don't take this as 'you're wrong', but I notice that we would never apply this logic to laws, or environmental standards, or contracts. You'd never hear someone say, "if we have a clearly defined tax code, that will just make it easier for people to find loopholes." Where I do hear this argument come up is explicitly in contexts of moderation and abuse policies. And it's something that sounds very reasonable, but it's hard for me to get away from the fact that in most other contexts it sounds problematic to me. Maybe part of the problem is Cloudflare's scale? Maybe the reason it feels bad to have a police officer pull me over because, "we think you're going too fast" instead of "you went over a posted limit", is because that's a critical infrastructure that I can't avoid. Cloudflare is a private company, and even if it wasn't, I don't know if it would be big enough presence that I would worry about their policies. But when I hear things like this: > Our goal is nothing short of making it no longer viable to run a malicious bot on the Internet. And we think, with our scale, we can do exactly that. That maybe shifts the situation a tiny bit farther away from "moderation policy on a personal blog" towards "policeman pulling me over because I broke a law I didn't know existed." I dunno. I'm not 100% sure how to feel about it. I do think that Cloudflare should be able to filter traffic however they see fit, but that doesn't mean that every strategy they choose is inherently good, or that it might not be problematic for them to lack transparency about their standards.
  - avip 7y ago
    I know this is a cliché, but a fraud detection system cannot work transparently. Also for fairness - Does Akamai publicize the internals of BotManager? Imperva gives you much transparency into how DistillNetworks works?
    - yjftsjthsd-h 7y ago
      > I know this is a cliché, but a fraud detection system cannot work transparently. Then fraud detection system cannot work ethically.
mfontani 7y ago
If you're doing that using Googlebot's UA, you shouldn't. Pick your own.
- Crosseye_Jack 7y ago
  The way I took what they were saying is they wouldn't be using Googlebot's UA, so they wouldn't be a "well-known UA".

Crosseye_Jack 7y ago

Who says OP's crawler doesn't respect robots.txt? Can't become a well-known UA if your shutdown before you can make a name for yourself. And a Google IP block doesn't mean legitimate. I'm not trying to say that Google is bad, but why can't OP's crawler be legitimate?

avip 7y ago
>Google IP block doesn't mean legitimate It does. 2-way DNS lookup is how you verify legit indexers [0] (as ua itself is meaningless). [0] https://support.google.com/webmasters/answer/80553 https://support.google.com/webmasters/answer/80553
- Crosseye_Jack 7y ago
  Ok my wording was slightly off. What I was trying to say is, just because a crawl doesn't come from a Google IP block doesn't mean its not a legit crawl from someone outside of Google.
  - mfontani 7y ago
    The way I take that to mean is: if you're using a Googlebot UA, and your request doesn't come from a Google-owned IP range, you're very unlikely to be a legit crawl from Google, and you'll be detected and dealt with.
    - Crosseye_Jack 7y ago
      Which would be fine imo. But as the person I was replying to says you can already detect a bad bot pretending to be google and deal with them how you wish (my fav is just feeding them random values). But what are CF (and others like akamai) using to determine you are a bad bots? My objection is more the threat they will use their position to kick bad bots offline. Now don't get me wrong, I had bad bots as much as the next person. But what about misfires? They will happen. I hope CF have a plan in place to deal with such cases. Personally, due to family issues I've been working remote-remote (I'm a remote worker anyways, but I've been working remotely away from my normal office). As such I've been using a 4G Mobile broadband connection and the amount of CloudFlare "We are just checking you are not a bot" hoops I've had to jump though recently has opened my eyes on how much of the net is actually behind CF's network because CF still associate "IP as Identity" and my 4G connection is CG-NAT'ed. I just fear at times we forget that there is the rest of the world out there, not just our little bubble and the choices we make have a huge impact on the usability of the net for others. EDIT: LOL.. Just checked your profile. The Reg is one of the sites that doesn't like my 4G connection and as such I visit less. What are the chances. (Though you only make me sit there and wait to access the content....)