5 ms·

If a website puts something on the public internet, it should not even be aware if it is being accessed by a scraper or a human. Maybe we should just ban User

by smegel 9y ago

If a website puts something on the public internet, it should not even be aware if it is being accessed by a scraper or a human. Maybe we should just ban User Agent strings and be done with it.

rev_bird 9y ago
I'd be willing to bet that the user-agent field isn't the problem; it's patterns that everything looks for now, right? People have been lying in the request headers for decades.
- Maultasche 9y ago
  Yeah, I remember a plugin in the early days of Firefox that allowed Firefox pretend to be Internet Explorer. It would send request headers that made it appear to be IE. There were websites at the time that would display just fine in Firefox, but would refuse to display anything if they detected a non-IE browser.
sbarre 9y ago
You call it the "public internet" but it's most definitely not a public space or anything like it. Private entities own and operate all(most of) the servers, services and conduits, and that does need to be paid for and maintained. I'm not saying I agree with Linkedin in this particular scenario, but this is about two commercial for-profit entities arguing over money, so let's not make it about something it's not.
- smegel 9y ago
  > Private entities own and operate all(most of) the servers, services and conduits, and that does need to be paid for and maintained And are MORE than happy to send the content of their servers to unsolicited, uninvited, anonymous guests on mere request. No-one is forcing them to do so!
  - sbarre 9y ago
    And that's exactly LinkedIn's position here. No one should be forcing them to send their content to anyone. They claim they should be allowed to discriminate at their discretion who they respond to, since they own and operate the servers. This "no one is forcing you to send your content" goes both ways.. If one side is going to say they're entitled to receive the content on request, the other side wants to be able to say they're entitled to refuse to answer that request..
- bogomipz 9y ago
  >"You call it the "public internet" but it's most definitely not a public space or anything like it." How is it not a "pubic space"? They publish publicly visible A records for their site as well as route their public IP space to transit providers in order for the public to be able to reach their site.
  - umanwizard 9y ago
    If I publicly distribute posters about a party in my apartment and leave the door unlocked, do I still have the right to kick out anyone I don't like?
    - bogomipz 9y ago
      That's a pretty diengenuous strawman. A more accurate comparison would be that you put up an advertisement on a billboard along a busy street and then decided to tell people who passed by that they weren't allowed to take a picture of it. And to continue with this absurdity you feel entitled to enforce who can or can not look at your billboard because despite it being publicly viewable its your advertisement on the billboard.
      - sbarre 9y ago
        That's also not accurate. There is no "public space" on the Internet.. There's no un-owned territory or resource that is free to use or metaphorically "stand around" in to take those pictures from. You are consuming privately-owned resources in all your online activities, and as such some will argue that they can decide to limit your consumption of those resources at their own discretion. Again, I am not siding with either party here, just trying to dispel this notion that "public space" - in the way we understand public space to exist in the physical world - exists on the internet. In your example, no one is controlling your right to take photos or stand around and look in any direction you choose. When you use the Internet, a private entity is allowing you to transit through their network and access sites, a different entity is allowing you to access and receive their content, etc.. LinkedIn owns the server you are accessing when you (or others) go to their site, and they are spending resources servicing those requests, and - they claim - can decide how and when they choose to do that..
laumars 9y ago
It really doesn't take much effort to detect the majority of scrapers. Usually you do so by monitoring patterns of any given IP. Is each request a profile page incremented (/users/1, /users/2, etc) or dozens of requests a minute (faster than a typical user would read)? Is static content (particularly images and CSS) being downloaded too or just the HTML content? Sometimes the referrer HTTP header can give clues too - though you have to be careful there as that's as unreliable as the user agent header. However if you're really paranoid about scrapers you can also throw in some honeypots. eg a fake user (/users/13) which is a user account that doesn't exist so that page wouldn't have any links from within your site. ie you only reach it if you're incrementing through the user IDs. Or perhaps a link within your HTML which doesn't render so it's only reachable via automated scripts that don't check what links are rendered inside the display view. Anyone that gets ensnared in your honeypot could then be put on a temporary IP blacklist. Though the danger of doing this is you accidentally blacklist good crawlers if you're not careful about setting appropriate robots rules.
CaptSpify 9y ago
I've kind of always thought that we shouldn't be using UA strings. Just give the requester the data that they requested according to the current open standards. If they choose not to render it correctly, then thats their problem. Yes, I realize that it's not that simple, but I think browsers would have tried much harder to adhere to standards if we had done it that way.

by smegel 9y ago

If a website puts something on the public internet, it should not even be aware if it is being accessed by a scraper or a human. Maybe we should just ban User Agent strings and be done with it.

rev_bird 9y ago

I'd be willing to bet that the user-agent field isn't the problem; it's patterns that everything looks for now, right? People have been lying in the request headers for decades.

Maultasche 9y ago
Yeah, I remember a plugin in the early days of Firefox that allowed Firefox pretend to be Internet Explorer. It would send request headers that made it appear to be IE. There were websites at the time that would display just fine in Firefox, but would refuse to display anything if they detected a non-IE browser.

sbarre 9y ago

You call it the "public internet" but it's most definitely not a public space or anything like it. Private entities own and operate all(most of) the servers, services and conduits, and that does need to be paid for and maintained. I'm not saying I agree with Linkedin in this particular scenario, but this is about two commercial for-profit entities arguing over money, so let's not make it about something it's not.

smegel 9y ago
> Private entities own and operate all(most of) the servers, services and conduits, and that does need to be paid for and maintained And are MORE than happy to send the content of their servers to unsolicited, uninvited, anonymous guests on mere request. No-one is forcing them to do so!
- sbarre 9y ago
  And that's exactly LinkedIn's position here. No one should be forcing them to send their content to anyone. They claim they should be allowed to discriminate at their discretion who they respond to, since they own and operate the servers. This "no one is forcing you to send your content" goes both ways.. If one side is going to say they're entitled to receive the content on request, the other side wants to be able to say they're entitled to refuse to answer that request..
bogomipz 9y ago
>"You call it the "public internet" but it's most definitely not a public space or anything like it." How is it not a "pubic space"? They publish publicly visible A records for their site as well as route their public IP space to transit providers in order for the public to be able to reach their site.
- umanwizard 9y ago
  If I publicly distribute posters about a party in my apartment and leave the door unlocked, do I still have the right to kick out anyone I don't like?
  - bogomipz 9y ago
    That's a pretty diengenuous strawman. A more accurate comparison would be that you put up an advertisement on a billboard along a busy street and then decided to tell people who passed by that they weren't allowed to take a picture of it. And to continue with this absurdity you feel entitled to enforce who can or can not look at your billboard because despite it being publicly viewable its your advertisement on the billboard.
    - sbarre 9y ago
      That's also not accurate. There is no "public space" on the Internet.. There's no un-owned territory or resource that is free to use or metaphorically "stand around" in to take those pictures from. You are consuming privately-owned resources in all your online activities, and as such some will argue that they can decide to limit your consumption of those resources at their own discretion. Again, I am not siding with either party here, just trying to dispel this notion that "public space" - in the way we understand public space to exist in the physical world - exists on the internet. In your example, no one is controlling your right to take photos or stand around and look in any direction you choose. When you use the Internet, a private entity is allowing you to transit through their network and access sites, a different entity is allowing you to access and receive their content, etc.. LinkedIn owns the server you are accessing when you (or others) go to their site, and they are spending resources servicing those requests, and - they claim - can decide how and when they choose to do that..

laumars 9y ago

It really doesn't take much effort to detect the majority of scrapers. Usually you do so by monitoring patterns of any given IP. Is each request a profile page incremented (/users/1, /users/2, etc) or dozens of requests a minute (faster than a typical user would read)? Is static content (particularly images and CSS) being downloaded too or just the HTML content? Sometimes the referrer HTTP header can give clues too - though you have to be careful there as that's as unreliable as the user agent header. However if you're really paranoid about scrapers you can also throw in some honeypots. eg a fake user (/users/13) which is a user account that doesn't exist so that page wouldn't have any links from within your site. ie you only reach it if you're incrementing through the user IDs. Or perhaps a link within your HTML which doesn't render so it's only reachable via automated scripts that don't check what links are rendered inside the display view. Anyone that gets ensnared in your honeypot could then be put on a temporary IP blacklist. Though the danger of doing this is you accidentally blacklist good crawlers if you're not careful about setting appropriate robots rules.

CaptSpify 9y ago

I've kind of always thought that we shouldn't be using UA strings. Just give the requester the data that they requested according to the current open standards. If they choose not to render it correctly, then thats their problem. Yes, I realize that it's not that simple, but I think browsers would have tried much harder to adhere to standards if we had done it that way.