6 ms·

I was always under the impression job sites don’t like to be scraped. What are the risks in using a tool like this? I would suspect possibly getting your ip ban

by city41 7y ago

I was always under the impression job sites don’t like to be scraped. What are the risks in using a tool like this? I would suspect possibly getting your ip banned.

slenk 7y ago
Yeah I was under a similar impression. If not, this is great as I am currently looking
walshemj 7y ago
This is indeed's business model - certainly that's what the big uk recruitment sites thought.
everdev 7y ago
As always, consult a lawyer. But, my understanding is that you are allowed to scrape sites as long as you don't impact their ability to deliver the site to other users. In other words, don't DDoS them. Sites tried to ban scraping but last I remember seeing it was not enforceable unless it was done at high rates or maliciously. As for content, certain things like recipes aren't protected content because they are "listings of ingredients". I can imagine a job opening / title might follow that same logic, maybe the job requirements and benefits as well. What is protected is the way it's presented and organized. So if you have the same jobs with the same filters and same layout you might be exposing yourself to trouble. You have to make changes in the way it's presented, which for copyright law is changing 50% or more. So if you're scraping a job site and only showing remote jobs or jobs in a certain skill set, or you're showing the job along with some stats about the company you're probably in a safe spot. But again, if you want to make a business out of it it's probably good to talk to a lawyer first.
- sverhagen 7y ago
  I don't think the question was: "is it legal to scrape", rather: "do they like to be scraped". If they wanted to, I'm sure they could make scraping harder (more annoying). Which, I suppose, thus comes down to: is being aggregated a net plus or minus for their business. Probably "it depends" on what the aggregator does with the data. So, just to get this pun in: it then comes down to: is being aggregated a net plus or minus, IN AGGREGATE.
  - everdev 7y ago
    Then no, very few for profit companies like to be scraped. It would be easier for them to provide an RSS feed or a free API if they wanted to share their data.

by city41 7y ago

I was always under the impression job sites don’t like to be scraped. What are the risks in using a tool like this? I would suspect possibly getting your ip banned.

slenk 7y ago

Yeah I was under a similar impression. If not, this is great as I am currently looking

walshemj 7y ago

This is indeed's business model - certainly that's what the big uk recruitment sites thought.

everdev 7y ago

As always, consult a lawyer. But, my understanding is that you are allowed to scrape sites as long as you don't impact their ability to deliver the site to other users. In other words, don't DDoS them. Sites tried to ban scraping but last I remember seeing it was not enforceable unless it was done at high rates or maliciously. As for content, certain things like recipes aren't protected content because they are "listings of ingredients". I can imagine a job opening / title might follow that same logic, maybe the job requirements and benefits as well. What is protected is the way it's presented and organized. So if you have the same jobs with the same filters and same layout you might be exposing yourself to trouble. You have to make changes in the way it's presented, which for copyright law is changing 50% or more. So if you're scraping a job site and only showing remote jobs or jobs in a certain skill set, or you're showing the job along with some stats about the company you're probably in a safe spot. But again, if you want to make a business out of it it's probably good to talk to a lawyer first.

sverhagen 7y ago
I don't think the question was: "is it legal to scrape", rather: "do they like to be scraped". If they wanted to, I'm sure they could make scraping harder (more annoying). Which, I suppose, thus comes down to: is being aggregated a net plus or minus for their business. Probably "it depends" on what the aggregator does with the data. So, just to get this pun in: it then comes down to: is being aggregated a net plus or minus, IN AGGREGATE.
- everdev 7y ago
  Then no, very few for profit companies like to be scraped. It would be easier for them to provide an RSS feed or a free API if they wanted to share their data.