3 ms·

So from the website's point of view there is no difference between 'crawling' and 'scraping'. Census.gov I assume has a ton of very useful information which is

by JackFr 6y ago

So from the website's point of view there is no difference between 'crawling' and 'scraping'. Census.gov I assume has a ton of very useful information which is in the public domain which a host of potential companies could monetize by regularly scraping census.gov. Census.gov's purpose to make this information available to people is served by google, yahoo and bing. On the other hand if I have a business which is based on that data, in fact I'm at cross purposes to them.

mulmen 6y ago
But Google, Yahoo and Bing are also monetizing the data. Why are they allowed to provide “benefits” but “scrapers” are not? Why is it wrong to monetize public data?
njharman 6y ago
I'm generally anti business. But I have to disagree. "The Public" that the government serves includes businesses. Businesses (ignoring corporate personhood bullshit) are owned and operated by people. I do not want the government deciding "what purposes" e.g. non-commercial, serve the public good. The public gets to decide that. (charging a license for commercial use is maybe ok (assuming supporting that use costs government "too much"). And I very do not want current situation with the government anointing a handful of corporations (the farthest thing from the public possible) access and denying everyone else including all of the actual public.
- pessimizer 6y ago
  A specific case where this favorite-picking by government enables corruption: https://en.wikipedia.org/wiki/Nationally_recognized_statistical_rating_organization https://en.wikipedia.org/wiki/Nationally_recognized_statisti... And an example from the quickly-approaching future, when there will be Nationally Recognized Media Organizations who license "Fact-Checkers," through which posts to public-facing will have to be submitted for certification and correction.
  - marcosdumay 6y ago
    Favorite-picking by the government is corruption by itself already.
- hnbroseph 6y ago
  > I do not want the government deciding "what purposes" e.g. non-commercial, serve the public good. The public gets to decide that. the public's "decision" on things like this is made manifest by government policy, no?
  - danShumway 6y ago
    In theory. In practice, is every single policy that our government upholds currently popular with the majority of people? It's possible to have government policies that the majority of people disagree with, that remain for complicated reasons related to apathy, lobbying, party ideology, or just because those issues get drowned out by more important debates. Government is an extension of the will of the people, but the farther out that extension gets, the more divorced from the will of the people it's possible to be. That's not to say that businesses are immune from that effect either -- there are markets where the majority of people participating in them aren't happy with what the market is offering. All of these systems are abstractions, they're ways of trying to get closer to public will, and they're all imperfect. But government is particularly abstracted, especially because the US is not a direct democracy. I'm personally of the opinion that this discussion is moot, because I think that people have a fundamental Right to Delegate[0], and I include web scraping public content under that right. But ignoring that, because not everyone agrees with me that delegation is right, allowing the government to unilaterally rule on who isn't allowed to access public information is still particularly susceptible to abuse above and beyond what the market is capable of. [0]: https://anewdigitalmanifesto.com/#right-to-delegate https://anewdigitalmanifesto.com/#right-to-delegate
jonas21 6y ago
The census data is available for bulk download, mostly as CSV (for example [1]). Scraping census.gov is worse for both the Census Bureau (which might have to do an expensive database query for each page) and for the scraper (who has to parse the page). Blocking scrapers in robots.txt is more of a way of saying, "hey, you're doing it wrong." It's also worth noting that the original article is out of date. The current robots.txt at census.gov is basically wide-open [2]. [1] https://www.census.gov/programs-surveys/acs/data/data-via-ftp.html https://www.census.gov/programs-surveys/acs/data/data-via-ft... [2] https://www.census.gov/robots.txt https://www.census.gov/robots.txt
- foobar33333 6y ago
  Scrapers don't care about robots.txt. I have scraped multiple websites in a previous job and the robots.txt means nothing. Bigger sites might detect and block you but most don't.
ricardo81 6y ago
Having data in the right format as a download or via an API would be the best way to go for public data. If people have to 'scrape' that data from a public resource, I'd say they're presenting the data in the wrong way.
vharuck 6y ago
In the case of Census.gov, they offer an API to get the data[0]. It's actually pretty nice. Stable, ton of data, fairly uniform data structure across the different products. Very high rate limits, considering most data only needs retrieved once a year. I think they understand the difference between crawling and scraping. [1] https://www.census.gov/data/developers.html https://www.census.gov/data/developers.html
indymike 6y ago
I used to run a fairly large job ad scraping operation. Our scraped data was used by many US state and federal job sites. "Scraping" is just using software to load a page and extracting content. "Crawling" is just load a page, find hyperlinks (hmm... a kind of content), and then crawling those links. Crawling is just a kind of scraping.

by JackFr 6y ago

mulmen 6y ago

But Google, Yahoo and Bing are also monetizing the data. Why are they allowed to provide “benefits” but “scrapers” are not? Why is it wrong to monetize public data?

njharman 6y ago

I'm generally anti business. But I have to disagree. "The Public" that the government serves includes businesses. Businesses (ignoring corporate personhood bullshit) are owned and operated by people. I do not want the government deciding "what purposes" e.g. non-commercial, serve the public good. The public gets to decide that. (charging a license for commercial use is maybe ok (assuming supporting that use costs government "too much"). And I very do not want current situation with the government anointing a handful of corporations (the farthest thing from the public possible) access and denying everyone else including all of the actual public.

pessimizer 6y ago
A specific case where this favorite-picking by government enables corruption: https://en.wikipedia.org/wiki/Nationally_recognized_statistical_rating_organization https://en.wikipedia.org/wiki/Nationally_recognized_statisti... And an example from the quickly-approaching future, when there will be Nationally Recognized Media Organizations who license "Fact-Checkers," through which posts to public-facing will have to be submitted for certification and correction.
- marcosdumay 6y ago
  Favorite-picking by the government is corruption by itself already.
hnbroseph 6y ago
> I do not want the government deciding "what purposes" e.g. non-commercial, serve the public good. The public gets to decide that. the public's "decision" on things like this is made manifest by government policy, no?
- danShumway 6y ago
  In theory. In practice, is every single policy that our government upholds currently popular with the majority of people? It's possible to have government policies that the majority of people disagree with, that remain for complicated reasons related to apathy, lobbying, party ideology, or just because those issues get drowned out by more important debates. Government is an extension of the will of the people, but the farther out that extension gets, the more divorced from the will of the people it's possible to be. That's not to say that businesses are immune from that effect either -- there are markets where the majority of people participating in them aren't happy with what the market is offering. All of these systems are abstractions, they're ways of trying to get closer to public will, and they're all imperfect. But government is particularly abstracted, especially because the US is not a direct democracy. I'm personally of the opinion that this discussion is moot, because I think that people have a fundamental Right to Delegate[0], and I include web scraping public content under that right. But ignoring that, because not everyone agrees with me that delegation is right, allowing the government to unilaterally rule on who isn't allowed to access public information is still particularly susceptible to abuse above and beyond what the market is capable of. [0]: https://anewdigitalmanifesto.com/#right-to-delegate https://anewdigitalmanifesto.com/#right-to-delegate

jonas21 6y ago

The census data is available for bulk download, mostly as CSV (for example [1]). Scraping census.gov is worse for both the Census Bureau (which might have to do an expensive database query for each page) and for the scraper (who has to parse the page). Blocking scrapers in robots.txt is more of a way of saying, "hey, you're doing it wrong." It's also worth noting that the original article is out of date. The current robots.txt at census.gov is basically wide-open [2]. [1] https://www.census.gov/programs-surveys/acs/data/data-via-ftp.html https://www.census.gov/programs-surveys/acs/data/data-via-ft... [2] https://www.census.gov/robots.txt https://www.census.gov/robots.txt

foobar33333 6y ago
Scrapers don't care about robots.txt. I have scraped multiple websites in a previous job and the robots.txt means nothing. Bigger sites might detect and block you but most don't.

ricardo81 6y ago

Having data in the right format as a download or via an API would be the best way to go for public data. If people have to 'scrape' that data from a public resource, I'd say they're presenting the data in the wrong way.

vharuck 6y ago

In the case of Census.gov, they offer an API to get the data[0]. It's actually pretty nice. Stable, ton of data, fairly uniform data structure across the different products. Very high rate limits, considering most data only needs retrieved once a year. I think they understand the difference between crawling and scraping. [1] https://www.census.gov/data/developers.html https://www.census.gov/data/developers.html

indymike 6y ago

I used to run a fairly large job ad scraping operation. Our scraped data was used by many US state and federal job sites. "Scraping" is just using software to load a page and extracting content. "Crawling" is just load a page, find hyperlinks (hmm... a kind of content), and then crawling those links. Crawling is just a kind of scraping.