5 ms·

Google, a trillion dollar company, is essentially the world's largest web scraper. So...yes! You'll almost certainly find a way to monetize that. Monopolies, l

by apienx 5y ago

Google, a trillion dollar company, is essentially the world's largest web scraper. So...yes! You'll almost certainly find a way to monetize that. Monopolies, lobbying and protectionism got in the way of keeping the web truly machine readable. There's tremendous value in restoring some of it.

fforflo 5y ago
> keeping the web truly machine readable. There's tremendous value in restoring some of it. :-)
dbtc 5y ago
Web scraping is just one way to get the data to facilitate the sale and service of ads.
elliottinvent 5y ago
> Monopolies, lobbying and protectionism got in the way of keeping the web truly machine readable. Exactly and that ship has long since sailed. The good ship Web 3.0 (semantic web) launched in ‘99 and was a ghost ship until recently when it was boarded by crypto pirates now flying the web 3.0 flag. > There's tremendous value in restoring some of it. To this comment and OP, my startup is using web scraping to pre-populate machine-readable data for a DNS-based protocol called NUM [0]. So as others have said, whilst web scraping itself may be difficult to build into a viable business, it can be a key component of a viable business. Email in my profile if you want to discuss. 0. https://num.uk/blog/we-crawled-5m-uk-websites-and-published-23m-data-points-to-dns https://num.uk/blog/we-crawled-5m-uk-websites-and-published-...
- melony 5y ago
  Web3 (without the decimal point) these days usually refer to internet Ponzi schemes.
- ozfive 5y ago
  "boarded by crypto pirates now flying the web 3.0 flag." --Sad truth The Semantic Web would have given much better search among other benefits.
- deleted 5y ago
  [deleted]
asdfasgasdgasdg 5y ago
> Google, a trillion dollar company, is essentially the world's largest web scraper. Even just considering the parts of Google that it takes to bring you the N blue links part of the Google SERP, the web scraper is probably the least interesting and significant piece of technology in the stack. It's beyond reductive to say that Google is in essence a large web scraper, or a web scraper of any kind. It is like saying that a person is, in essence, the world's largest mouth. I think it's incredibly difficult to build a profitable business in this space. The number of customers who a) need to scrape the web b) aren't sophisticated enough to do it themselves and c) are big enough to make $$$ from are small. The important bit is always processing the web pages for whatever content is salient to the given customer. Which means that you need to deliver the web pages to them. So effectively your business is providing nothing more than URL lists and potentially some additional metadata compared to what the customer would get if they fetched the pages themselves. There are definitely some other complexities you could resolve, but it's hard to imagine that those benefits would be enough to build a business on.
- bagacrap 5y ago
  why is Google's web scraping boring/easy? I'm sure it must be very sophisticated/optimized since they want it to be done often and cheaply
  - asdfasgasdgasdg 5y ago
    It is not boring, it is just not as interesting/difficult as the remainder of the stack.
lifefeed 5y ago
And there's another entire ecosystem around scraping Google's results and wrapping it in an API. Just scraping upon scraping.

by apienx 5y ago

fforflo 5y ago

> keeping the web truly machine readable. There's tremendous value in restoring some of it. :-)

dbtc 5y ago

Web scraping is just one way to get the data to facilitate the sale and service of ads.

elliottinvent 5y ago

> Monopolies, lobbying and protectionism got in the way of keeping the web truly machine readable. Exactly and that ship has long since sailed. The good ship Web 3.0 (semantic web) launched in ‘99 and was a ghost ship until recently when it was boarded by crypto pirates now flying the web 3.0 flag. > There's tremendous value in restoring some of it. To this comment and OP, my startup is using web scraping to pre-populate machine-readable data for a DNS-based protocol called NUM [0]. So as others have said, whilst web scraping itself may be difficult to build into a viable business, it can be a key component of a viable business. Email in my profile if you want to discuss. 0. https://num.uk/blog/we-crawled-5m-uk-websites-and-published-23m-data-points-to-dns https://num.uk/blog/we-crawled-5m-uk-websites-and-published-...

melony 5y ago
Web3 (without the decimal point) these days usually refer to internet Ponzi schemes.
ozfive 5y ago
"boarded by crypto pirates now flying the web 3.0 flag." --Sad truth The Semantic Web would have given much better search among other benefits.
deleted 5y ago
[deleted]

asdfasgasdgasdg 5y ago

> Google, a trillion dollar company, is essentially the world's largest web scraper. Even just considering the parts of Google that it takes to bring you the N blue links part of the Google SERP, the web scraper is probably the least interesting and significant piece of technology in the stack. It's beyond reductive to say that Google is in essence a large web scraper, or a web scraper of any kind. It is like saying that a person is, in essence, the world's largest mouth. I think it's incredibly difficult to build a profitable business in this space. The number of customers who a) need to scrape the web b) aren't sophisticated enough to do it themselves and c) are big enough to make $$$ from are small. The important bit is always processing the web pages for whatever content is salient to the given customer. Which means that you need to deliver the web pages to them. So effectively your business is providing nothing more than URL lists and potentially some additional metadata compared to what the customer would get if they fetched the pages themselves. There are definitely some other complexities you could resolve, but it's hard to imagine that those benefits would be enough to build a business on.

bagacrap 5y ago
why is Google's web scraping boring/easy? I'm sure it must be very sophisticated/optimized since they want it to be done often and cheaply
- asdfasgasdgasdg 5y ago
  It is not boring, it is just not as interesting/difficult as the remainder of the stack.

lifefeed 5y ago

And there's another entire ecosystem around scraping Google's results and wrapping it in an API. Just scraping upon scraping.