10 ms·
I read these things and I think "So much wasted energy and effort" In the beginning was the web, and it was good. Content came along. Some was good, some was c
by DanielBMarkham 9y ago
I read these things and I think "So much wasted energy and effort"
In the beginning was the web, and it was good. Content came along. Some was good, some was cats. Then paid sites with sign-up. Then search engines. Then ads.
Pretty soon folks thought "I not only own this content, I own how it will be presented to the end user. If I choose to add in cats, or Flash ads, or whatnot? They're stuck consuming it. I own everything about the content from the server to the mind of the person consuming it, the entire pipe."
Many people did not like this idea. Ads were malicious, they installed malware. The practice of using ads on content caused sites to track users like lab rats. Armies of people majoring in psychology were hired to try to make the rats do more of what we wanted them to do.
Ad blockers were born. Then anti-ad-blockers. Then headless browsers. Now anti-headeless browsers.
It's just a huge waste of time and energy. The model is broken, and no amount of secret hacker ninja shit is going to make it work. You want to know where we'll end up? We'll end up with multiple VMs, each with a statistically common setup, each consuming content on the web looking just like a human doing it. (We'll be able to do that by tracking actual humans as they consume content). But nobody will be looking at those VMs. Instead, those invisible screens will be read by image recognition software which will then condense what's on there and send the results back to whoever wants it.
Content providers will never win at this. Nor should they. Instead, we're just going to sink billions into a busted-ass business model over the next couple of decades throwing good money after bad.
</rant>
- ivanhoe 9y agoIt's more complicated than that, almost all spidering that I do lately for clients is not about scraping the content in a classical sense, but some form of "spying" of one's competition, mainly in e-commerce. Tracking inventories, fine-tuning prices, tracking how they promote certain products online, etc. Not to mention really blackhat stuff that many people do like attacking competition with fraudulent ad clicks and similar. Many times "the content providers" have every right to wish to protect themselves.
- jon-wood 9y agoI'll add my piece of anecdata to this. I've done a few projects that involved scraping over my career. 1. Scraping results from a property listing website, specifically to pull the properties an agent had listed and put them on their website. The agent didn't want to pay the fee for API access (they probably ended up paying my employer more to scrape it, and keep that updated, but hey). 2. Scraping an e-commerce website of a company my employer were working with to keep our product catalogues in sync - the partner had their own platform, but no API for it. 3. Automating requests to a price comparison website to find out what prices competitors are offering for particular types of customer.
- jandrese 9y agoAPI keys can be revoked at any time or the API service can be suddenly terminated. You can't trust them. Scraping is more maintenance, but it is more reliable.
- emodendroket 9y agoBut whoever you're scraping can suddenly break your scraping tool too.
- jandrese 9y agoTemporarily until you work around what they're doing. API key revocation leaves you dead in the water until you rewrite your app to do scraping.
- emodendroket 9y agoYeah, but imagine they totally redesign their Web site. The "workaround" is a complete rewrite.
- jandrese 9y agoUltimately it has to show the same data unless the site is completely discontinuing the service (in which case you are up a creek in either scenario). Even in the face of a total redesign you're usually just adjusting a few signatures and maybe tweaking the traversal tree. You can run into problems when they remove things like chronological listing view in favor of some "algorithm" that you can't control, but even that just adds a little work on the back end to detect duplicates and an asterisk that some of the content will be missed. There can also be an arms race where the site developers will start trying all kinds of crazy techniques to make scraping harder. Ironically this often happens after they revoke API access, meaning users don't have an alternative except to participate in the arms race.
- kuschku 9y ago> "spying" of one's competition, mainly in e-commerce. Tracking inventories, fine-tuning prices, tracking how they promote certain products online, etc. That’s not spying, that’s a completely legitimate and normal use case of the web. Everything on the web is public, and the web was also created to allow that.
- klodolph 9y agoThat’s why the word “spying” is in quotes.
- dx034 9y agoThat has been done for decades. Most manufacturers employ scores of people going into shops to check how products are presented. Supermarkets employ people to check traffic and sales at competing stores. Doing this online can be easier (if you know how to do it) but it's hardly new. In the offline world companies also try to protect themselves (you can get banned from stores when they find out that you work for the competition) but it's harder to do.
- remus 9y agoLet's not forget the part where people write shitty scripts that bombard your server with 10x the quantity of traffic you see from actual users. There's shittiness from both producers and consumers.
- DanielBMarkham 9y agoAgreed. It's a terrible waste of time, money, and energy from tens of thousands of people. All for nothing.
- emodendroket 9y agoI think you might be misunderstanding. Someone's scripted use of a Web site can burn more resources, he's saying, but it's not "for nothing" -- they're deriving some benefit from the scripting.
- TeMPOraL 9y agoThey wouldn't have to do that if the website wasn't - purposefully or through crappy engineering - making the data unnecessarily difficult to get and process by machines.
- intrasight 9y agoProducers should the make a rate-limited API - and even charge for it
- TeMPOraL 9y agoOr rate-limit the website itself - if you're dealing with scrappers.
- jamespo 9y agoThis isn't really about ad-blocking
- deleted 9y ago[deleted]
- philipodonnell 9y agoThe irony, I think, is that the standards that the current web is descended from GML/SGML/HTML were created explicitly for the purposes of making documents easier to read and understand by computers. Here now we are trying to make it as difficult as possible to read those same documents by computers.
- masklinn 9y ago> Here now we are trying to make it as difficult as possible to read those same documents by computers. They're as easy as ever to read, as that's necessary for the documents to be displayed to the end-user.
- philipodonnell 9y agoThe whole article and discussion is about blocking the ability for computers to read and understand the content in these documents?
- f4rker 9y ago>"In the beginning was the web, and it was good. Content came along. Some was good, some was cats. Then paid sites with sign-up. Then search engines. Then ads." Your history of the web is woefully incomplete, even for a joke.
- js2 9y agotl;dr: fucking JavaScript.
- coolspot 9y agoUsername checks out.
- walshemj 9y agoYes but from experience in the 1980's the punters realy did not like being nickel and dimed on pre internet based services Prestel, Tymnet/Telecom Gold and so on.
- bane 9y agoWithout any irony, at my workplace, we've spun up a docker swarm with headless browsers and a simply RESTful api that takes screen shots of websites and serves them back to the employee's browser so that our employees can surf the web without even the possibility of exposing themselves to malicious software garbage. It's not required, but it's an available corporate service. Another guy built a version that returns PDFs with working links so they can even click links. It's completely fucking stupid that we have to come up with this nonsense.
- imtringued 9y agoWhy not use VNC?