5 ms·

That's funny, for a few years I worked for an organization that did the front end crawling / brutalizing of most of the major retailers for their data. Afaik mo

That's funny, for a few years I worked for an organization that did the front end crawling / brutalizing of most of the major retailers for their data. Afaik most still do not intentionally provide any data publicly and many spend extravagant amounts on anti-crawling tech to prevent it, resulting in an expensive arms race over ostensibly public data.

beat 7y ago
Yeah, it's just dumb, pseudo-paranoid thinking. Crawlers are going to get it anyway, and it's publicly available, so why not make it easier for everyone? And like I said, our biggest customer was our own company. It saved an incredible amount of hassle.
marcosdumay 7y ago
I always wondered what those companies think they'll gain by hiding the products they sell.
gregmac 7y ago
If you put something on a public site it can be harvested, and there's really no stopping that. It's like video DRM: you can spend billions on the tech, but ultimately, if nothing else, anyone can still aim any camera at the screen and make a copy. There's only two reasons I can think of that companies would bother with this: 1) they are ignorant or delusional about it and think they can beat the crawler (which could be a third party making money by convincing them they can do it) 2) It forces the crawler to spend more money chasing their blocking I'm not sure what the benefit of 2 would be, though, unless you could make them spend so much money that their entire operation is unprofitable. Aside, as a developer, it might actually be entertaining for a while to work on (2) - constantly changing strategy to defeat scrapers - but I can also see getting burned out pretty quickly.
- leggomylibro 7y ago
  >Aside, as a developer, it might actually be entertaining for a while to work on (2) - constantly changing strategy to defeat scrapers - but I can also see getting burned out pretty quickly. Well if you replace 'scrapers' with 'advertisements', there are plenty of worthwhile OSS projects that might welcome help :) Making that sort of business unprofitable might not be a super realistic goal, though.

beat 7y ago

Yeah, it's just dumb, pseudo-paranoid thinking. Crawlers are going to get it anyway, and it's publicly available, so why not make it easier for everyone? And like I said, our biggest customer was our own company. It saved an incredible amount of hassle.

marcosdumay 7y ago

I always wondered what those companies think they'll gain by hiding the products they sell.

gregmac 7y ago

If you put something on a public site it can be harvested, and there's really no stopping that. It's like video DRM: you can spend billions on the tech, but ultimately, if nothing else, anyone can still aim any camera at the screen and make a copy. There's only two reasons I can think of that companies would bother with this: 1) they are ignorant or delusional about it and think they can beat the crawler (which could be a third party making money by convincing them they can do it) 2) It forces the crawler to spend more money chasing their blocking I'm not sure what the benefit of 2 would be, though, unless you could make them spend so much money that their entire operation is unprofitable. Aside, as a developer, it might actually be entertaining for a while to work on (2) - constantly changing strategy to defeat scrapers - but I can also see getting burned out pretty quickly.

leggomylibro 7y ago
>Aside, as a developer, it might actually be entertaining for a while to work on (2) - constantly changing strategy to defeat scrapers - but I can also see getting burned out pretty quickly. Well if you replace 'scrapers' with 'advertisements', there are plenty of worthwhile OSS projects that might welcome help :) Making that sort of business unprofitable might not be a super realistic goal, though.