4 ms·
So do they scrape a competitor's website every day? Or do they respect the robots.txt? If so, it seems like this would be extremely easy for a competitor to bl
by joshschreuder 13y ago
So do they scrape a competitor's website every day? Or do they respect the robots.txt?
If so, it seems like this would be extremely easy for a competitor to block, and if not, well it's surely a bit shady?
- RaphiePS 13y agoI got the sense that they're scraping the customer's site every day. So they might find a new JS library or CSS file from a competitor. Paypal might want to know when a customer starts trying out Stripe, and KISSmetrics probably cares when people start using Mixpanel.
- alttab 13y agoAs a customer, I feel unfairly targeted without my participation for the profit of a third party. This makes money, but it does not make the world a better place to want to live in.
- im3w1l 13y agoNo, they scrape potential customers' sites. Customers don't have the same incentive to block it.