> Wow. That's starting to get really shady. Yes, there is a massive amount of crawling that happens in the service of financial forecasting. Satellite imagery,

> Wow. That's starting to get really shady. Yes, there is a massive amount of crawling that happens in the service of financial forecasting. Satellite imagery, drones, web pages, API endpoints, receipt data (free financial aggregators sell this), location data (free geolocation services sell this), purchase history (free email clients sell this), etc. This rabbit hole goes very deep. Some of it is actively sold by free services for "market research", some of it is collected from sources that don't bother with authentication or access control and make all of their data public. > Note to self: All things facing the web in ecommerce must be cryptographically randomized. :( The bigger issue is that sensitive information should be behind authentication. If you require authentication instead of just a "browsewrap" terms and conditions clause at the bottom of the page, that data becomes legally actionable if it's ever found. Otherwise you're relying on (at best) a robots.txt to do the enforcement for you. Many companies overlook this, even if they're told about it, because it doesn't compromise users or constitute a security vulnerability. So savvy firms take the data as a competitive advantage and use it for market research.