3 ms·

You had a paid API, and people wanted the information for free.... Not unexpected I guess.

• mlevental 8y ago
  you've misunderstood: the site used something like algolia (which the site paid for) to index. the scrapers were hitting that service (which was costing the site) rather than parsing the xml (which already had everything).
• learc83 8y ago
  The paid API was just to handle the search of the company database (couldn't change that for political reasons). They weren't getting any information that they couldn't get through the XML.
• bo1024 8y ago
  After searching "algolia" mentioned below, I figured out the misunderstanding. The company was paying somebody else per search made on their web site. So every time a scraper called the website's search function, it cost the website money.