I had no idea who or what Dataminr was before this, but I recently started a simple Twitter bot [0], and noticed they were crawling every single link I posted m

I had no idea who or what Dataminr was before this, but I recently started a simple Twitter bot [0], and noticed they were crawling every single link I posted multiple times, even though I explicitly disallow in my robots.txt. Their User-Agent just says to contact them, and there didn't seem to be a way to get them to stop, so I just banned their user agent from my servers. It is scary how little information there is about these random companies like Clearview AI or this that just gobble up everything online with no care. https://twitter.com/CatalogScanner https://twitter.com/CatalogScanner