5 ms·
I'm developing a text-heavy site and philosophically I'm trying to view documents as just that... documents [1]. But I don't get good results for "rug pull".
by timdaub 5y ago
I'm developing a text-heavy site and philosophically I'm trying to view documents as just that... documents [1].
But I don't get good results for "rug pull".
- 1 https://rugpullindex.com https://rugpullindex.com
- marginalia_nu 5y agoYeah it's hosted by cloudflare. I'm currently IP-blocking them, as because they keep prompting my crawler with a captcha, presumably because it's made millions of requests from their CDN. Some rigmarole getting recognized as a good bot by the CDNs. I've submitted a request fairly recently, but haven't heard back from them yet. Like I would like to be on good terms with them, and other websites that block small independent crawlers. I can't blame them though, there's a lot of bad bots out there. But I'm doing my best not be part of the problem.
- petercooper 5y agoAha, I was going to ask how you were coping with CDNs like Cloudflare blocking bots. It's sad we've got to this point where basically only the established search engines are grandfathered in to be able to crawl sites.
- deleted 5y ago[deleted]
- Aachen 5y agoI wonder if google had to plead the same way or if already-big players are given a free advantage.
- BugsJustFindMe 5y ago> I'm developing a text-heavy site I looked at the source for your site's front page. That's not text-heavy; that's markup-heavy. I didn't bother looking at the rest of the pages because it appears to be yet another crypto market site.