3 ms·
Ask HN: Alternatives to Common Crawl?
- spronket_news 3y agoI'm trying to use common crawl for an ML project/search engine, but: - Requests to download even a small amount of data get rate ACLed (it says slow down/too many requests) - It seems like this is a known issue and that common crawl is no longer well maintained. https://groups.google.com/g/common-crawl/c/BvMGYUY-dro https://groups.google.com/g/common-crawl/c/BvMGYUY-dro Are there any alternatives for accessing a large amount of web crawl data? Thanks!
- ccgreg 3y agoPlease check out the blog post at: https://commoncrawl.org/blog/oct-nov-2023-performance-issues https://commoncrawl.org/blog/oct-nov-2023-performance-issues And the new status website at: https://status.commoncrawl.org/ https://status.commoncrawl.org/