4 ms·
Built an open-source web scraper for AI agents – seeking feedback
- ntbperst 1y agoAnyone else here building AI agents that need web data? So I built *AnyCrawl* specifically for AI use cases: Handles JS-heavy sites (React, Vue, etc.) 20x performance boost through multi-processing Support proxies (no more blocks!) Returns LLM-friendly structured data Open source (MIT) + optional SaaS
- mtmail 1y agoIt doesn't seem to respect robots.txt.
- ntbperst 1y agothe coming version will support.