3 ms·
> though various sites will block it from time to time. The page itself describes a --ignore-robots-txt and customizing the user agent. Guess we can just all c
by MattSayar 2y ago
> though various sites will block it from time to time.
The page itself describes a --ignore-robots-txt and customizing the user agent. Guess we can just all copy OpenAI and continue to make SourceHut's life miserable /s
This is a cool tool, thanks for sharing
- knowaveragejoe 2y agoFor what it's worth, it's fetch under the hood. More akin to curl than automated scraping.