3 ms·

Using the fetch MCP tool(which is very basic) works fairly well for me, though various sites will block it from time to time. https://github.com/modelcontextpr

• MattSayar 2y ago
  > though various sites will block it from time to time. The page itself describes a --ignore-robots-txt and customizing the user agent. Guess we can just all copy OpenAI and continue to make SourceHut's life miserable /s This is a cool tool, thanks for sharing
  • knowaveragejoe 2y ago
    For what it's worth, it's fetch under the hood. More akin to curl than automated scraping.