3 ms·

Fetch-MCP: Playwright-Based MCP Server with Batch URL Fetching Support

64 points by Sulfide6416 2y ago

Sulfide6416 2y ago
Fetch-MCP is a MCP server built on Playwright, designed for efficient web page content fetching. It excels at retrieving content from both static and dynamic websites, leveraging Playwright's powerful headless browser capabilities. Key features include `fetch_url` for single page retrieval and `fetch_urls` for high-performance batch fetching of multiple URLs in parallel. Fetch-MCP intelligently extracts main content, supports Markdown conversion, and is easily configurable, making it an ideal tool for developers needing robust and scalable web scraping capabilities.
chazeon 2y ago
What's MCP?
- pizza 2y ago
  Model Context Protocol
  - dSebastien 2y ago
    I shared some notes about it here. Well worth exploring right now: https://notes.dsebastien.net/30+Areas/33+Permanent+notes/33.02+Content/Model+Context+Protocol+(MCP) https://notes.dsebastien.net/30+Areas/33+Permanent+notes/33....
    - hi_hi 2y ago
      Thanks for this. I'm not familiar with MCP, but having (briefly) read your link it appears to enable a use case I've been expecting where a chat window could replace the entire website experience (probably better suited to larger enterprise style websites) to provide tailored information for a company/product. Would you know if it's possible to use this approach to constrain an LLM to only a specific context of information (For example, on the Microsoft site, any question related to CRMs would answer with information about Dynamics but never Salesforce)?
- DogRunner 2y ago
  A simple explanation can be seen here: https://www.youtube.com/watch?v=7j_NE6Pjv-E https://www.youtube.com/watch?v=7j_NE6Pjv-E
andrethegiant 2y ago
Check out https://pure.md https://pure.md for a REST API version of this
wejick 2y ago
Is there any example how an agent can interact with MCP? I imagine it will replace / complement Tools interface.
- tuananh 2y ago
  it can be either stdio or SSE.
tomjen3 2y ago
Cool, but playwright doesn’t use your cookies. Increasingly I want to stop spending time on twitter, but it’s also where the AI news drops first - and I can’t just scrape the data without being logged in. If there was a way to have the ai go ahead and gather the data for me, that would be great.
- yonl 2y ago
  I would agree to this point as well. Speaking of implementation, i don’t mind if a browser extension forward cookies from my browser to the automation (privacy and security is an issue of course, and i’d ideally want the cookies to not leave my device, but personally i’m okay with some trade off).
- omneity 2y ago
  This is something I am building. Herd[0] gives you a puppeteer-like API over your own browser, in effect allowing you to use your session seamlessly for automation and data extraction (and avoid bot detection as a nice side effect) 0: https://herd.garden https://herd.garden
- aschobel 2y ago
  Playwright can actually use your existing browser cookies if you connect it through Chrome's debugging protocol. Launch Chrome with the flag: --remote-debugging-port=9222 Then connect via CDP in Playwright like this: const browser = await chromium.connectOverCDP('http://localhost:9222'); http://localhost:9222');
- dd36 2y ago
  Can’t you just have it login?

64 points by Sulfide6416 2y ago

Fetch-MCP is a MCP server built on Playwright, designed for efficient web page content fetching. It excels at retrieving content from both static and dynamic websites, leveraging Playwright's powerful headless browser capabilities. Key features include `fetch_url` for single page retrieval and `fetch_urls` for high-performance batch fetching of multiple URLs in parallel. Fetch-MCP intelligently extracts main content, supports Markdown conversion, and is easily configurable, making it an ideal tool for developers needing robust and scalable web scraping capabilities.

chazeon 2y ago

What's MCP?

pizza 2y ago
Model Context Protocol
- dSebastien 2y ago
  I shared some notes about it here. Well worth exploring right now: https://notes.dsebastien.net/30+Areas/33+Permanent+notes/33.02+Content/Model+Context+Protocol+(MCP) https://notes.dsebastien.net/30+Areas/33+Permanent+notes/33....
  - hi_hi 2y ago
    Thanks for this. I'm not familiar with MCP, but having (briefly) read your link it appears to enable a use case I've been expecting where a chat window could replace the entire website experience (probably better suited to larger enterprise style websites) to provide tailored information for a company/product. Would you know if it's possible to use this approach to constrain an LLM to only a specific context of information (For example, on the Microsoft site, any question related to CRMs would answer with information about Dynamics but never Salesforce)?
DogRunner 2y ago
A simple explanation can be seen here: https://www.youtube.com/watch?v=7j_NE6Pjv-E https://www.youtube.com/watch?v=7j_NE6Pjv-E

andrethegiant 2y ago

Check out https://pure.md https://pure.md for a REST API version of this

wejick 2y ago

Is there any example how an agent can interact with MCP? I imagine it will replace / complement Tools interface.

tuananh 2y ago
it can be either stdio or SSE.

tomjen3 2y ago

Cool, but playwright doesn’t use your cookies. Increasingly I want to stop spending time on twitter, but it’s also where the AI news drops first - and I can’t just scrape the data without being logged in. If there was a way to have the ai go ahead and gather the data for me, that would be great.

yonl 2y ago
I would agree to this point as well. Speaking of implementation, i don’t mind if a browser extension forward cookies from my browser to the automation (privacy and security is an issue of course, and i’d ideally want the cookies to not leave my device, but personally i’m okay with some trade off).
omneity 2y ago
This is something I am building. Herd[0] gives you a puppeteer-like API over your own browser, in effect allowing you to use your session seamlessly for automation and data extraction (and avoid bot detection as a nice side effect) 0: https://herd.garden https://herd.garden
aschobel 2y ago
Playwright can actually use your existing browser cookies if you connect it through Chrome's debugging protocol. Launch Chrome with the flag: --remote-debugging-port=9222 Then connect via CDP in Playwright like this: const browser = await chromium.connectOverCDP('http://localhost:9222'); http://localhost:9222');
dd36 2y ago
Can’t you just have it login?