3 ms·

I gotta say, having white blurry blobs of something in the background floating behind white/grey text maybe wasn't the best design-choice out there. None the l

by diggan 1y ago

I gotta say, having white blurry blobs of something in the background floating behind white/grey text maybe wasn't the best design-choice out there. None the less, I tried to find the actual APIs/service/software used for the "search" part, as I've found that to be the hardest to actually get right (at least for as-local-as-possible usage) for my own "Deep Research Agent". I've experimented with Brave's search API which worked OK, but seems pricey for agent usage. Currently experimenting with using my own (local) YaCy instance right now, which actually gives me higher quality artifacts at the end, as there are no rate-limits and the model can do hundreds of search calls without me worrying about the cost. But it isn't very quick at picking up some stuff like news and more, otherwise works OK too. What is the author doing here for the actual searching? Anyone else have any other ideas/approaches to this?

saqadri 1y ago
Haha, I didn't have control on the blog website, just the content. The readme and code is the ultimate source of truth (and easier to read):https://github.com/lastmile-ai/mcp-agent/blob/main/src/mcp_agent/workflows/deep_orchestrator/README.md https://github.com/lastmile-ai/mcp-agent/blob/main/src/mcp_a... So the core idea is the Deep Orchestrator is pretty unopinionated on what to use for searching, as long as it is exposed over MCP. I tried with a basic fetch server that's one of the reference MCP servers (with a single tool called `fetch`), and also tried with Brave. I think the folks at Jina wrote some really good stuff on the actual search part: https://jina.ai/news/a-practical-guide-to-implementing-deepsearch-deepresearch/ https://jina.ai/news/a-practical-guide-to-implementing-deeps... -- and how to do page/url ranking over the course of the flow. My recommendation would be to do all that in an MCP server itself. That keeps the "deep orchestrator" architecture fairly clean, and you can plug in increasingly sophisticated search techniques over time.
- jimmySixDOF 1y ago
  I'd be interested if you did any comparison testing to the langchain project which was, at least a month ago, the top open source approach https://huggingface.co/spaces/Ayanami0730/DeepResearch-Leaderboard https://huggingface.co/spaces/Ayanami0730/DeepResearch-Leade...
  - saqadri 1y ago
    Thanks for sharing this! We've reached out to the benchmark owners are are going to get our deep research agent benchmarked soon.
Zetaphor 1y ago
Self host an instance of SearXNG[1] either locally or on a remote server with a simple docker container and use its JSON API [2]. You have to enable the JSON API in the config manually [3]. [1] https://docs.searxng.org/admin/installation-docker.html#installation-container https://docs.searxng.org/admin/installation-docker.html#inst... [2] https://docs.searxng.org/dev/search_api.html https://docs.searxng.org/dev/search_api.html [3] https://github.com/searxng/searxng/discussions/3542 https://github.com/searxng/searxng/discussions/3542
- saqadri 1y ago
  Thanks for sharing, this looks great! Do they have an MCP server? It should be easy to wrap around their JSON API but I couldn't see MCP support in the repo/docs.
  - Zetaphor 1y ago
    Not that I'm aware of, but it's an extremely simple API. It's should be really easy to wrap into an MCP

by diggan 1y ago

saqadri 1y ago

Haha, I didn't have control on the blog website, just the content. The readme and code is the ultimate source of truth (and easier to read):https://github.com/lastmile-ai/mcp-agent/blob/main/src/mcp_agent/workflows/deep_orchestrator/README.md https://github.com/lastmile-ai/mcp-agent/blob/main/src/mcp_a... So the core idea is the Deep Orchestrator is pretty unopinionated on what to use for searching, as long as it is exposed over MCP. I tried with a basic fetch server that's one of the reference MCP servers (with a single tool called `fetch`), and also tried with Brave. I think the folks at Jina wrote some really good stuff on the actual search part: https://jina.ai/news/a-practical-guide-to-implementing-deepsearch-deepresearch/ https://jina.ai/news/a-practical-guide-to-implementing-deeps... -- and how to do page/url ranking over the course of the flow. My recommendation would be to do all that in an MCP server itself. That keeps the "deep orchestrator" architecture fairly clean, and you can plug in increasingly sophisticated search techniques over time.

jimmySixDOF 1y ago
I'd be interested if you did any comparison testing to the langchain project which was, at least a month ago, the top open source approach https://huggingface.co/spaces/Ayanami0730/DeepResearch-Leaderboard https://huggingface.co/spaces/Ayanami0730/DeepResearch-Leade...
- saqadri 1y ago
  Thanks for sharing this! We've reached out to the benchmark owners are are going to get our deep research agent benchmarked soon.

Zetaphor 1y ago

Self host an instance of SearXNG[1] either locally or on a remote server with a simple docker container and use its JSON API [2]. You have to enable the JSON API in the config manually [3]. [1] https://docs.searxng.org/admin/installation-docker.html#installation-container https://docs.searxng.org/admin/installation-docker.html#inst... [2] https://docs.searxng.org/dev/search_api.html https://docs.searxng.org/dev/search_api.html [3] https://github.com/searxng/searxng/discussions/3542 https://github.com/searxng/searxng/discussions/3542

saqadri 1y ago
Thanks for sharing, this looks great! Do they have an MCP server? It should be easy to wrap around their JSON API but I couldn't see MCP support in the repo/docs.
- Zetaphor 1y ago
  Not that I'm aware of, but it's an extremely simple API. It's should be really easy to wrap into an MCP