4 ms·

This looks interesting! I'll be keeping an eye on it. A few questions that come to mind: - Is there a way to run it locally/self-host? - Are there discovery e

This looks interesting! I'll be keeping an eye on it. A few questions that come to mind: - Is there a way to run it locally/self-host? - Are there discovery endpoints to list the available servers? - The 'Test & Eval' page is interesting to me, as I think unpredictability of results with multiple MCP prompts/sources interacting is generally a pretty big issue with MCP, so a good integrated eval system could be valuable. I see it's not launched yet, but it would be great to know more about how this will work.

wirehack 1y ago
> Is there a way to run it locally/self-host? Yes our github page has README for all MCP servers and clients. You can checkout https://github.com/Klavis-AI/klavis https://github.com/Klavis-AI/klavis. > Are there discovery endpoints to list the available servers? Yes. https://docs.klavis.ai/api-reference/mcp-server/get-all-servers https://docs.klavis.ai/api-reference/mcp-server/get-all-serv.... And we are adding more MCP servers. > Testing and Eval Yes it is early access now. If you are interested, shoot me an email at xiangkaiz@klavis.ai and we can talk more.
- danenania 1y ago
  Thanks! So the MCP stuff is open source but the directory/api is cloud only?
  - wirehack 1y ago
    Yes that is correct.
    - danenania 1y ago
      Cool, that's fine with me, but my suggestion fwiw would be to make this crystal clear upfront so you don't risk people feeling misled. I would also personally consider just open sourcing everything (or at least core features) as I think you'd still have plenty of customers who would prefer a hosted solution. In my experience, there is surprisingly little overlap between the people who prefer cloud vs. self-hosting, so having a full open source option doesn't cannibalize the product side as much as you might think... people who prefer cloud will mostly still prefer it even if they could self-host instead, and people who prefer self-hosting will generally just look for another project rather than using your cloud service. Just my unsolicited 2 cents.
      - wirehack 1y ago
        Got it, thank you for sharing this. Yes I have updated the post so that it is more clear that there are two versions. Thank you for the feedback.
mlenhard 1y ago
Agree on the unpredictability of results issue. Tool call selection is still sort of a black box. How do you know what variations of a prompt trigger a given tool to be called or how many tools is too many before you start seeing degradation issues because of the context window. If you are building a client and not a server the issue becomes even more pronounced. I even extracted the Claude electron source to see if I could figure out how they were doing it, but it's abstracted behind a network request. I'm guessing the system prompt handles tool call selection. PS: I released an open source evals package if you're curious. Still a WIP, but does the basics https://github.com/mclenhard/mcp-evals https://github.com/mclenhard/mcp-evals
- danenania 1y ago
  Thanks, I'll check it out. I'm working on a coding agent, and MCP has been a frequently requested feature, but yeah this issue has been my main hesitation. Getting even basic prompts that are designed to do one or two things to work reliably requires so much testing and iteration that I'm inherently pretty skeptical that "here are 10 community-contributed MCPs—choose the right one for the task" will have any hope of working reliably. Of course the benefits if it would work are very clear, so I'm keeping a close watch on it. Evals seem like a key piece of the puzzle, though you still might end up in combinatorial explosion territory by trying to test all the potential interactions with multiple MCPs. I could also see it getting very expensive to test this way.
  - mlenhard 1y ago
    I actually came across Plandex the other day. I haven't had the chance to play around with it yet, but it looked really cool. But agree that even basic prompts can be a struggle. You often need to name the tool in the prompt to get things to work reliably, but that's an awful user experience. Tool call descriptions play a pretty vital role, but most MCP servers are severely lacking in this regard. I hope this a result of everything being so new and the tooling and models will evolve to solve these issues over time.
    - danenania 1y ago

wirehack 1y ago

> Is there a way to run it locally/self-host? Yes our github page has README for all MCP servers and clients. You can checkout https://github.com/Klavis-AI/klavis https://github.com/Klavis-AI/klavis. > Are there discovery endpoints to list the available servers? Yes. https://docs.klavis.ai/api-reference/mcp-server/get-all-servers https://docs.klavis.ai/api-reference/mcp-server/get-all-serv.... And we are adding more MCP servers. > Testing and Eval Yes it is early access now. If you are interested, shoot me an email at xiangkaiz@klavis.ai and we can talk more.

danenania 1y ago
Thanks! So the MCP stuff is open source but the directory/api is cloud only?
- wirehack 1y ago
  Yes that is correct.
  - danenania 1y ago
    Cool, that's fine with me, but my suggestion fwiw would be to make this crystal clear upfront so you don't risk people feeling misled. I would also personally consider just open sourcing everything (or at least core features) as I think you'd still have plenty of customers who would prefer a hosted solution. In my experience, there is surprisingly little overlap between the people who prefer cloud vs. self-hosting, so having a full open source option doesn't cannibalize the product side as much as you might think... people who prefer cloud will mostly still prefer it even if they could self-host instead, and people who prefer self-hosting will generally just look for another project rather than using your cloud service. Just my unsolicited 2 cents.
    - wirehack 1y ago
      Got it, thank you for sharing this. Yes I have updated the post so that it is more clear that there are two versions. Thank you for the feedback.

mlenhard 1y ago

Agree on the unpredictability of results issue. Tool call selection is still sort of a black box. How do you know what variations of a prompt trigger a given tool to be called or how many tools is too many before you start seeing degradation issues because of the context window. If you are building a client and not a server the issue becomes even more pronounced. I even extracted the Claude electron source to see if I could figure out how they were doing it, but it's abstracted behind a network request. I'm guessing the system prompt handles tool call selection. PS: I released an open source evals package if you're curious. Still a WIP, but does the basics https://github.com/mclenhard/mcp-evals https://github.com/mclenhard/mcp-evals

danenania 1y ago
Thanks, I'll check it out. I'm working on a coding agent, and MCP has been a frequently requested feature, but yeah this issue has been my main hesitation. Getting even basic prompts that are designed to do one or two things to work reliably requires so much testing and iteration that I'm inherently pretty skeptical that "here are 10 community-contributed MCPs—choose the right one for the task" will have any hope of working reliably. Of course the benefits if it would work are very clear, so I'm keeping a close watch on it. Evals seem like a key piece of the puzzle, though you still might end up in combinatorial explosion territory by trying to test all the potential interactions with multiple MCPs. I could also see it getting very expensive to test this way.
- mlenhard 1y ago
  I actually came across Plandex the other day. I haven't had the chance to play around with it yet, but it looked really cool. But agree that even basic prompts can be a struggle. You often need to name the tool in the prompt to get things to work reliably, but that's an awful user experience. Tool call descriptions play a pretty vital role, but most MCP servers are severely lacking in this regard. I hope this a result of everything being so new and the tooling and models will evolve to solve these issues over time.
  - danenania 1y ago