21 ms·

I graded 36 popular MCP servers on agent usability. A third got a D or F

32 points by tengbyte 2mo ago

tengbyte 2mo ago
[flagged]
codeonline 2mo ago
Somewhat ironically the link to your results table is broken
- nemoniac 2mo ago
  Remove the ] from then end
- tengbyte 2mo ago
  Fixed — thanks for flagging (and to nemoniac for the workaround).
lolive 2mo ago
JSON and CSV [and a massive amount of custom-made spaghetti code] gave the impression that semantics was totally unnecessary in dataflows. Now that we live in a world where dataflows build themselves, we will need a description language that is much more robust. Probably generated from chatbot interactions, but formally defining how the consumer can discover content and mix them.
melembre 2mo ago
[flagged]
edifierxuhao 2mo ago
[flagged]
brookst 2mo ago
> well-documented big catalogs are possible; they're just rare They’re not that rare, it’s just that flat big catalogs don’t work well. The most common failure mode for MCP server design is mapping MCP tools to system APIs. This results in a tool to list files, a tool to rename a file, a tool to copy a file, etc. You can have 100 “tools”, you just need to define 10-12 areas (actual tools in the MCP model) and actions within each one. It creates a hierarchy that models can dive into when needed and avoid context clutter when that area is not needed. So instead of 15 tools for file operations you have a single “file” tool, with actions like list, copy, rename, etc. Single biggest arch requirement for any MCP server (well, any that has more than a handful of operations).
- tengbyte 2mo ago
  [flagged]
crefiz 2mo ago
Be warned, AI Slop ahead
arthiarumugam18 2mo ago
[flagged]
d1l 2mo ago
Bro this is just slop.
Hitton 2mo ago
This is stupid analysis. Complaining without nuance about "D004 - parameter has no description" suggests the article was written by an AI agent, because some parameters simply don't need any description. Prime example is "url" parameter explicitly mentioned in the article - unless there are multiple possible url parameters, adding the description just bloats context.
- tengbyte 2mo ago
  [flagged]
deleted 2mo ago
[deleted]
luciana1u 2mo ago
[flagged]
kittikitti 2mo ago
I'm not going to standardize an MCP server. I come from designing REST API's and the leniency of Python. In comparison to other frameworks, MCP is relatively new and there's valid skepticism because of Anthropic's proprietary schemes. It sounds like AI companies are gatekeeping MCP servers, based on definitions they made up, and can just have their goons go around purity testing them.
graphsmith 2mo ago
[dead]
sebazzz 2mo ago
Figma is notoriously bad, by having the agent write javascript to communicate with the mcp.

32 points by tengbyte 2mo ago

tengbyte 2mo ago

[flagged]

codeonline 2mo ago

Somewhat ironically the link to your results table is broken

nemoniac 2mo ago
Remove the ] from then end
tengbyte 2mo ago
Fixed — thanks for flagging (and to nemoniac for the workaround).

lolive 2mo ago

JSON and CSV [and a massive amount of custom-made spaghetti code] gave the impression that semantics was totally unnecessary in dataflows. Now that we live in a world where dataflows build themselves, we will need a description language that is much more robust. Probably generated from chatbot interactions, but formally defining how the consumer can discover content and mix them.

melembre 2mo ago

[flagged]

edifierxuhao 2mo ago

[flagged]

brookst 2mo ago

> well-documented big catalogs are possible; they're just rare They’re not that rare, it’s just that flat big catalogs don’t work well. The most common failure mode for MCP server design is mapping MCP tools to system APIs. This results in a tool to list files, a tool to rename a file, a tool to copy a file, etc. You can have 100 “tools”, you just need to define 10-12 areas (actual tools in the MCP model) and actions within each one. It creates a hierarchy that models can dive into when needed and avoid context clutter when that area is not needed. So instead of 15 tools for file operations you have a single “file” tool, with actions like list, copy, rename, etc. Single biggest arch requirement for any MCP server (well, any that has more than a handful of operations).

tengbyte 2mo ago
[flagged]

crefiz 2mo ago

Be warned, AI Slop ahead

arthiarumugam18 2mo ago

[flagged]

d1l 2mo ago

Bro this is just slop.

Hitton 2mo ago

This is stupid analysis. Complaining without nuance about "D004 - parameter has no description" suggests the article was written by an AI agent, because some parameters simply don't need any description. Prime example is "url" parameter explicitly mentioned in the article - unless there are multiple possible url parameters, adding the description just bloats context.

tengbyte 2mo ago
[flagged]

deleted 2mo ago

[deleted]

luciana1u 2mo ago

[flagged]

kittikitti 2mo ago

I'm not going to standardize an MCP server. I come from designing REST API's and the leniency of Python. In comparison to other frameworks, MCP is relatively new and there's valid skepticism because of Anthropic's proprietary schemes. It sounds like AI companies are gatekeeping MCP servers, based on definitions they made up, and can just have their goons go around purity testing them.

graphsmith 2mo ago

[dead]

sebazzz 2mo ago

Figma is notoriously bad, by having the agent write javascript to communicate with the mcp.