3 ms·

Hmm yeah but I think at some point ad-hoc code becomes a signal that something is wrong. eg. If your LLM is continuously writing python to join customers to ord

by firasd 2mo ago

Hmm yeah but I think at some point ad-hoc code becomes a signal that something is wrong. eg. If your LLM is continuously writing python to join customers to orders at some point that's a signal that customers_aggregate('topspenders') needs to be a thing like a deterministic API call

drdexebtjl 2mo ago
At that point you would add a `your-service-cli list-customers --order-by=spent` command, which would also be useful to humans and scripts, as opposed to an MCP tool call, which is only ergonomic to models.
- firasd 2mo ago
  Right we can quibble about the wire protocol but the point remains that the arbitrary REPL becomes suboptimal as a solution when there’s not a deterministic endpoint. So ‘scripting’ is not really what we are looking for; we just want the final data output and scripting is fine when it’s a one off
  - drdexebtjl 2mo ago
    They’re not the same thing under a different wire protocol! An MCP tool call that returns the top N spenders puts the result in the agent’s context window. A CLI tool call on the other hand… The agent can pipe their results into other completely unrelated CLI tools. It can write a loop that calls multiple tools. These things are not possible with MCP but are trivial with CLIs.
    - simonw 2mo ago
      Anthropic proposed a way to programmatically chain MCPs together a while back, I'm not sure if it's been implemented much yet: https://www.anthropic.com/engineering/code-execution-with-mcp https://www.anthropic.com/engineering/code-execution-with-mc...
      - chmod775 2mo ago
        I don't even need to read that to know they're re-inventing PowerShell now. Edit: I read it. Yep. We have text interfaces refined by humans for decades and there's an endless sea of training data for them, but they imagine these amateur-hour homegrown solutions will ever outdo an agent with shell access?
  - deleted 2mo ago
    [deleted]
eddythompson80 2mo ago
Every arbitrary REPL or program written by an LLM to solve a problem can absolutely become a toolcall that does the same thing. Are you planning to provide hundreds, or thousands, of tools to cover all the possibilities an agent might consider AND have the foresight to include all of them? I think that’s a future plenty of people see or want. But simpler and composable Linux commands are far more flexible than pre-defined Windows or Mac “Settings App”. The barrier to entry for the former was knowledge and familiarity, while the latter optimizes for simpler user experience. Agents don’t have to worry about knowledge or familiarity the way users do. Ok, now you have customers_aggregate(‘topspenders’), but the prompt to the agent was “Get me the top spenders that are public companies and show me ones that increase their spend proportionally to their revenue increase and ones that don’t”. If the agent had access to the needed APIs (or CLIs) for your data and a finance API provider, it can generate a script that crunches down the top 10 or 10,000 for all that data. You could obviously come up with a toolcall that encapsulates that or rely on multiple toolcalls and on the LLM to fill in data correctly between them then do the math using inference. Edit: even if you look at that customers_aggregate(‘topspenders’) tool, you wouldn’t have needed that if the agent could execute a SQL query. It can also express anything the data can do. You obviously trade the control and correctness guarantees you can give for a `customers_aggregate(‘topspenders’)` vs a potentially wrong SQL query generated by the LLM with the infinite flexibility of a SQL query to be able to express far more than you could in a customers_aggregate call.
- fauigerzigerk 2mo ago
  I think MCP (or any kind of external API) serves a different purpose than ad-hoc code generation and scripting. In my view, the MCP should be part of the specification. It should expresses intended actions and/or restrictions while the code generated and executed by the LLM is an implementation detail. This distinction is completely obvious in some cases such as using an external payment gateway. But let's consider a more borderline case. Say your database is multi-tenant and every query absolutely must include a tenant_id. Now that doesn't necessarily mean you have to use MCP. It could be some command line tool that requires a tenant_id parameter. But what you can't do is ask the LLM politely to please include a where clause for the tenant_id in any and all SQL statements it decides to generate. You have to impose some things deterministically and MCP is one way to do that.

by firasd 2mo ago

drdexebtjl 2mo ago

At that point you would add a `your-service-cli list-customers --order-by=spent` command, which would also be useful to humans and scripts, as opposed to an MCP tool call, which is only ergonomic to models.

firasd 2mo ago
Right we can quibble about the wire protocol but the point remains that the arbitrary REPL becomes suboptimal as a solution when there’s not a deterministic endpoint. So ‘scripting’ is not really what we are looking for; we just want the final data output and scripting is fine when it’s a one off
- drdexebtjl 2mo ago
  They’re not the same thing under a different wire protocol! An MCP tool call that returns the top N spenders puts the result in the agent’s context window. A CLI tool call on the other hand… The agent can pipe their results into other completely unrelated CLI tools. It can write a loop that calls multiple tools. These things are not possible with MCP but are trivial with CLIs.
  - simonw 2mo ago
    Anthropic proposed a way to programmatically chain MCPs together a while back, I'm not sure if it's been implemented much yet: https://www.anthropic.com/engineering/code-execution-with-mcp https://www.anthropic.com/engineering/code-execution-with-mc...
    - chmod775 2mo ago
      I don't even need to read that to know they're re-inventing PowerShell now. Edit: I read it. Yep. We have text interfaces refined by humans for decades and there's an endless sea of training data for them, but they imagine these amateur-hour homegrown solutions will ever outdo an agent with shell access?
- deleted 2mo ago
  [deleted]

eddythompson80 2mo ago

Every arbitrary REPL or program written by an LLM to solve a problem can absolutely become a toolcall that does the same thing. Are you planning to provide hundreds, or thousands, of tools to cover all the possibilities an agent might consider AND have the foresight to include all of them? I think that’s a future plenty of people see or want. But simpler and composable Linux commands are far more flexible than pre-defined Windows or Mac “Settings App”. The barrier to entry for the former was knowledge and familiarity, while the latter optimizes for simpler user experience. Agents don’t have to worry about knowledge or familiarity the way users do. Ok, now you have customers_aggregate(‘topspenders’), but the prompt to the agent was “Get me the top spenders that are public companies and show me ones that increase their spend proportionally to their revenue increase and ones that don’t”. If the agent had access to the needed APIs (or CLIs) for your data and a finance API provider, it can generate a script that crunches down the top 10 or 10,000 for all that data. You could obviously come up with a toolcall that encapsulates that or rely on multiple toolcalls and on the LLM to fill in data correctly between them then do the math using inference. Edit: even if you look at that customers_aggregate(‘topspenders’) tool, you wouldn’t have needed that if the agent could execute a SQL query. It can also express anything the data can do. You obviously trade the control and correctness guarantees you can give for a `customers_aggregate(‘topspenders’)` vs a potentially wrong SQL query generated by the LLM with the infinite flexibility of a SQL query to be able to express far more than you could in a customers_aggregate call.

fauigerzigerk 2mo ago
I think MCP (or any kind of external API) serves a different purpose than ad-hoc code generation and scripting. In my view, the MCP should be part of the specification. It should expresses intended actions and/or restrictions while the code generated and executed by the LLM is an implementation detail. This distinction is completely obvious in some cases such as using an external payment gateway. But let's consider a more borderline case. Say your database is multi-tenant and every query absolutely must include a tenant_id. Now that doesn't necessarily mean you have to use MCP. It could be some command line tool that requires a tenant_id parameter. But what you can't do is ask the LLM politely to please include a where clause for the tenant_id in any and all SQL statements it decides to generate. You have to impose some things deterministically and MCP is one way to do that.