6 ms·

The real best case scenario is using LLMs to help build deterministic systems. Instead of asking an LLM to do some task that you know will be repeated, instead

The real best case scenario is using LLMs to help build deterministic systems. Instead of asking an LLM to do some task that you know will be repeated, instead ask the LLM to build a program (Python script or whatever) to do the task.

JCTheDenthog 4mo ago
Or just write it yourself?
- dukeyukey 4mo ago
  If you already know what the inputs/outputs are, why should you spend days or weeks of your life typing it out rather than giving it in a well-specified and tested form to an LLM to get it done a hundred times faster?
  - chasd00 4mo ago
    This is a truth that many are having a hard time accepting. Getting shoved into the light so fast is blinding.
    - anon7725 4mo ago
      We understand what you’re claiming, we just think that you are wrong.
  - dosisking 4mo ago
    Because the LLM version will have countless number of bugs and security holes, which means you will spend weeks or months of your life fixing them.
  - skydhash 4mo ago
    Because it’s rarely so black and white. Knowing the inputs and outputs is merely the first steps, you need to think about the transitions too as they have their own costs. Those costs don’t disappear and it’s truly naive to think they don’t matter. Take security issues, they may arise because what you thinks was the input is merely a subset of the true input range. And the extra possibilities lead to unforeseen behavior. A lot of programming is about ensuring that the input and the output are the sets defined in the specs. And the rest is that the transition/relation is the right tradeoffs of performance, correctness, and costs.
  - deleted 4mo ago
    [deleted]
  - JCTheDenthog 4mo ago
    >rather than giving it in a well-specified and tested form So, code?
  - xigoi 4mo ago
    The behavior of an LLM is not and cannot be “well-specified”.
    - Der_Einzige 4mo ago
      trumps voice wrong https://developers.openai.com/api/docs/guides/structured-outputs https://developers.openai.com/api/docs/guides/structured-out... https://github.com/guidance-ai/guidance https://github.com/guidance-ai/guidance https://github.com/noamgat/lm-format-enforcer https://github.com/noamgat/lm-format-enforcer https://github.com/mlc-ai/xgrammar https://github.com/mlc-ai/xgrammar https://github.com/dottxt-ai/outlines https://github.com/dottxt-ai/outlines
      - JCTheDenthog 4mo ago
        Libraries like Guidance guarantee that the output of an LLM will be syntactically correct (i.e. it will be valid JSON or whatever output format you are wanting). They do not, and fundamentally cannot guarantee that the data contained in them is actually correct, and cannot make the actual behavior of the LLM "well-specified". Or, as you put it, trumps voice wrong.
- whehhshs 4mo ago
  Because typing “code” takes time and significant amounts of it. We are slowly waking up to the fact, which was always true, that “coding” is just a fanciful preparatory task in order to appease the spirits properly so that we may invoke the spirit of what we are actually after: a live, running process that does useful things. Code is completely useless when separated from that fact. Typing it is a complete waste of time unless getting up close and personal with it will result in some kind of useful and actionable improvement in you or your understanding. Knowing when it does and when it does not have this property is a skill of its own.
  - quacked 4mo ago
    > Typing it is a complete waste of time unless getting up close and personal with it will result in some kind of useful and actionable improvement in you or your understanding. I believe this is the general belief about basically every human skill, that if you stop doing the technical fundamentals you get worse at understanding the activity. The question is whether coding is like sailing a square-rigged wooden ship, which became completely useless knowledge after the invention of the steam engine, or if it's like playing an instrument, which while technically unnecessary after the advent of MIDI and other tools, absolutely hurts your ability to arrange, compose and perform if the skill is neglected. For my money: I think the AI scenario is more like the latter, but "humans are worse at coding" isn't the consequence I see coming. I worry that in ten years we will be awash in software that's impossible to understand. I don't think that's happened in any human industry ever. Someone has always understood how the machines are built, even if they're very remote from the users of the machine.
    - taybin 4mo ago
      The sci-fi novel A Fire in the Deep starts with describing a Software Archeologist, who digs through millennia of strata of layers of indirection and I think we could end up needing that one day.
      - saltcured 4mo ago
        Do they end up determining that every weird piece of code they find must have been used for religious or ritualistic purposes?
dakiol 4mo ago
If it's a one-off script/program that doesn't require additional "domain knowledge", sure. But what if you need to give as context your whole backend repository because you need to take into account a few business rules? Why give anthropic/openai access to my "secret sauce" (e.g., company private repos)? In that case, it's way better to simply write the code yourself.
- daishi55 4mo ago
  I promise OpenAI is not going to steal your “secret sauce”
  - oblio 4mo ago
    Based on...?
    - daishi55 4mo ago
      Your sauce is not worth stealing to OpenAI. They have bigger fish to fry In fact - had there ever been a single case in the history of the internet where a service provider of any kind - hosting, storage, email, search, anything - has ever stolen intellectual property or trade secrets from a user and benefitted from it? Has that literally ever happened?
- mhss 4mo ago
  From all possible concerns, "giving access to anthropic/openai" to your "secret sauce" is the least important one for 99% of the companies out there. No, is not way better to simply write the code yourself. Most of the code is written faster and better with Claude Code or equivalent. Very niche code is better written by hand. Even then, you're probably better off nudging something like Claude Code in the direction you need it to go. There's nothing interesting about writing it yourself unless you're still learning to code (in which case is a learning exercise for you, not only about the outcome).
jacobgold 4mo ago
Making systems fully deterministic ignores the entire purpose of having agents involved. IMHO the best of both worlds option is agents working with deterministic CLIs. Where the agent does the reasoning (and text generation) but uses CLIs to carry out all of the actions (issuing refunds, unblocking accounts, or whatever). It's possible to get very reliable and consistent work out of agents when they're using well written prompts with well designed CLIs.
- bethekidyouwant 4mo ago
  How else would anyone do something like issue a refund if not through a programmatic interface?
  - jacobgold 4mo ago
    At some level everything an agent does is through a "programmatic interface" (tool calls). Some people might use skill-based scripts, MCPs, or some kind of raw access to a database. My point is that well designed CLIs are the optimal programmatic interface, for many reasons.
    - bethekidyouwant 4mo ago
      Sorry what other option is there? Is it going to create an API call from scratch every time after reading a page of documentation? Wait raw access to the database? That’s one of the options for issuing a refund?
      - deleted 4mo ago
        [deleted]
      - cflewis 4mo ago
        Yes, it can do. At Big Tech Company I Work At the LLM is quite happy to make raw API calls. If it thinks the data is big, then it'll write a Python tool to do it. The reason crafted backing CLIs are useful is you can guide the LLM towards stuff that is immediately useful rather than hoping the nondetermism can separate the wheat from the chaff. Take CI: is it interesting to know which tests passed? Maybe, but probably not. What is really interesting is what failed. Instead of having the LLM go out and talk directly to the CI system, write an intermediate CLI that filters out less actionable stuff by default, and have a flag that'll deliver the full dump if necessary. It's a skill to do this stuff, and it's a lot of hard won experience than something I think is easily teachable. You kind of have to feel out your model and how it "thinks" about solving problems. And then a new model version comes out and you have to learn it all again!
AlienRobot 4mo ago
The best case scenario of LLM is transforming input into output where both are languages and accuracy doesn't matter, e.g. "rewrite this poem in pirate speech." But that's not worth trillions of dollars...
alexpotato 4mo ago
100% this. I've already commented on other posts that having LLMs build deterministic and testable tools is the real unlock. Even for things like customer service, a LLM that analyzes customer support transcripts and then updates your call tree to better route people is a huge win.

JCTheDenthog 4mo ago

Or just write it yourself?

dukeyukey 4mo ago
If you already know what the inputs/outputs are, why should you spend days or weeks of your life typing it out rather than giving it in a well-specified and tested form to an LLM to get it done a hundred times faster?
- chasd00 4mo ago
  This is a truth that many are having a hard time accepting. Getting shoved into the light so fast is blinding.
  - anon7725 4mo ago
    We understand what you’re claiming, we just think that you are wrong.
- dosisking 4mo ago
  Because the LLM version will have countless number of bugs and security holes, which means you will spend weeks or months of your life fixing them.
- skydhash 4mo ago
  Because it’s rarely so black and white. Knowing the inputs and outputs is merely the first steps, you need to think about the transitions too as they have their own costs. Those costs don’t disappear and it’s truly naive to think they don’t matter. Take security issues, they may arise because what you thinks was the input is merely a subset of the true input range. And the extra possibilities lead to unforeseen behavior. A lot of programming is about ensuring that the input and the output are the sets defined in the specs. And the rest is that the transition/relation is the right tradeoffs of performance, correctness, and costs.
- deleted 4mo ago
  [deleted]
- JCTheDenthog 4mo ago
  >rather than giving it in a well-specified and tested form So, code?
- xigoi 4mo ago
  The behavior of an LLM is not and cannot be “well-specified”.
  - Der_Einzige 4mo ago
    trumps voice wrong https://developers.openai.com/api/docs/guides/structured-outputs https://developers.openai.com/api/docs/guides/structured-out... https://github.com/guidance-ai/guidance https://github.com/guidance-ai/guidance https://github.com/noamgat/lm-format-enforcer https://github.com/noamgat/lm-format-enforcer https://github.com/mlc-ai/xgrammar https://github.com/mlc-ai/xgrammar https://github.com/dottxt-ai/outlines https://github.com/dottxt-ai/outlines
    - JCTheDenthog 4mo ago
      Libraries like Guidance guarantee that the output of an LLM will be syntactically correct (i.e. it will be valid JSON or whatever output format you are wanting). They do not, and fundamentally cannot guarantee that the data contained in them is actually correct, and cannot make the actual behavior of the LLM "well-specified". Or, as you put it, trumps voice wrong.
whehhshs 4mo ago
Because typing “code” takes time and significant amounts of it. We are slowly waking up to the fact, which was always true, that “coding” is just a fanciful preparatory task in order to appease the spirits properly so that we may invoke the spirit of what we are actually after: a live, running process that does useful things. Code is completely useless when separated from that fact. Typing it is a complete waste of time unless getting up close and personal with it will result in some kind of useful and actionable improvement in you or your understanding. Knowing when it does and when it does not have this property is a skill of its own.
- quacked 4mo ago
  > Typing it is a complete waste of time unless getting up close and personal with it will result in some kind of useful and actionable improvement in you or your understanding. I believe this is the general belief about basically every human skill, that if you stop doing the technical fundamentals you get worse at understanding the activity. The question is whether coding is like sailing a square-rigged wooden ship, which became completely useless knowledge after the invention of the steam engine, or if it's like playing an instrument, which while technically unnecessary after the advent of MIDI and other tools, absolutely hurts your ability to arrange, compose and perform if the skill is neglected. For my money: I think the AI scenario is more like the latter, but "humans are worse at coding" isn't the consequence I see coming. I worry that in ten years we will be awash in software that's impossible to understand. I don't think that's happened in any human industry ever. Someone has always understood how the machines are built, even if they're very remote from the users of the machine.
  - taybin 4mo ago
    The sci-fi novel A Fire in the Deep starts with describing a Software Archeologist, who digs through millennia of strata of layers of indirection and I think we could end up needing that one day.
    - saltcured 4mo ago
      Do they end up determining that every weird piece of code they find must have been used for religious or ritualistic purposes?

dakiol 4mo ago

If it's a one-off script/program that doesn't require additional "domain knowledge", sure. But what if you need to give as context your whole backend repository because you need to take into account a few business rules? Why give anthropic/openai access to my "secret sauce" (e.g., company private repos)? In that case, it's way better to simply write the code yourself.

daishi55 4mo ago
I promise OpenAI is not going to steal your “secret sauce”
- oblio 4mo ago
  Based on...?
  - daishi55 4mo ago
    Your sauce is not worth stealing to OpenAI. They have bigger fish to fry In fact - had there ever been a single case in the history of the internet where a service provider of any kind - hosting, storage, email, search, anything - has ever stolen intellectual property or trade secrets from a user and benefitted from it? Has that literally ever happened?
mhss 4mo ago
From all possible concerns, "giving access to anthropic/openai" to your "secret sauce" is the least important one for 99% of the companies out there. No, is not way better to simply write the code yourself. Most of the code is written faster and better with Claude Code or equivalent. Very niche code is better written by hand. Even then, you're probably better off nudging something like Claude Code in the direction you need it to go. There's nothing interesting about writing it yourself unless you're still learning to code (in which case is a learning exercise for you, not only about the outcome).

jacobgold 4mo ago

Making systems fully deterministic ignores the entire purpose of having agents involved. IMHO the best of both worlds option is agents working with deterministic CLIs. Where the agent does the reasoning (and text generation) but uses CLIs to carry out all of the actions (issuing refunds, unblocking accounts, or whatever). It's possible to get very reliable and consistent work out of agents when they're using well written prompts with well designed CLIs.

bethekidyouwant 4mo ago
How else would anyone do something like issue a refund if not through a programmatic interface?
- jacobgold 4mo ago
  At some level everything an agent does is through a "programmatic interface" (tool calls). Some people might use skill-based scripts, MCPs, or some kind of raw access to a database. My point is that well designed CLIs are the optimal programmatic interface, for many reasons.
  - bethekidyouwant 4mo ago
    Sorry what other option is there? Is it going to create an API call from scratch every time after reading a page of documentation? Wait raw access to the database? That’s one of the options for issuing a refund?
    - deleted 4mo ago
      [deleted]
    - cflewis 4mo ago
      Yes, it can do. At Big Tech Company I Work At the LLM is quite happy to make raw API calls. If it thinks the data is big, then it'll write a Python tool to do it. The reason crafted backing CLIs are useful is you can guide the LLM towards stuff that is immediately useful rather than hoping the nondetermism can separate the wheat from the chaff. Take CI: is it interesting to know which tests passed? Maybe, but probably not. What is really interesting is what failed. Instead of having the LLM go out and talk directly to the CI system, write an intermediate CLI that filters out less actionable stuff by default, and have a flag that'll deliver the full dump if necessary. It's a skill to do this stuff, and it's a lot of hard won experience than something I think is easily teachable. You kind of have to feel out your model and how it "thinks" about solving problems. And then a new model version comes out and you have to learn it all again!

AlienRobot 4mo ago

The best case scenario of LLM is transforming input into output where both are languages and accuracy doesn't matter, e.g. "rewrite this poem in pirate speech." But that's not worth trillions of dollars...

alexpotato 4mo ago

100% this. I've already commented on other posts that having LLMs build deterministic and testable tools is the real unlock. Even for things like customer service, a LLM that analyzes customer support transcripts and then updates your call tree to better route people is a huge win.