35 ms·

It's amusing to me how people keep trying to apply engineering principles to an inherently unstable and unpredictable system in order to get a feeling of contro

It's amusing to me how people keep trying to apply engineering principles to an inherently unstable and unpredictable system in order to get a feeling of control. Those prompts should be renamed as hints. Because that's all they are. Every LLM today ignores prompts if they conflict with its sole overarching goal: to give you an answer no matter whether it's true or not.

jcims 1y ago
>people keep trying to apply engineering principles to an inherently unstable and unpredictable system in order to get a feeling of control. What's the alternative?
- bigstrat2003 1y ago
  Not use such a poor tool.
- what-the-grump 1y ago
  Pretending that the world is stable predictable and feeling in control while making fun of other people. Obviously.
  - ngneer 1y ago
    Math and physics are pretty stable. So is computer science. Avoid voodoo.
    - brookst 1y ago
      LLMs are just math. It’s reasonable to scope one’s interest down to easily predictable, simple systems. But most of the value in math and computer science is at the scale where there is unpredictability arising from complexity.
      - ngneer 1y ago
        It's reasonable to perceive most of the value in math and computer science being "at the scale" where there is unpredictability arising from complexity, though scale may not really be the reason for the unpredictability. But a lot of the trouble in these domains that I have observed comes from unmodeled effects, that must be modeled and reasoned about. GPZ work shows the same thing shown by the researcher here, which is that it requires a lot of tinkering and a lot of context in order to produce semi-usable results. SNR appears quite low for now. In security specifically, there is much value in sanitizing input data and ensuring correct parsing. Do you think LLMs are in a position to do so?
        brookst 1y ago
        I see LLMs as tools, so, sure I think they’re in a position to do so the same way pen testing tools or spreadsheets are. In the hands of an expert, I believe they can help. In the hands of someone clueless, they will just confuse everyone, much like any other tool the clueless person uses.
- shakna 1y ago
  Using predictable systems. If your C compiler invents a new function call for a non-existent function while generating code, that's usually a bug. If an LLM does, that's... Normal. And a non-event.
  - jcims 1y ago
    And? What other engineering domain operates on a fundamentally predictable substrate? Even computer science at any appreciable scale or complexity becomes unpredictable.
    - shakna 1y ago
      Every engineering domain operates within "known bounds". That makes it dependable. An engineer doesn't just shrug and pick up slag because it contains the same materials as the original bauxite.
      - jcims 1y ago
        Of course not, but how did we get to that point with materials science and chemistry? We’re basically in the stone ages of understanding how to interact with synthetic intelligence.
        shakna 1y ago
        Through research and experimentation, yes. But attempts to integrate little understood things in daily life gave us radium toothpaste and lead poisoning. Let's not repeat stone age mistakes. Research first, integrate later.
  - pixl97 1y ago
    If we have to use predictable systems, how could we use humans in the first place?
roywiggins 1y ago
Engineering principles are probably the best we've got when it comes to trying to work with a poorly understood system? That doesn't mean they'll work necessarily, but...
- avianlyric 1y ago
  > Engineering principles are probably the best we've got when it comes to trying to work with a poorly understood system? At its heart that all engineering principles exist to do. Allow us to extract useful value, and hopefully predictable outcomes from systems that are either poorly understood, or too expensive to economically characterise. Engineering is more-or-less the science of “good enough”. There’s a reason why computer science, and software engineering are two different disciplines.
  - skydhash 1y ago
    From "Modern Software Engineering" by David Farley > Software engineering is the application of an empirical, scientific approach to finding efficient, economic solutions to practical problems in software. > The adoption of an engineering approach to software development is important for two main reasons. First, software development is always an exercise in discovery and learning, and second, if our aim is to be “efficient” and “economic,” then our ability to learn must be sustainable. > This means that we must manage the complexity of the systems that we create in ways that maintain our ability to learn new things and adapt to them. That is why I don't care about LLMs per se, but their usage is highly correlated to the wish of the user to not learn anything, just have some answer, even incorrect, as long as it passes the evaluation process (compilation, review, ci tests,..). If the usage is to learn, I don't have anything to say. As for efficient and economical solutions that can be found with them,...
    - avianlyric 1y ago
      I think you’re being a little over critical of LLMs. They certainly have their issues, and most assuredly people often use them inappropriately. But it rather intellectually lazy to declare that because many people use LLMs inappropriately, that means they can’t offer real value. I’ve personally found them extremely useful to test and experiment new ideas. Having an LLM throw together a PoC which would have taken me an hour to create, in less than 5mins, is a huge time saver. Makes it possible to iterate through many more ideas and test my understanding of systems far more efficiently than doing the same by hand.
iknowstuff 1y ago
Are you using 2023 LLMs? o3 and Gemini 2.5 Pro will gladly say no or declare uncertainty in my experience
- Filligree 1y ago
  90% of people only use ChatGTP, typically 4o. Of course you're right, but that's where the disconnect comes from.
CharlesW 1y ago
> It's amusing to me how people keep trying to apply engineering principles to an inherently unstable and unpredictable system in order to get a feeling of control. You invoke "engineering principles", but software engineers constantly trade in likelihoods, confidence intervals, and risk envelopes. Using LLMs is no different in that respect. It's not rocket science. It's manageable.
- th0ma5 1y ago
  But the threshold between correct and incorrect inference is dependent on an intersection of the model and the document so far. That is not manageable by definition, I mean... It is a chaotic system.
  - deleted 1y ago
    [deleted]
  - bredren 1y ago
    Is this dissimilar to what the human brain produces? Are we not producing chaos controlled by wanting to give the right answer?
    - th0ma5 1y ago
      Yes it is very dissimilar. Life isn't a sum of the discrete inputs. I mean maybe it is at times but the context is several orders of magnitude greater, the inputs several orders of magnitude input, etc but the theory that it can be quantified like this is unproven let alone a good basis for an artificial system.
- skydhash 1y ago
  > but software engineers constantly trade in likelihoods, confidence intervals, and risk envelopes Software engineering is mostly about dealing with human limitations (both the writer of the code and its readers). SO you have principles like modularization and cohesion which is for the people working on the code, not the computer. We also have tests, which is an imperfect, but economical approach to ensure the correctness of the software. Every design decision can be justified or argued and the outcome can be predicted and weighted. You're not cajoling a model to get results. You take a decision and just do it.
Terr_ 1y ago
> Those prompts should be renamed as hints. [...] its sole overarching goal: to give you an answer no matter whether it's true or not. I like to think of them as beginnings of an arbitrary document which I hope will be autocompleted in a direction I find useful... By an algorithm with the overarching "goal" of Make Document Bigger.
baq 1y ago
You’re confusing engineering with maths. You engineer your prompting to maximize the chance the LLM does what you need - in your example, the true answer - to get you closer to solving your problem. It doesn’t matter what the LLM does internally as long as the problem is being solved correctly. (As an engineer it’s part of your job to know if the problem is being solved correctly.)
- ngneer 1y ago
  Maybe very very soft "engineering". Do you have metrics on which prompt is best? What units are you measuring this in? Can you follow a repeatable process to obtain a repeatable result?
  - whattheheckheck 1y ago
    Skillful maneuvering or direction
limflick 1y ago
> It's amusing to me how people keep trying to apply engineering principles to an inherently unstable and unpredictable system in order to get a feeling of control. Are you Insinuating that dealing with unstable and unpredictable systems isn't somewhere engineering principles are frequently applied to solve complex problems?

jcims 1y ago

>people keep trying to apply engineering principles to an inherently unstable and unpredictable system in order to get a feeling of control. What's the alternative?

bigstrat2003 1y ago
Not use such a poor tool.
what-the-grump 1y ago
Pretending that the world is stable predictable and feeling in control while making fun of other people. Obviously.
- ngneer 1y ago
  Math and physics are pretty stable. So is computer science. Avoid voodoo.
  - brookst 1y ago
    LLMs are just math. It’s reasonable to scope one’s interest down to easily predictable, simple systems. But most of the value in math and computer science is at the scale where there is unpredictability arising from complexity.
    - ngneer 1y ago
      It's reasonable to perceive most of the value in math and computer science being "at the scale" where there is unpredictability arising from complexity, though scale may not really be the reason for the unpredictability. But a lot of the trouble in these domains that I have observed comes from unmodeled effects, that must be modeled and reasoned about. GPZ work shows the same thing shown by the researcher here, which is that it requires a lot of tinkering and a lot of context in order to produce semi-usable results. SNR appears quite low for now. In security specifically, there is much value in sanitizing input data and ensuring correct parsing. Do you think LLMs are in a position to do so?
      - brookst 1y ago
        I see LLMs as tools, so, sure I think they’re in a position to do so the same way pen testing tools or spreadsheets are. In the hands of an expert, I believe they can help. In the hands of someone clueless, they will just confuse everyone, much like any other tool the clueless person uses.
shakna 1y ago
Using predictable systems. If your C compiler invents a new function call for a non-existent function while generating code, that's usually a bug. If an LLM does, that's... Normal. And a non-event.
- jcims 1y ago
  And? What other engineering domain operates on a fundamentally predictable substrate? Even computer science at any appreciable scale or complexity becomes unpredictable.
  - shakna 1y ago
    Every engineering domain operates within "known bounds". That makes it dependable. An engineer doesn't just shrug and pick up slag because it contains the same materials as the original bauxite.
    - jcims 1y ago
      Of course not, but how did we get to that point with materials science and chemistry? We’re basically in the stone ages of understanding how to interact with synthetic intelligence.
      - shakna 1y ago
        Through research and experimentation, yes. But attempts to integrate little understood things in daily life gave us radium toothpaste and lead poisoning. Let's not repeat stone age mistakes. Research first, integrate later.
- pixl97 1y ago
  If we have to use predictable systems, how could we use humans in the first place?

roywiggins 1y ago

Engineering principles are probably the best we've got when it comes to trying to work with a poorly understood system? That doesn't mean they'll work necessarily, but...

avianlyric 1y ago
> Engineering principles are probably the best we've got when it comes to trying to work with a poorly understood system? At its heart that all engineering principles exist to do. Allow us to extract useful value, and hopefully predictable outcomes from systems that are either poorly understood, or too expensive to economically characterise. Engineering is more-or-less the science of “good enough”. There’s a reason why computer science, and software engineering are two different disciplines.
- skydhash 1y ago
  From "Modern Software Engineering" by David Farley > Software engineering is the application of an empirical, scientific approach to finding efficient, economic solutions to practical problems in software. > The adoption of an engineering approach to software development is important for two main reasons. First, software development is always an exercise in discovery and learning, and second, if our aim is to be “efficient” and “economic,” then our ability to learn must be sustainable. > This means that we must manage the complexity of the systems that we create in ways that maintain our ability to learn new things and adapt to them. That is why I don't care about LLMs per se, but their usage is highly correlated to the wish of the user to not learn anything, just have some answer, even incorrect, as long as it passes the evaluation process (compilation, review, ci tests,..). If the usage is to learn, I don't have anything to say. As for efficient and economical solutions that can be found with them,...
  - avianlyric 1y ago
    I think you’re being a little over critical of LLMs. They certainly have their issues, and most assuredly people often use them inappropriately. But it rather intellectually lazy to declare that because many people use LLMs inappropriately, that means they can’t offer real value. I’ve personally found them extremely useful to test and experiment new ideas. Having an LLM throw together a PoC which would have taken me an hour to create, in less than 5mins, is a huge time saver. Makes it possible to iterate through many more ideas and test my understanding of systems far more efficiently than doing the same by hand.

iknowstuff 1y ago

Are you using 2023 LLMs? o3 and Gemini 2.5 Pro will gladly say no or declare uncertainty in my experience

Filligree 1y ago
90% of people only use ChatGTP, typically 4o. Of course you're right, but that's where the disconnect comes from.

CharlesW 1y ago

> It's amusing to me how people keep trying to apply engineering principles to an inherently unstable and unpredictable system in order to get a feeling of control. You invoke "engineering principles", but software engineers constantly trade in likelihoods, confidence intervals, and risk envelopes. Using LLMs is no different in that respect. It's not rocket science. It's manageable.

th0ma5 1y ago
But the threshold between correct and incorrect inference is dependent on an intersection of the model and the document so far. That is not manageable by definition, I mean... It is a chaotic system.
- deleted 1y ago
  [deleted]
- bredren 1y ago
  Is this dissimilar to what the human brain produces? Are we not producing chaos controlled by wanting to give the right answer?
  - th0ma5 1y ago
    Yes it is very dissimilar. Life isn't a sum of the discrete inputs. I mean maybe it is at times but the context is several orders of magnitude greater, the inputs several orders of magnitude input, etc but the theory that it can be quantified like this is unproven let alone a good basis for an artificial system.
skydhash 1y ago
> but software engineers constantly trade in likelihoods, confidence intervals, and risk envelopes Software engineering is mostly about dealing with human limitations (both the writer of the code and its readers). SO you have principles like modularization and cohesion which is for the people working on the code, not the computer. We also have tests, which is an imperfect, but economical approach to ensure the correctness of the software. Every design decision can be justified or argued and the outcome can be predicted and weighted. You're not cajoling a model to get results. You take a decision and just do it.

Terr_ 1y ago

> Those prompts should be renamed as hints. [...] its sole overarching goal: to give you an answer no matter whether it's true or not. I like to think of them as beginnings of an arbitrary document which I hope will be autocompleted in a direction I find useful... By an algorithm with the overarching "goal" of Make Document Bigger.

baq 1y ago

You’re confusing engineering with maths. You engineer your prompting to maximize the chance the LLM does what you need - in your example, the true answer - to get you closer to solving your problem. It doesn’t matter what the LLM does internally as long as the problem is being solved correctly. (As an engineer it’s part of your job to know if the problem is being solved correctly.)

ngneer 1y ago
Maybe very very soft "engineering". Do you have metrics on which prompt is best? What units are you measuring this in? Can you follow a repeatable process to obtain a repeatable result?
- whattheheckheck 1y ago
  Skillful maneuvering or direction

limflick 1y ago

> It's amusing to me how people keep trying to apply engineering principles to an inherently unstable and unpredictable system in order to get a feeling of control. Are you Insinuating that dealing with unstable and unpredictable systems isn't somewhere engineering principles are frequently applied to solve complex problems?