5 ms·

It's like talking into a void. The issue with AI is that it is too subtle, too easy to get acceptable junk answers and too subtle for the majority to realize we

It's like talking into a void. The issue with AI is that it is too subtle, too easy to get acceptable junk answers and too subtle for the majority to realize we've made a universal crib sheet, software developers included, perhaps one of the worst populations due to their extremely weak communications as a community. To be repeatedly successful with AI, one has to exert mental effort to prompt AI effectively, but pretty much nobody is willing to even consider that. Attempts to discuss the language aspects of using an LLM get ridiculed as 'prompt engineer is not engineering' and dismissed, while that is exactly what it is: prompt engineering using a new software language, natural language, that the industry refuses to take seriously, but is in fact an extremely technical programming language so subtle few to none of you realize it, nor the power that is embodied by it within LLMs. They are incredible, they are subtle, to the degree the majority think they are fraud.

einrealist 1y ago
Isn't "Engineering" is based on predictability, on repeatability? LLMs are not very predictable. And that's not just true for the output. Each change to the model impacts how it parses and computes the input. For someone claiming to be a "Prompt Engineer", this cannot work. There are so many variables that are simply unknown to the casual user: training methods, the training set, biases, ... If I get the feeling I am creating good prompts for Gemini 2.5 Pro, the next version might render those prompts useless. And that might get even worse with dynamic, "self-improving" models. So when we talk about "Vibe coding", aren't we just doing "Vibe prompting", too?
- oceanplexian 1y ago
  > LLMs are not very predictable. And that's not just true for the output. If you run an open source model from the same seed on the same hardware they are completely deterministic. It will spit out the same answer every time. So it’s not an issue with the technology and there’s nothing stopping you from writing repeatable prompts and promoting techniques.
  - dimitri-vs 1y ago
    Realistically, how many people do you think have the time, skills and hardware required to do this?
  - mafuy 1y ago
    Who's saying that the model stays the same and the seed is not random for most of the companies that run AI? There is no drawback to randomness for them.
  - enragedcacti 1y ago
    Predictable does not necessarily follow from deterministic. Hash algorithms, for instance, are valuable specifically because they are both deterministic and unpredictable. Relying on model, seed, and hardware to get "repeatable" prompts essentially reduces an LLM to a very lossy natural language decompression algorithm. What other reason would someone have for asking the same question over and over and over again with the same input? If that's a problem you need solve then you need a database, not a deterministic LLM.
  - CoastalCoder 1y ago
    > If you run an open source model from the same seed on the same hardware they are completely deterministic. Are you sure of that? Parallel scatter/gather operations may still be at the mercy of scheduling variances, due to some forms of computer math not being associative.
    - atemerev 1y ago
      Sure. Just set the temperature to 0 in every model and see it become deterministic. Or use a fully deterministic PRNG like random123.
  - o11c 1y ago
    By "unpredictability", we mean that AIs will return completely different results if a single word is changed to a close synonym, or an adverb or prepositional phrase is moved to a semantically identical location, etc. Very often this simple change will move you from "get the correct answer 90% of the time" (about the best that AIs can do) to "get the correct answer <10% of the time". Whenever people talk about "prompt engineering", they're referring to randomly changing these kinds of things, in hopes of getting a query pattern where you get meaningful results 90% of the time.
- smohare 1y ago
  [dead]
20k 1y ago
The issue is that you have to put in more effort to solve a problem using AI, than to just solve it yourself If I have to do extensive subtle prompt engineering and use a lot of mental effort to solve my problem... I'll just solve the problem instead. Programming is a mental discipline - I don't need help typing, and if using an AI means putting in more brainpower, its fundamentally failed at improving my ability to engineer software
- cube2222 1y ago
  As with many productivity-boosting tools, it’s slower to begin with, but once you get used to it, and become “fluent”, it’s faster.
- milkshakes 1y ago
  > The issue is that you have to put in more effort to solve a problem using AI, than to just solve it yourself conceding that this may be the case, there are entire categories of problems that i am now able to approach that i have felt discouraged from in the past. even if the code is wrong (which, for the most part, it isn't), there is a value for me to have a team of over-eager puppies fearlessly leading me into the most uninviting problems, and somehow the mess they may or may not create makes solving the problem more accessible to me. even if i have to clean up almost every aspect of their work (i usually don't), the "get your feet wet" part is often the hardest part for me, even with a design and some prototyping. i don't have this problem at work really, but for personal projects it's been much more fun to work with the robots than always bouncing around my own head.
- handfuloflight 1y ago
  This overlooks a new category of developer who operates in natural language, not in syntax.
  - 20k 1y ago
    Natural language is inherently a bad programming language. No developer, even with the absolute best AI tools, can avoid understanding the code that AI generates for very long The only way to successfully use AI is to have sufficient skill to review the code it generates for correctness - which is a problem that is at least as skilful as simply writing the code
add-sub-mul-div 1y ago
If this nondeterministic software engineering had been invented first we'd have built statues of whoever gave us C.

einrealist 1y ago

Isn't "Engineering" is based on predictability, on repeatability? LLMs are not very predictable. And that's not just true for the output. Each change to the model impacts how it parses and computes the input. For someone claiming to be a "Prompt Engineer", this cannot work. There are so many variables that are simply unknown to the casual user: training methods, the training set, biases, ... If I get the feeling I am creating good prompts for Gemini 2.5 Pro, the next version might render those prompts useless. And that might get even worse with dynamic, "self-improving" models. So when we talk about "Vibe coding", aren't we just doing "Vibe prompting", too?

oceanplexian 1y ago
> LLMs are not very predictable. And that's not just true for the output. If you run an open source model from the same seed on the same hardware they are completely deterministic. It will spit out the same answer every time. So it’s not an issue with the technology and there’s nothing stopping you from writing repeatable prompts and promoting techniques.
- dimitri-vs 1y ago
  Realistically, how many people do you think have the time, skills and hardware required to do this?
- mafuy 1y ago
  Who's saying that the model stays the same and the seed is not random for most of the companies that run AI? There is no drawback to randomness for them.
- enragedcacti 1y ago
  Predictable does not necessarily follow from deterministic. Hash algorithms, for instance, are valuable specifically because they are both deterministic and unpredictable. Relying on model, seed, and hardware to get "repeatable" prompts essentially reduces an LLM to a very lossy natural language decompression algorithm. What other reason would someone have for asking the same question over and over and over again with the same input? If that's a problem you need solve then you need a database, not a deterministic LLM.
- CoastalCoder 1y ago
  > If you run an open source model from the same seed on the same hardware they are completely deterministic. Are you sure of that? Parallel scatter/gather operations may still be at the mercy of scheduling variances, due to some forms of computer math not being associative.
  - atemerev 1y ago
    Sure. Just set the temperature to 0 in every model and see it become deterministic. Or use a fully deterministic PRNG like random123.
- o11c 1y ago
  By "unpredictability", we mean that AIs will return completely different results if a single word is changed to a close synonym, or an adverb or prepositional phrase is moved to a semantically identical location, etc. Very often this simple change will move you from "get the correct answer 90% of the time" (about the best that AIs can do) to "get the correct answer <10% of the time". Whenever people talk about "prompt engineering", they're referring to randomly changing these kinds of things, in hopes of getting a query pattern where you get meaningful results 90% of the time.
smohare 1y ago
[dead]

20k 1y ago

The issue is that you have to put in more effort to solve a problem using AI, than to just solve it yourself If I have to do extensive subtle prompt engineering and use a lot of mental effort to solve my problem... I'll just solve the problem instead. Programming is a mental discipline - I don't need help typing, and if using an AI means putting in more brainpower, its fundamentally failed at improving my ability to engineer software

cube2222 1y ago
As with many productivity-boosting tools, it’s slower to begin with, but once you get used to it, and become “fluent”, it’s faster.
milkshakes 1y ago
> The issue is that you have to put in more effort to solve a problem using AI, than to just solve it yourself conceding that this may be the case, there are entire categories of problems that i am now able to approach that i have felt discouraged from in the past. even if the code is wrong (which, for the most part, it isn't), there is a value for me to have a team of over-eager puppies fearlessly leading me into the most uninviting problems, and somehow the mess they may or may not create makes solving the problem more accessible to me. even if i have to clean up almost every aspect of their work (i usually don't), the "get your feet wet" part is often the hardest part for me, even with a design and some prototyping. i don't have this problem at work really, but for personal projects it's been much more fun to work with the robots than always bouncing around my own head.
handfuloflight 1y ago
This overlooks a new category of developer who operates in natural language, not in syntax.
- 20k 1y ago
  Natural language is inherently a bad programming language. No developer, even with the absolute best AI tools, can avoid understanding the code that AI generates for very long The only way to successfully use AI is to have sufficient skill to review the code it generates for correctness - which is a problem that is at least as skilful as simply writing the code

add-sub-mul-div 1y ago

If this nondeterministic software engineering had been invented first we'd have built statues of whoever gave us C.