3 ms·

It's all guesses, heuristics, and folk wisdom.

by ModernMech 1y ago

tptacek 1y ago
Which part? Be specific. They took the trouble to actually give a full prompt.
- ModernMech 1y ago
  I cite literally the entire thing, specifically lines 1-N. Is there any actual engineering advice in here? Do they cite any hard facts? Any actual science? Real statistics? No. The post can be summarized as: "Here are some things I tried; it seems to work for me", which is astrology-level support.
  - tptacek 1y ago
    Are we talking about the same comment? I'm talking about this one: https://news.ycombinator.com/item?id=44854637 https://news.ycombinator.com/item?id=44854637 ... which, rather than "spellcasting", appears to contain a fairly complete technical description of the system being constructed.
    - ModernMech 1y ago
      That's the spell. It's proffered soley because it "compiled and worked on the first try". This only flies as engineering advice in the software world because our version of engineering is "just try this out and see if it works. If it does, ship. Bugs be damned we'll patch them live!", which has resulted in... well I guess acceptance of this current push toward vibe coding.
      - tptacek 1y ago
        If you read something like "Modify PlayerServiceActor to subscribe to GameTickEvents and cache latestTick" as spell-casting and not as a pretty boring and straightforward description of a programming task, I think the disconnect between us is that you apparently perceive most of programming as magic. I do not.
        ModernMech 1y ago
        You're strawmanning. If invoking "Modify PlayerServiceActor to subscribe to GameTickEvents and cache latestTick" was enough to get good results, no one would be asking for advice on how to get good results.
        tptacek 1y ago
        You're not even keeping the argument straight. Which is it: is it "spellcasting" to get an LLM to generate a project plan like this, or "spellcasting" when it implements the project plan as written?
        ModernMech 1y ago
        This is pedantic, good day.
      - dnh44 1y ago
        I've actually connected Claude Code to XcodeBuildMCP and it will actually run the iOS app in a simulator and test the new code by scrolling and clicking on it. It's pretty cool to watch it find its mistake and fix it. Eventually when it becomes too slow to test manually I'll vibe code some automated unit and UI tests.
    - jodrellblank 1y ago
      There is some unquestionable "engineering advice": it needs access to the information/files so it can work on them, and keep a log of problems to fix. The rest is not so clear: - "I prime the context of the LLM with any extra rules or instructions I want it to follow." What are "extra" rules - extra to what? How could I tell if I am doing this or not doing this? - "I instruct the LLM in bullet point format for smaller edits. For larger edits I ask the llm to convert my prompt to markdown. Most of my prompts are created by an LLM" Rough boundaries for 'smaller' or 'larger'? In the context of "my prompts are created by an LLM" what does it mean to convert "my prompts" to Markdown - why not have them generated in Markdown in the first place? Why bring Markdown into this at all? Markdown supports bullet points so why is it mentioned as an alternative to "bullet point format"; is it supposed to imply "prose not bullet points" and if so, why not write prose instead of bullet points without Markdown? What difference does any of this make, either measurably or in vibes? Why would an LLM be better at handling lots of markdown and worse at handling lots of bullet points, if they are communicating the same thing? Why create prompts by LLM at all? Presumably you have to start with inputting what you want to happen, so why pass it through two LLMs instead of one? - "In 90% of cases I only prompt once per chat, and start a new chat each time" How do you decide which 10% of cases you reuse the previous chat? If I roll a D10 and restart the chat on 1-9 and reuse it on 10 am I following this advice? - "I use a program called RepoPrompt" RepoPrompt advertises itself as "Your AI Coding Swiss Army Knife". Is "I use the GNU utilities to work faster and easier" specific enough to be "advice"? > "They took the trouble to actually give a full prompt." Did they take the trouble? They say Claude Opus made the prompt. The important bit - how did they get Opus to write the prompt without them writing all those things in the meta-prompt - is missing. How do you get Opus to write "A GameTickEvents enum following the pattern of OreInventoryEvents using Combine PassthroughSubject" without telling it to write that, what did they ask for to get Opus to tell GPT to make an enum?
      - tptacek 1y ago
        I don't understand the argument here. People falsely claim that LLM-generated code is unverifiable because LLMs are stochastic (you didn't claim that, but others on HN have); that argument is risible because code is code and you can just read it and reject it if you're not certain what it does. Here, though, we're claiming that a generated prompt is so unverifiable that it qualifies as "spell-casting". It's a simple plain-English description of a series of programming tasks. If you can't evaluate this, you can't program.
  - dnh44 1y ago
    >Do they cite any hard facts? Any actual science? Real statistics? I guess I could do those things, but I'd rather work on my game. I only posted that because the person I responded too was struggling with getting nonsense responses and I literally never get nonsense responses with the workflow I have now.

It's all guesses, heuristics, and folk wisdom.

tptacek 1y ago

Which part? Be specific. They took the trouble to actually give a full prompt.

ModernMech 1y ago
I cite literally the entire thing, specifically lines 1-N. Is there any actual engineering advice in here? Do they cite any hard facts? Any actual science? Real statistics? No. The post can be summarized as: "Here are some things I tried; it seems to work for me", which is astrology-level support.
- tptacek 1y ago
  Are we talking about the same comment? I'm talking about this one: https://news.ycombinator.com/item?id=44854637 https://news.ycombinator.com/item?id=44854637 ... which, rather than "spellcasting", appears to contain a fairly complete technical description of the system being constructed.
  - ModernMech 1y ago
    That's the spell. It's proffered soley because it "compiled and worked on the first try". This only flies as engineering advice in the software world because our version of engineering is "just try this out and see if it works. If it does, ship. Bugs be damned we'll patch them live!", which has resulted in... well I guess acceptance of this current push toward vibe coding.
    - tptacek 1y ago
      If you read something like "Modify PlayerServiceActor to subscribe to GameTickEvents and cache latestTick" as spell-casting and not as a pretty boring and straightforward description of a programming task, I think the disconnect between us is that you apparently perceive most of programming as magic. I do not.
      - ModernMech 1y ago
        You're strawmanning. If invoking "Modify PlayerServiceActor to subscribe to GameTickEvents and cache latestTick" was enough to get good results, no one would be asking for advice on how to get good results.
        tptacek 1y ago
        You're not even keeping the argument straight. Which is it: is it "spellcasting" to get an LLM to generate a project plan like this, or "spellcasting" when it implements the project plan as written?
        ModernMech 1y ago
        This is pedantic, good day.
    - dnh44 1y ago
      I've actually connected Claude Code to XcodeBuildMCP and it will actually run the iOS app in a simulator and test the new code by scrolling and clicking on it. It's pretty cool to watch it find its mistake and fix it. Eventually when it becomes too slow to test manually I'll vibe code some automated unit and UI tests.
  - jodrellblank 1y ago
    There is some unquestionable "engineering advice": it needs access to the information/files so it can work on them, and keep a log of problems to fix. The rest is not so clear: - "I prime the context of the LLM with any extra rules or instructions I want it to follow." What are "extra" rules - extra to what? How could I tell if I am doing this or not doing this? - "I instruct the LLM in bullet point format for smaller edits. For larger edits I ask the llm to convert my prompt to markdown. Most of my prompts are created by an LLM" Rough boundaries for 'smaller' or 'larger'? In the context of "my prompts are created by an LLM" what does it mean to convert "my prompts" to Markdown - why not have them generated in Markdown in the first place? Why bring Markdown into this at all? Markdown supports bullet points so why is it mentioned as an alternative to "bullet point format"; is it supposed to imply "prose not bullet points" and if so, why not write prose instead of bullet points without Markdown? What difference does any of this make, either measurably or in vibes? Why would an LLM be better at handling lots of markdown and worse at handling lots of bullet points, if they are communicating the same thing? Why create prompts by LLM at all? Presumably you have to start with inputting what you want to happen, so why pass it through two LLMs instead of one? - "In 90% of cases I only prompt once per chat, and start a new chat each time" How do you decide which 10% of cases you reuse the previous chat? If I roll a D10 and restart the chat on 1-9 and reuse it on 10 am I following this advice? - "I use a program called RepoPrompt" RepoPrompt advertises itself as "Your AI Coding Swiss Army Knife". Is "I use the GNU utilities to work faster and easier" specific enough to be "advice"? > "They took the trouble to actually give a full prompt." Did they take the trouble? They say Claude Opus made the prompt. The important bit - how did they get Opus to write the prompt without them writing all those things in the meta-prompt - is missing. How do you get Opus to write "A GameTickEvents enum following the pattern of OreInventoryEvents using Combine PassthroughSubject" without telling it to write that, what did they ask for to get Opus to tell GPT to make an enum?
    - tptacek 1y ago
      I don't understand the argument here. People falsely claim that LLM-generated code is unverifiable because LLMs are stochastic (you didn't claim that, but others on HN have); that argument is risible because code is code and you can just read it and reject it if you're not certain what it does. Here, though, we're claiming that a generated prompt is so unverifiable that it qualifies as "spell-casting". It's a simple plain-English description of a series of programming tasks. If you can't evaluate this, you can't program.
- dnh44 1y ago
  >Do they cite any hard facts? Any actual science? Real statistics? I guess I could do those things, but I'd rather work on my game. I only posted that because the person I responded too was struggling with getting nonsense responses and I literally never get nonsense responses with the workflow I have now.