4 ms·

The main thing to understand about the impact of AI tools: Somehow the more senior you are [in the field of use], the better results you get. You can run faste

by Swizec 9mo ago

The main thing to understand about the impact of AI tools: Somehow the more senior you are [in the field of use], the better results you get. You can run faster and get more done! If you're good, you get great results faster. If you're bad, you get bad results faster. You still gotta understand what you're doing. GeLLMan Amnesia is real.

simonw 9mo ago
Right: these things amplify existing skills. The more skill you have, the bigger the effect after it gets amplified.
- perrygeo 9mo ago
  I jumped into a new-to-me Typescript application and asked Claude to build a thing, in vague terms matching my own uncertainty and unfamiliarity. The result was similarly vague garbage. Three shots and I threw them all away. Then I watched a someone familiar with the codebase ask Claude to build the thing, in precise terms matching their expertise and understanding of the code. It worked flawlessly the first time. Neither of us "coded", but their skill with the underlying theory of the program allowed them to ask the right questions, infinitely more productive in this case. Skill and understanding matter now more than ever! LLMs are pushing us rapidly away from specialized technicians to theory builders.
  - steve_adams_86 8mo ago
    For sure, directing attention to valuable context and outlining problems to solve within it works way, way better than vague uncertainty. Good LLMing seems to be about isolating the right information and instructing it correctly from there. Both the context and the prompt make a tremendous difference. I've been finding recently that I can get significantly better results with fewer tokens by paying mind to this more often. I'm definitely a casual though. There are probably plenty of nuances and tricks I'm unaware of.
- keeda 9mo ago
  Interestingly, this observation holds even when you scale AI use up from individuals to organizations, only at that level it amplifies your organization's overal development trajectory. The DORA 2025 and the DX developer survey reports find that teams with strong quality control practices enjoy higher velocity, whereas teams with weak or no processes suffer elevated issues and outages. It makes sense considering that these practices could be thought of as "institutionalized skills."
mikkupikku 9mo ago
Agreed. How well you understand the problem domain determines the quality of your instructions a s feedback to the LLM, which in turn determines the quality of the results. This has been my experience, it works well for things I know well, and poorly for things I'm bad at. I've read a lot of people saying that they tried it on "hard problems" and it failed; I interpret this as the problem being hard not in absolute terms, but relative to the skill level of the user.
victrflow 9mo ago
Word.
SecretDreams 9mo ago
> Somehow the more senior you are [in the field of use], the better results you get. It's a K-type curve. People that know things will benefit greatly. Everyone else will probably get worse. I am especially worried about all young minds that are probably going to have significant gaps in their ability to learn and reason based on how much exposure they've had with AI to solve the problems for them.
9rx 9mo ago
> You still gotta understand what you're doing. Of course, but how do you begin to understand the "stochastic parrot"? Yesterday I used LLMs all day long and everything worked perfectly. Productivity was great and I was happy. I was ready to embrace the future. Now, today, no matter what I try, everything LLMs have produced has been a complete dumpster fire and waste of my time. Not even Opus will follow basic instructions. My day is practically over now and I haven't accomplished anything other than pointlessly fighting LLMs. Yesterday's productivity gains are now gone, I'm frustrated, exhausted, and wonder why I didn't just do it myself. This is a recurring theme for me. Every time I think I've finally cracked the code, next time it is like I'm back using an LLM for the first time in my life. What is the formal approach that finds consistency?
- acuozzo 9mo ago
  You're experiencing throttling. Use the API instead and pay per token. You also have to treat this as outsourcing labor to a savant with a very, very short memory, so: 1. Write every prompt like a government work contract in which you're required to select the lowest bidder, so put guardrails everywhere. Keep a text editor open with your work contract, edit the goal at the bottom, and then fire off your reply. 2. Instruct the model to keep a detailed log in a file and, after a context compaction, instruct it to read this again. 3. Use models from different companies to review one another's work. If you're using Opus-4.5 for code generation, then consider using GPT-5.2-Codex for review. 4. Build a mental model for which models are good at which tasks. Mine is: 3a. Mathematical Thinking (proofs, et al.): Gemini DeepThink 3b. Software Architectural Planning: GPT5-Pro (not 5.1 or 5.2) 3c. Web Search & Deep Research: Gemini 3-Pro 3d. Technical Writing: GPT-4.5 3e. Code Generation & Refactoring: Opus-4.5 3f. Image Generation: Nano Banana Pro
  - 9rx 9mo ago
    > You're experiencing throttling. Use the API instead and pay per token. That was using pay per token. > Write every prompt like a government work contract in which you're required to select the lowest bidder, so put guardrails everywhere. That is what I was doing yesterday. Worked fantastically. Today, I do the very same thing and... Nope. Can't even stick to the simplest instructions that have been perfectly fine in the past. > If you're using Opus-4.5 for code generation, then consider using GPT-5.2-Codex for review. As mentioned, I tried using Opus, but it didn't even get the point of producing anything worth reviewing. I've had great luck with it before, but not today. > Instruct the model to keep a detailed log in a file and, after a context compaction No chance of getting anywhere close to needing compaction today. I had to abort long before that. > Build a mental model for which models are good at which tasks. See, like I mentioned before, I thought I had this figured out, but now today it has all gone out the window.
tills13 8mo ago
Yeah. It's a force multiplier. And if you aren't careful, the force it multiplies can be dangerous or destructive.

by Swizec 9mo ago

simonw 9mo ago

Right: these things amplify existing skills. The more skill you have, the bigger the effect after it gets amplified.

perrygeo 9mo ago
I jumped into a new-to-me Typescript application and asked Claude to build a thing, in vague terms matching my own uncertainty and unfamiliarity. The result was similarly vague garbage. Three shots and I threw them all away. Then I watched a someone familiar with the codebase ask Claude to build the thing, in precise terms matching their expertise and understanding of the code. It worked flawlessly the first time. Neither of us "coded", but their skill with the underlying theory of the program allowed them to ask the right questions, infinitely more productive in this case. Skill and understanding matter now more than ever! LLMs are pushing us rapidly away from specialized technicians to theory builders.
- steve_adams_86 8mo ago
  For sure, directing attention to valuable context and outlining problems to solve within it works way, way better than vague uncertainty. Good LLMing seems to be about isolating the right information and instructing it correctly from there. Both the context and the prompt make a tremendous difference. I've been finding recently that I can get significantly better results with fewer tokens by paying mind to this more often. I'm definitely a casual though. There are probably plenty of nuances and tricks I'm unaware of.
keeda 9mo ago
Interestingly, this observation holds even when you scale AI use up from individuals to organizations, only at that level it amplifies your organization's overal development trajectory. The DORA 2025 and the DX developer survey reports find that teams with strong quality control practices enjoy higher velocity, whereas teams with weak or no processes suffer elevated issues and outages. It makes sense considering that these practices could be thought of as "institutionalized skills."

mikkupikku 9mo ago

Agreed. How well you understand the problem domain determines the quality of your instructions a s feedback to the LLM, which in turn determines the quality of the results. This has been my experience, it works well for things I know well, and poorly for things I'm bad at. I've read a lot of people saying that they tried it on "hard problems" and it failed; I interpret this as the problem being hard not in absolute terms, but relative to the skill level of the user.

victrflow 9mo ago

Word.

SecretDreams 9mo ago

> Somehow the more senior you are [in the field of use], the better results you get. It's a K-type curve. People that know things will benefit greatly. Everyone else will probably get worse. I am especially worried about all young minds that are probably going to have significant gaps in their ability to learn and reason based on how much exposure they've had with AI to solve the problems for them.

9rx 9mo ago

> You still gotta understand what you're doing. Of course, but how do you begin to understand the "stochastic parrot"? Yesterday I used LLMs all day long and everything worked perfectly. Productivity was great and I was happy. I was ready to embrace the future. Now, today, no matter what I try, everything LLMs have produced has been a complete dumpster fire and waste of my time. Not even Opus will follow basic instructions. My day is practically over now and I haven't accomplished anything other than pointlessly fighting LLMs. Yesterday's productivity gains are now gone, I'm frustrated, exhausted, and wonder why I didn't just do it myself. This is a recurring theme for me. Every time I think I've finally cracked the code, next time it is like I'm back using an LLM for the first time in my life. What is the formal approach that finds consistency?

acuozzo 9mo ago
You're experiencing throttling. Use the API instead and pay per token. You also have to treat this as outsourcing labor to a savant with a very, very short memory, so: 1. Write every prompt like a government work contract in which you're required to select the lowest bidder, so put guardrails everywhere. Keep a text editor open with your work contract, edit the goal at the bottom, and then fire off your reply. 2. Instruct the model to keep a detailed log in a file and, after a context compaction, instruct it to read this again. 3. Use models from different companies to review one another's work. If you're using Opus-4.5 for code generation, then consider using GPT-5.2-Codex for review. 4. Build a mental model for which models are good at which tasks. Mine is: 3a. Mathematical Thinking (proofs, et al.): Gemini DeepThink 3b. Software Architectural Planning: GPT5-Pro (not 5.1 or 5.2) 3c. Web Search & Deep Research: Gemini 3-Pro 3d. Technical Writing: GPT-4.5 3e. Code Generation & Refactoring: Opus-4.5 3f. Image Generation: Nano Banana Pro
- 9rx 9mo ago
  > You're experiencing throttling. Use the API instead and pay per token. That was using pay per token. > Write every prompt like a government work contract in which you're required to select the lowest bidder, so put guardrails everywhere. That is what I was doing yesterday. Worked fantastically. Today, I do the very same thing and... Nope. Can't even stick to the simplest instructions that have been perfectly fine in the past. > If you're using Opus-4.5 for code generation, then consider using GPT-5.2-Codex for review. As mentioned, I tried using Opus, but it didn't even get the point of producing anything worth reviewing. I've had great luck with it before, but not today. > Instruct the model to keep a detailed log in a file and, after a context compaction No chance of getting anywhere close to needing compaction today. I had to abort long before that. > Build a mental model for which models are good at which tasks. See, like I mentioned before, I thought I had this figured out, but now today it has all gone out the window.

tills13 8mo ago

Yeah. It's a force multiplier. And if you aren't careful, the force it multiplies can be dangerous or destructive.