5 ms·
This article sets a ludicrous bar ("10x"), then documents the author's own attempt over some indeterminate time to clear that bar. As a result, the author has
by tptacek 1y ago
This article sets a ludicrous bar ("10x"), then documents the author's own attempt over some indeterminate time to clear that bar. As a result, the author has classified all the AI-supporters in the industry into three categories: (1) people who are wrong in good faith, (2) people who are selling AI tools, and (3) evil bosses trying to find leverage in programmer anxiety.
That aside: I still think complaining about "hallucination" is a pretty big "tell".
- coltonv 1y agoHi there! I appreciate your comment, and I remember reading your article about AI and some of the counterarguments to it helped me get over the imposter syndrome I was feeling. To be clear, I did not classify "all the AI-supporters" as being in those three categories, I specifically said the people posting that they are getting 10x improvements thanks to AI. Can you tell me about what you've done to no longer have any hallucinations? I notice them particularly in a language like Terraform, the LLMs add properties that do not exist. They are less common in languages like Javascript but still happen when you import libraries that are less common (e.g. DrizzleORM).
- tptacek 1y agoCan you help me understand which articles you're referring to? A link to the biggest "AI made me a 10x developer" article you've read would certainly clear this up.
- coltonv 1y agoMy goal here was not to publicly call out any specific individual or article. I don't want to make enemies and I don't want to be cast as dunking on someone. I get that that opens me up to criticism that I'm fighting a strawman, I accept that. Your article does not specifically say 10x, but it does say this: > Kids today don’t just use agents; they use asynchronous agents. They wake up, free-associate 13 different things for their LLMs to work on, make coffee, fill out a TPS report, drive to the Mars Cheese Castle, and then check their notifications. They’ve got 13 PRs to review. Three get tossed and re-prompted. Five of them get the same feedback a junior dev gets. And five get merged. > “I’m sipping rocket fuel right now,” a friend tells me. “The folks on my team who aren’t embracing AI? It’s like they’re standing still.” He’s not bullshitting me. He doesn’t work in SFBA. He’s got no reason to lie. That's not quantifying it specifically enough to say "10x", but it is saying no uncertain terms that AI engineers are moving fast and everyone else is standing still by comparison. Your article was indeed one of the ones I specifically wanted to respond to as the language directly contributed to the anxiety I described here. It made me worry that maybe I was standing still. To me, the engineer you described as sipping rocket fuel is an example both of the "degrees of separation" concept (it confuses me you are pointing to a third party and saying they are trustworthy, why not simply describe your workflow?), and the idea that a quick burst of productivity can feel huge but it just doesn't scale in my experience. Again, can you tell me about what you've done to no longer have any hallucinations? I'm fully open to learning here. As I stated in the article, I did my best to give full AI agent coding a try, I'm open to being proven wrong and adjusting my approach.
- tptacek 1y agoWait, now you're saying I set the 10x bar? No, I did not.
- coltonv 1y ago> Wait, now you're saying I set the 10x bar? No, I did not. I distinctly did not say that. I said your article was one of the ones that made me feel anxious. And it's one of the ones that spurred me to write this article. I demonstrated how your language implies a massive productivity boost from AI. Does it not? Is this not the entire point of what you wrote? That engineers who aren't using AI are crazy (literally the title) because they are missing out on all this "rocket fuel" productivity? The difference between rocket fuel and standing still has to be a pretty big improvement. The points I make here still apply, there is not some secret well of super-productivity sitting out in the open that luddites are just too grumpy to pick up and use. Those who feel they have gotten massive productivity boosts are being tricked by occasional, rare boosts in productivity. You said you solved hallucinations, could you share some of how you did that?
- tptacek 1y agoI asked for an example of one of the articles you'd read that said that LLMs were turning ordinary developers into 10x developers. You cited my article. My article says nothing of the sort; I find the notion of "10x developers" repellant.
- Quarrelsome 1y agohow much faster does an engine on rocket fuel go, than one not on rocket fuel? The article in question[0] has the literal tag line: > My AI Skeptic Friends Are All Nuts how much saner is someone who isn't nuts to someone who is nuts? 10x saner? What do the specific numbers matter given you're not writing a paper? You're enjoying the click bait benefits of using strong language and then acting offended when someone calls you out on it. Yes, maybe you didn't literally say "10x" but you said or quoted things in exactly that same ballpark and its worthy of a counter point like the OP has provided. They're both interesting articles with strong opinions that make the world a more interesting place so idk why you're trying to disown the strength with which you wrote your article. [0] : https://fly.io/blog/youre-all-nuts/ https://fly.io/blog/youre-all-nuts/
- deleted 1y ago[deleted]
- claytongulick 1y agoA cursory scroll on X, LinkedIn, etc... will show you. That seemed to me be to be the author's point. His article resonated with me. After 30 years of development and dealing with hype cycles, offshoring, no-code "platforms", endless framework churn (this next version will make everything better!), coder tribes ("if you don't do typescript, you're incompetent and should be fired"), endless bickering, improper tech adopting following the FANGs (your startup with 0 users needs kubernetes?) and a gazillion other annoyances we're all familiar with, this AI stuff might be the thing that makes me retire. To be clear: it's not AI that I have a problem with. I'm actually deeply interested in it and actively researching it from a math's up approach. I'm also a big believer in it, I've implemented it in a few different projects that have had remarkable efficiency gains for my users, things like automatically extracting values from a PDF to create a structured record. It is a wonderful way to eliminate a whole class of drudgery based tasks. No, the thing that has me on the verge of throwing in the towel is the wholesale rush towards devaluing human expertise. I'm not just talking about developers, I'm talking about healthcare providers, artists, lawyers, etc... Highly skilled professionals that have, in some cases, spent their entire lives developing mastery of their craft. They demand a compensation rate commensurate to that value, and in response society gleefully says "meh, I think you can be replaced with this gizmo for a fraction of the cost." It's an insult. It would be one thing if it were true - my objection could safely be dismissed as the grumbling of a buggy whip manufacturer, however this is objectively, measurably wrong. Most of the energy of the people pushing the AI hype goes towards obscuring this. When objective reality is presented to them in irrefutable ways, the response is inevitably: "but the next version will!" It won't. Not with the current approach. The stochastic parrot will never learn to think. That doesn't mean it's not useful. It demonstrably is, it's an incredibly valuable tool for entire classes of problems, but using it as a cheap replacement for skilled professionals is madness. What will the world be left with when we drive those professionals out? Do you want an AI deciding your healthcare? Do you want a codebase that you've invested your life savings into written by an AI that can't think? How will we innovate? Who will be able to do fundamental research and create new things? Why would you bother going into the profession at all? So we're left with AIs training on increasingly polluted data, and relying on them to push us forward. It's a farce. I've been seriously considering hanging up my spurs and munching popcorn through the inevitable chaos that will come if we don't course correct.
- BoxFour 1y ago> I still think complaining about "hallucination" is a pretty big "tell". The conversation around LLMs is so polarized. Either they’re dismissed as entirely useless, or they’re framed as an imminent replacement for software developers altogether. Hallucinations are worth talking about! Just yesterday, for example, Claude 4 Sonnet confidently told me Godbolt was wrong wrt how clang would compile something (it wasn’t). That doesn’t mean I didn’t benefit heavily from the session, just that it’s not a replacement for your own critical thinking. Like any transformative tool, LLMs can offer a major productivity boost but only if the user can be realistic about the outcome. Hallucinations are real and a reason to be skeptical about what you get back; they don’t make LLMs useless. To be clear, I’m not suggesting you specifically are blind to this fact. But sometimes it’s warranted to complain about hallucinations!
- tptacek 1y agoThat's not what people mean when they bring up "hallucinations". What the author apparently meant was that they had an agent generating Terraform for them, and that Terraform was broken. That's not surprising to me! I'm sure LLMs are helpful for writing Terraform, but I wouldn't expect that agents are at the point of being able to reliably hand off Terraform that actually does anything, because I can't imagine an agent being given permission to iterate Terraform. Now have an agent write Java for you. That problem goes away: you aren't going to be handed code with API calls that literally don't exist (this is what people mean by "hallucination"), because that could wouldn't pass a compile or linter pass.
- coltonv 1y agoWith terraform, using a property or a resource that doesn't exist is effectively the same as an API call that does not exist. It's almost exactly the same really, because under the hood terraform will try to make a gcloud/aws API call with your param and it will not work because it doesn't exist. You are making a distinction without a difference. Just because it can be caught at runtime doesn't make it insignificant. Anyway, I still see hallucinations in all languages, even javascript, attempting to use libraries or APIs that do not exist. Could you elaborate on how you have solved this problem?
- bgwalter 1y agoThat bar is industry standard in the hype machine. Altman and others have set it: https://www.windowscentral.com/software-apps/sam-altman-ai-will-make-coders-10x-more-productive-not-replace-them https://www.windowscentral.com/software-apps/sam-altman-ai-w... https://brianchristner.io/how-cursor-ai-can-make-developers-10x-more-productive/ https://brianchristner.io/how-cursor-ai-can-make-developers-... https://thenewstack.io/the-future-belongs-to-ai-augmented-10x-developers-not-bigger-dev-teams/ https://thenewstack.io/the-future-belongs-to-ai-augmented-10...
- dontlikeyoueith 1y ago> That aside: I still think complaining about "hallucination" is a pretty big "tell". And I think that sentence is a pretty big tell, so ...
- gulbanana 1y agoThe embittered tone and obvious bad faith of your comments here make it clear how seriously we should take your opinions about AI.