6 ms·

> I am managing projects in languages I am not fluent in—TypeScript, Rust and Go—and seem to be doing pretty well. This framing reminds me of the classic probl

> I am managing projects in languages I am not fluent in—TypeScript, Rust and Go—and seem to be doing pretty well. This framing reminds me of the classic problem in media literacy: people know when a journalistic source is poor when they’re a subject matter expert, but tend to assume that the same source is at least passably good when less familiar with the subject. I’ve had the same experience as the author when doing web development with LLMs: it seems to be doing a pretty good job, at least compared to the mess I would make. But I’m not actually qualified to make that determination, and I think a nontrivial amount of AI value is derived from engineers thinking that they are qualified as such.

giantrobot 1y ago
Gell-Mann Amnesia [0] [0] https://en.m.wikipedia.org/wiki/Gell-Mann_amnesia_effect https://en.m.wikipedia.org/wiki/Gell-Mann_amnesia_effect
- woodruffw 1y ago
  Thank you! I couldn’t remember the term.
  - giantrobot 1y ago
    > I couldn’t remember the term. That's lethologica! Or maybe in this specific case lethonomia. [0] [0] https://en.m.wikipedia.org/wiki/Tip_of_the_tongue https://en.m.wikipedia.org/wiki/Tip_of_the_tongue
bravesoul2 1y ago
Why I only use it on stuff I can properly judge.
deleted 1y ago
[deleted]
muglug 1y ago
Yup — this doesn't match my experience using Rust with Claude. I've spent 2.5 years writing Rust professionally, and I'm pretty good at it. Claude will hallucinate things about Rust code because it’s a statistical model, not a static analysis tool. When it’s able to create code that compiles, the code is invariably inefficient and ugly. But if you want it to generate chunks of usable and eloquent Python from scratch, it’s pretty decent. And, FWIW, I’m not fluent in Python.
- js2 1y ago
  Hah... yeah, no, its Python isn't great. I'd definitely workable and better than what I see from 9/10 junior engineers, but it tends to be pretty verbose and over-engineered. My repos all have pre-commit hooks which run the linters/formatters/type-checkers. Both Claude and Gemini will sometimes write code that won't get past mypy and they'll then struggle to get it typed correct before eventually by passing the pre-commit check with `git commit -n`. I've had to add some fairly specific instructions to CLAUDE.md/GEMINI.md to get them to cut this out. Claude is better about following the rules. Gemini just flat out ignores instructions. I've also found Gemini is more likely to get stuck in a loop and give up. That said, I'm saying this after about 100 hours of experience with these LLMs. I'm sure they'll get better with their output and I'll get better with my input.
  - physicsguy 1y ago
    To be fair, depending on what libraries you’re using, Python typing isn’t exactly easy even for a human, I spend more time battling with type checkers and stubs than I would like.
  - hkt 1y ago
    I can confirm input matters a lot. I'm a couple of hundred hours ahead of you and my prompting has come along a lot. I recommend test cycles, prompts to reflect on product-implementation fit (eg, is this what you've been asked to do?) and lots of interactivity. Despite what I've written elsewhere in these comments, the best work is a good oneshot followed by small iterations and attentive steering.
js2 1y ago
After decades of writing software, I feel like I have a pretty good sense for "this can't possibly be idiomatic" in a new language. If I sniff something is off, I start Googling for reference code, large projects in that language, etc. You can also just ask the LLM: are you sure this is idiomatic? Of course it may lie to you...
- woodruffw 1y ago
  > If I sniff something is off, I start Googling for reference code, large projects in that language, etc. This works so long as you know how to ask the question. But it's been my experience that an LLM directed on a task will do something, and I don't even know how to frame its behavior in language in a way that would make sense to search for. (My experience here is with frontend in particular: I'm not much of a JS/TS/HTML/CSS person, and LLMs produce outputs that look really good to me. But I don't know how to even begin to verify that they are in fact good or idiomatic, since there's more often than not multiple layers of intermediating abstractions that I'm not already familiar with.)
  - fc417fc802 1y ago
    > and I don't even know how to frame its behavior in language in a way that would make sense to search for. Have you tried recursion? Something like: "Using idiomatic terminology from the foo language ecosystem, explain what function x is doing." If all goes well it will hand you the correct terminology to frame your earlier question. Then you can do what the adjacent comment describes and ask it what the idiomatic way of doing p in q is.
    - woodruffw 1y ago
      I think you’re missing the point. The point is that I’m not qualified to evaluate the LLM’s output in this context. Having it self-report doesn’t change that fact, it’s just playing hide the pickle by moving the evaluation around.
      - fc417fc802 1y ago
Cthulhu_ 1y ago
I think the concept of "readability" is good, it's a program within Google where your code gets reviewed by an expert in that language (but not necessarily your application / domain); once you're up to a level of writing idiomatic code and fully understanding the language etc, you get readability yourself. When reviewing LLM code, you should have this readability in the given language yourself - or the code should not be important.
deleted 1y ago
[deleted]

giantrobot 1y ago

Gell-Mann Amnesia [0] [0] https://en.m.wikipedia.org/wiki/Gell-Mann_amnesia_effect https://en.m.wikipedia.org/wiki/Gell-Mann_amnesia_effect

woodruffw 1y ago
Thank you! I couldn’t remember the term.
- giantrobot 1y ago
  > I couldn’t remember the term. That's lethologica! Or maybe in this specific case lethonomia. [0] [0] https://en.m.wikipedia.org/wiki/Tip_of_the_tongue https://en.m.wikipedia.org/wiki/Tip_of_the_tongue

bravesoul2 1y ago

Why I only use it on stuff I can properly judge.

deleted 1y ago

[deleted]

muglug 1y ago

Yup — this doesn't match my experience using Rust with Claude. I've spent 2.5 years writing Rust professionally, and I'm pretty good at it. Claude will hallucinate things about Rust code because it’s a statistical model, not a static analysis tool. When it’s able to create code that compiles, the code is invariably inefficient and ugly. But if you want it to generate chunks of usable and eloquent Python from scratch, it’s pretty decent. And, FWIW, I’m not fluent in Python.

js2 1y ago
Hah... yeah, no, its Python isn't great. I'd definitely workable and better than what I see from 9/10 junior engineers, but it tends to be pretty verbose and over-engineered. My repos all have pre-commit hooks which run the linters/formatters/type-checkers. Both Claude and Gemini will sometimes write code that won't get past mypy and they'll then struggle to get it typed correct before eventually by passing the pre-commit check with `git commit -n`. I've had to add some fairly specific instructions to CLAUDE.md/GEMINI.md to get them to cut this out. Claude is better about following the rules. Gemini just flat out ignores instructions. I've also found Gemini is more likely to get stuck in a loop and give up. That said, I'm saying this after about 100 hours of experience with these LLMs. I'm sure they'll get better with their output and I'll get better with my input.
- physicsguy 1y ago
  To be fair, depending on what libraries you’re using, Python typing isn’t exactly easy even for a human, I spend more time battling with type checkers and stubs than I would like.
- hkt 1y ago
  I can confirm input matters a lot. I'm a couple of hundred hours ahead of you and my prompting has come along a lot. I recommend test cycles, prompts to reflect on product-implementation fit (eg, is this what you've been asked to do?) and lots of interactivity. Despite what I've written elsewhere in these comments, the best work is a good oneshot followed by small iterations and attentive steering.

js2 1y ago

After decades of writing software, I feel like I have a pretty good sense for "this can't possibly be idiomatic" in a new language. If I sniff something is off, I start Googling for reference code, large projects in that language, etc. You can also just ask the LLM: are you sure this is idiomatic? Of course it may lie to you...

woodruffw 1y ago
> If I sniff something is off, I start Googling for reference code, large projects in that language, etc. This works so long as you know how to ask the question. But it's been my experience that an LLM directed on a task will do something, and I don't even know how to frame its behavior in language in a way that would make sense to search for. (My experience here is with frontend in particular: I'm not much of a JS/TS/HTML/CSS person, and LLMs produce outputs that look really good to me. But I don't know how to even begin to verify that they are in fact good or idiomatic, since there's more often than not multiple layers of intermediating abstractions that I'm not already familiar with.)
- fc417fc802 1y ago
  > and I don't even know how to frame its behavior in language in a way that would make sense to search for. Have you tried recursion? Something like: "Using idiomatic terminology from the foo language ecosystem, explain what function x is doing." If all goes well it will hand you the correct terminology to frame your earlier question. Then you can do what the adjacent comment describes and ask it what the idiomatic way of doing p in q is.
  - woodruffw 1y ago
    I think you’re missing the point. The point is that I’m not qualified to evaluate the LLM’s output in this context. Having it self-report doesn’t change that fact, it’s just playing hide the pickle by moving the evaluation around.
    - fc417fc802 1y ago

Cthulhu_ 1y ago

I think the concept of "readability" is good, it's a program within Google where your code gets reviewed by an expert in that language (but not necessarily your application / domain); once you're up to a level of writing idiomatic code and fully understanding the language etc, you get readability yourself. When reviewing LLM code, you should have this readability in the given language yourself - or the code should not be important.

deleted 1y ago

[deleted]