5 ms·

I’ve found that LLMs will frequently do extremely silly things that no person would do to make typescript code pass the typechecker.

akdev1l 7mo ago
You need to very specific and also question the output if it does something insane
- justonceokay 7mo ago
  This decade’s version of “works on my box”
- mexicocitinluez 7mo ago
  I've found it's less about specificity and more about removing the # of critical assumptions it needs to make. Being too specific can be a hindrance in it's own regard. And that's also a decent barometer for what it's good at. The more amount of critical assumptions AI needs to make, the less likely it is to make good ones. For instance, when building a heat map, I don't have to get specific at all because the amount of consequential assumptions it needs to make is slim. I don't care or can change the colors, or the label placement.
dpkirchner 7mo ago
I caught it using Parameters<typeof otherfn>[2] the other day. It wanted to avoid importing a type, so it did this nonsense. (I might have the syntax slightly wrong here, I'm writing from memory.) But it's not all bad news. TIL about Parameters<T>.
9rx 7mo ago
Yeah, I've found LLMs cannot write good Typescript code period. The good news is that they are excellent at some other languages.
- darkstarsys 7mo ago
  I can't agree here. https://pelorus-nav.com/ https://pelorus-nav.com/ (one of my side projects) is 95-98% written by Claude Opus 4.6, all in very nice typescript which I carefully review and correct, and use good prompting and context hygiene to ensure it doesn't take shortcuts. It's taken a month or so but so worth it. And my packing list app packzen.org is also pretty decent typescript all through.
  - 9rx 7mo ago
    > which I carefully review and correct So you do agree? If you are having to review and correct then it's not really the LLM writing it anymore. I have little doubt that you can write good Typescript, but that's not what I said. I said LLMs cannot write good Typescript and it seems you agree given your purported actions towards it. Which is quite unlike some other languages where LLMs write good code all the time — no hand holding necessary.
    - darkstarsys 7mo ago
      I find correction is rarely necessary with Opus 4.6. Definitely not so much that "it's not really the LLM writing it anymore." More like it's the author and I'm the editor (in this limited case -- of course architecturally the ideas are all mine.) But I totally respect that my prompt style, the type of app I'm writing, and other factors could be influencing my success vs. others' lack of success.
      - 9rx 7mo ago
        > of course architecturally the ideas are all mine. What else would you need to correct? I've never had trouble with LLMs generating basic syntax in any language. Architecture is exactly the aspect of language where LLMs seem to like to go to crazytown when in Typescript. It seems you've noticed too if the ideas in that area have had to come all from you.
- camdenreslink 7mo ago
  I think it can write working TypeScript code, and it can write good TypeScript code if it is guided by a knowledgable programmer. It requires actually reviewing all the code and giving pointed feedback though (which at that point is only slightly more efficient than just writing it yourself).
gck1 7mo ago
I've noticed this too, but not necessarily type checkers, but more with linters. And can't really figure out if there's even a way to solve it. If you set up restrictive linters and don't explicitly prohibit agents from adding inline allows, most LOC will be allow comments. Based on this learning, I've decided to prohibit any inline allows. And then agents started doing very questionable things to satisfy clippy. Recent example: - Claude set up a test support module so that it could reuse things. Since this was not used in all tests, rust complained about dead_code. Instead of making it work, claude decided to remove test support module and just... blow up each test. If you enable thinking summaries, you'll always see agent saying something like: "I need to be pragmatic", which is the right choice 50% of the time.

akdev1l 7mo ago

You need to very specific and also question the output if it does something insane

justonceokay 7mo ago
This decade’s version of “works on my box”
mexicocitinluez 7mo ago
I've found it's less about specificity and more about removing the # of critical assumptions it needs to make. Being too specific can be a hindrance in it's own regard. And that's also a decent barometer for what it's good at. The more amount of critical assumptions AI needs to make, the less likely it is to make good ones. For instance, when building a heat map, I don't have to get specific at all because the amount of consequential assumptions it needs to make is slim. I don't care or can change the colors, or the label placement.

dpkirchner 7mo ago

I caught it using Parameters<typeof otherfn>[2] the other day. It wanted to avoid importing a type, so it did this nonsense. (I might have the syntax slightly wrong here, I'm writing from memory.) But it's not all bad news. TIL about Parameters<T>.

9rx 7mo ago

Yeah, I've found LLMs cannot write good Typescript code period. The good news is that they are excellent at some other languages.

darkstarsys 7mo ago
I can't agree here. https://pelorus-nav.com/ https://pelorus-nav.com/ (one of my side projects) is 95-98% written by Claude Opus 4.6, all in very nice typescript which I carefully review and correct, and use good prompting and context hygiene to ensure it doesn't take shortcuts. It's taken a month or so but so worth it. And my packing list app packzen.org is also pretty decent typescript all through.
- 9rx 7mo ago
  > which I carefully review and correct So you do agree? If you are having to review and correct then it's not really the LLM writing it anymore. I have little doubt that you can write good Typescript, but that's not what I said. I said LLMs cannot write good Typescript and it seems you agree given your purported actions towards it. Which is quite unlike some other languages where LLMs write good code all the time — no hand holding necessary.
  - darkstarsys 7mo ago
    I find correction is rarely necessary with Opus 4.6. Definitely not so much that "it's not really the LLM writing it anymore." More like it's the author and I'm the editor (in this limited case -- of course architecturally the ideas are all mine.) But I totally respect that my prompt style, the type of app I'm writing, and other factors could be influencing my success vs. others' lack of success.
    - 9rx 7mo ago
      > of course architecturally the ideas are all mine. What else would you need to correct? I've never had trouble with LLMs generating basic syntax in any language. Architecture is exactly the aspect of language where LLMs seem to like to go to crazytown when in Typescript. It seems you've noticed too if the ideas in that area have had to come all from you.
camdenreslink 7mo ago
I think it can write working TypeScript code, and it can write good TypeScript code if it is guided by a knowledgable programmer. It requires actually reviewing all the code and giving pointed feedback though (which at that point is only slightly more efficient than just writing it yourself).

gck1 7mo ago

I've noticed this too, but not necessarily type checkers, but more with linters. And can't really figure out if there's even a way to solve it. If you set up restrictive linters and don't explicitly prohibit agents from adding inline allows, most LOC will be allow comments. Based on this learning, I've decided to prohibit any inline allows. And then agents started doing very questionable things to satisfy clippy. Recent example: - Claude set up a test support module so that it could reuse things. Since this was not used in all tests, rust complained about dead_code. Instead of making it work, claude decided to remove test support module and just... blow up each test. If you enable thinking summaries, you'll always see agent saying something like: "I need to be pragmatic", which is the right choice 50% of the time.