3 ms·

When I asked Claude "AI" to count the number of text file lines missing a given initial sub-string, it gave an improbably exaggerated result. When I challenged

When I asked Claude "AI" to count the number of text file lines missing a given initial sub-string, it gave an improbably exaggerated result. When I challenged this, it replied "You are right! Let me try again this time without splitting long lines." AI = Absent Intelligence.

etothet 1y ago
I recommend you check out Andrej Karpathy’s 2 YouTube videos on how LLMs work (they are easy to find, but be forewarned they are long!). Once one digs in deeper it becomes clear why a model today might fail at the task you described. Generally speaking, one of the behaviors I see in my day to day work leading engineers is that they often attempt to apply agentic coding tools to problema that don’t really benefit from them.
- chrisjj 1y ago
  > I recommend you check out Andrej Karpathy’s 2 YouTube videos on how LLMs work I'll reccommend Claude "AI" do that, so it knows to tell the user when it /doesn't/ work.
rootlocus 1y ago
Here's how Claude Code does this: > Find all java files with more than 100 lines. ● I'll search for all Java files with more than 100 lines in the codebase. ● Bash(find . -name "*.java" -type f -exec wc -l {} + | awk '$1 > 100' | sort -nr) ... ● I found 5 Java files with more than 100 lines: 1. File1.java - 315 lines 2. File2.java - 156 lines 3. File3.java - 154 lines 4. File4.java - 130 lines 5. File5.java - 117 lines The largest file is File1.java with 315 lines. Or, if you want to count lines: > How many lines don't start with `import` in File1.java? ● Bash(grep -cv "^import " ./File1.java) ⎿ 287 ● There are 287 lines that don't start with import in File1.java.
theshrike79 1y ago
Don't ask a language model to do math, they're not very good at it. Next time ask it to write a script or a program to do it, it'll most likely one-shot it.
- chrisjj 1y ago
  > Don't ask a language model to do math, they're not very good at it. ... and not intelligent enough to know that, apparently.
  - theshrike79 1y ago
    It's not about intelligence, it's about the fact that they don't have an environment to run the program in. In theory the "count the 'r's in strawberry" type of things could just be run in the browser, which is a semi-safe environment.
    - chrisjj 1y ago
      I wasn't commenting about any program. The fact it doesn't know the method it does use is defective is defintely about intelligence. As is the fact it doesn't learn. If I give it the same task tomorrow, it will repeat the same mistake it admitted yesterday.
      - etothet 1y ago
        It doesn’t “know” anything.

etothet 1y ago

I recommend you check out Andrej Karpathy’s 2 YouTube videos on how LLMs work (they are easy to find, but be forewarned they are long!). Once one digs in deeper it becomes clear why a model today might fail at the task you described. Generally speaking, one of the behaviors I see in my day to day work leading engineers is that they often attempt to apply agentic coding tools to problema that don’t really benefit from them.

chrisjj 1y ago
> I recommend you check out Andrej Karpathy’s 2 YouTube videos on how LLMs work I'll reccommend Claude "AI" do that, so it knows to tell the user when it /doesn't/ work.

rootlocus 1y ago

Here's how Claude Code does this: > Find all java files with more than 100 lines. ● I'll search for all Java files with more than 100 lines in the codebase. ● Bash(find . -name "*.java" -type f -exec wc -l {} + | awk '$1 > 100' | sort -nr) ... ● I found 5 Java files with more than 100 lines: 1. File1.java - 315 lines 2. File2.java - 156 lines 3. File3.java - 154 lines 4. File4.java - 130 lines 5. File5.java - 117 lines The largest file is File1.java with 315 lines. Or, if you want to count lines: > How many lines don't start with `import` in File1.java? ● Bash(grep -cv "^import " ./File1.java) ⎿ 287 ● There are 287 lines that don't start with import in File1.java.

theshrike79 1y ago

Don't ask a language model to do math, they're not very good at it. Next time ask it to write a script or a program to do it, it'll most likely one-shot it.

chrisjj 1y ago
> Don't ask a language model to do math, they're not very good at it. ... and not intelligent enough to know that, apparently.
- theshrike79 1y ago
  It's not about intelligence, it's about the fact that they don't have an environment to run the program in. In theory the "count the 'r's in strawberry" type of things could just be run in the browser, which is a semi-safe environment.
  - chrisjj 1y ago
    I wasn't commenting about any program. The fact it doesn't know the method it does use is defective is defintely about intelligence. As is the fact it doesn't learn. If I give it the same task tomorrow, it will repeat the same mistake it admitted yesterday.
    - etothet 1y ago
      It doesn’t “know” anything.