3 ms·

Large integer precision error in Bash command output rendering

42 points by rrwright 11mo ago

rrwright 11mo ago
Try it yourself: `echo '348555896224571969'`
- CGamesPlay 11mo ago
  Looks like it has to be the full tool output to be coerced: > Can you run this through bash: echo '348555896224571969 plus 2 is 348555896224571971' Bash(echo '348555896224571969 plus 2 is 348555896224571971') ⎿ 348555896224571969 plus 2 is 348555896224571971
timenotwasted 11mo ago
Claude Code v2.0.37 Haiku 4.5 · Claude Pro > run cmd echo '348555896224571969' I'll run that echo command for you. Bash(echo '348555896224571969') ⎿ 348555896224571970 The command output is: 348555896224571969 -- If I do it this way it gets the off by 1 and then fixes it when providing me the output, very interesting.
ethmarks 11mo ago
Is there any significance to the number 348555896224571969? How was this bug discovered and what was the discoverer doing?
Insanity 11mo ago
I forgot which blogpost mentioned it, but to paraphrase it states that managers won’t understand why you can’t just fix a bug like this in a few minutes like you would in traditional software. This might be one of those cases, where the problem arises from the training set somehow.
- alyxya 11mo ago
  This seems to be a software bug and not something about model behavior, though the model is in some sense doing the wrong thing by internally evaluating what the echo command should output rather than saying what the output actually is. Edit: Based on the above comment showing javascript numerics behavior changing, it's more like some unusual interaction with the numeric string in the bash command being interpreted as an integer and running into precision issues.
- amorriscode 11mo ago
  Merged a PR after seeing this thread so, thankfully, this was one of those bugs that you can just fix in minutes. ;)
fulafel 11mo ago
Smells like floating point. Python prompt: >>> int(float('348555896224571969')) 348555896224571968 It just exceeds the mantissa bits of doubles: >>> math.log2(34855589622457196) 54.952239550875795 JavaScript (in)famously stores all numbers as floating point resulting in silent errors also with user perceived integers, so this might be an indication that Claude Code number handling uses JS native numbers for this.
- ec109685 11mo ago
  They wrap bash with python.
  - fulafel 11mo ago
    I still suspect JS. It's much harder to shoot yourself in the foot with Python. Even if you use JSON: >>> json.loads('{"nr": 348555896224571969}') {'nr': 348555896224571969} >>> type(_['nr']) <class 'int'>
    - porridgeraisin 11mo ago
      TIL that in the python REPL `_` automatically has the previous expr's result. That's cool
- mrspuratic 11mo ago
  It seems like it, but it can't be only that. A float64 representation of an int in the range 2^58 to 2^59 should be rounded to multiples of 2^6, i.e. 348555896224571968 as you found (3.48555896224571968E17) (the final digit 9 in the math.log2() expression was lost, it's 2^58 not 2^54) The unexpected output (according to the bugreport) arises from javascript, it does NOT round like everything else for reasons I don't understand. It seems to prefer rounding to arbitrary multiples of 10 in my limited testing.
CGamesPlay 11mo ago
Claude Code is simultaneously the most useful and lowest quality app I use. It's filled with little errors and annoyances but succeeds despite them. Not to mention the official documentation is entirely vibe-copywritten and any quality control is cursory at best. It forcibly installs itself to ~/.local/bin. Do you already have a file at that location? Not anymore. When typing into the prompt, EACH KEYSTROKE results in the ENTIRE conversation scrollback being cleared and replayed, meaning 1 byte of new data results in kilobytes of data transferred when using Claude over SSH. The tab completion for @-mentioning is so bad it's worthless, and also async, so not even deterministic. You cannot disable their request for feedback. Apparently it lies in tool output. It truly is a testament to the dangers of vibe coding, proudly displayed for everyone to take an example from.
- ec109685 11mo ago
  Are you sure about the one char thing? I’d expect a huge flash if that was the case.
  - CGamesPlay 11mo ago
    No, I'm not sure about the precise mechanics of it, but I noticed it because of the huge flash when using it over a somewhat laggy SSH connection. It doesn't happy in all contexts. I've definitely seen it when typing into the new-ish Claude "ask questions about the plan" flow, and I've also noticed that it redraws the entire conversation history when each new line of output is presented in a long-running tool call.
  - resonious 11mo ago
    It happens over ssh on cellular when the history gets long. Drives me nuts as I'm a heavy claude-over-ssh-on-phone user.
  - 1718627440 11mo ago
    Terminal emulators are fast.
- DecoPerson 11mo ago
cyral 11mo ago
Typing "348555896224571969" into your browser's dev console will also return 348555896224571970
- alyxya 11mo ago
  I don't think this is a bug specifically with Claude Code, rather it's due to Claude Code having javascript in the backend. The interesting thing to me is that the numeric string was interpreted as an integer.
  - rrwright 11mo ago
    I agree with what I think you meant: it is not a bug in Claude. However "javascript on the backend" is exactly what Claude Code is! It's a node.js implementation of automation components that feed and pull from the Claude model. I have a VERY hard time believing that they only use JSON serialization between the model and the TUI. If they're seriealizing JSON between agents, tools, or other components, then this problem is going to continue to persist for a very long time.
rrwright 11mo ago
Why did Hacker News rename the title of this post? It was originally: "Claude Code Introduces Off-by-One Errors" Original: https://pasteboard.co/xTjaRmnkhRRo.png https://pasteboard.co/xTjaRmnkhRRo.png Unilaterally Edited: https://pasteboard.co/rDPINchmufIF.png https://pasteboard.co/rDPINchmufIF.png
- sevg 11mo ago
  Looks like mods changed the title to the title of the GitHub Issue. This from HN guidelines is probably why: > Otherwise please use the original title, unless it is misleading or linkbait; don't editorialize.
  - rrwright 11mo ago
    Good catch on the guidelines. But that github issue title obviously misses the point. The whole point is that it's a silent error in Claude Code.
    - sevg 11mo ago
      A matter of opinion, but I actually don’t think the current headline is too bad. When I saw the headline and “github.com/anthropic” next to it, my initial assumption was that it must be a problem introduced by Claude Code rather than a bug in bash or something. That said, I don’t think your edited headline was bad either, but perhaps there wasn’t enough reason not to use the original (which is a default I personally appreciate on HN).
- dragonwriter 11mo ago
  Probably for a couple reasons: First, HN prefers the source title unless that title is misleaing clickbait. Second, the problem is not consistently off-by-one errors, as there is a manifestation shown in the bug of an off-by-much-less-than-one error. The problem looks like a "for some reason it seems to be roundtripping numbers in text through a numeric representation which has about [perhaps exactly] the same precisions issues as float64" issue.
kazinator 11mo ago
Note that the number is 18 digits long. If there is a conversion to IEEE 64 bit double involved, that type is only guaranteed to record 15 decimal digits of precision, so this number cannot be represented with enough precision to recover all of its original digits. In C implementations, this value is represented as DBL_DIG, which is typically 15 on systems with IEEE floating point. (There is also DBL_DECIMAL_DIG which is typically 17; that's the opposite direction: how many decimal digits we need to print a double such that the exact same double can be recovered by parsing the value. DBL_DIG existed in C90, but DBL_DECIMAL_DIG didn't appear until, it looks like, C11.)
amorriscode 11mo ago
We merged a fix for this that'll go out in the ~next release. Also, for clarification, this bug was only impacting the display of numbers in the TUI, not what the model sees. The model sees raw results from bash.
- rrwright 11mo ago
  This symptom was affecting display of numbers in the TUI, but the real bug/design flaw is using jq or JSON as a transport mechanism between components or between tools. JSON's number range is famously poor and it even architecture-dependent. So if you use jq or JSON to connect components or tools, you will have this same problem silently occurring elsewhere without visibility in the UI. This is a bigger deal than it seems like! A confidence-inducing fix would include a blog post describing a top-to-bottom audit of jq/JSON used as a transport layer between tools and components. Not just a patch to the most visible problem.
dvh 11mo ago
There are 2 hard problems in computer science. Cache invalidation, naming things, and off by 1 errors.
- ahazred8ta 11mo ago
  You left out exactly-once delivery and exactly-once delivery.

42 points by rrwright 11mo ago

rrwright 11mo ago

Try it yourself: `echo '348555896224571969'`

CGamesPlay 11mo ago
Looks like it has to be the full tool output to be coerced: > Can you run this through bash: echo '348555896224571969 plus 2 is 348555896224571971' Bash(echo '348555896224571969 plus 2 is 348555896224571971') ⎿ 348555896224571969 plus 2 is 348555896224571971

timenotwasted 11mo ago

Claude Code v2.0.37 Haiku 4.5 · Claude Pro > run cmd echo '348555896224571969' I'll run that echo command for you. Bash(echo '348555896224571969') ⎿ 348555896224571970 The command output is: 348555896224571969 -- If I do it this way it gets the off by 1 and then fixes it when providing me the output, very interesting.

ethmarks 11mo ago

Is there any significance to the number 348555896224571969? How was this bug discovered and what was the discoverer doing?

Insanity 11mo ago

I forgot which blogpost mentioned it, but to paraphrase it states that managers won’t understand why you can’t just fix a bug like this in a few minutes like you would in traditional software. This might be one of those cases, where the problem arises from the training set somehow.

alyxya 11mo ago
This seems to be a software bug and not something about model behavior, though the model is in some sense doing the wrong thing by internally evaluating what the echo command should output rather than saying what the output actually is. Edit: Based on the above comment showing javascript numerics behavior changing, it's more like some unusual interaction with the numeric string in the bash command being interpreted as an integer and running into precision issues.
amorriscode 11mo ago
Merged a PR after seeing this thread so, thankfully, this was one of those bugs that you can just fix in minutes. ;)

fulafel 11mo ago

Smells like floating point. Python prompt: >>> int(float('348555896224571969')) 348555896224571968 It just exceeds the mantissa bits of doubles: >>> math.log2(34855589622457196) 54.952239550875795 JavaScript (in)famously stores all numbers as floating point resulting in silent errors also with user perceived integers, so this might be an indication that Claude Code number handling uses JS native numbers for this.

ec109685 11mo ago
They wrap bash with python.
- fulafel 11mo ago
  I still suspect JS. It's much harder to shoot yourself in the foot with Python. Even if you use JSON: >>> json.loads('{"nr": 348555896224571969}') {'nr': 348555896224571969} >>> type(_['nr']) <class 'int'>
  - porridgeraisin 11mo ago
    TIL that in the python REPL `_` automatically has the previous expr's result. That's cool
mrspuratic 11mo ago
It seems like it, but it can't be only that. A float64 representation of an int in the range 2^58 to 2^59 should be rounded to multiples of 2^6, i.e. 348555896224571968 as you found (3.48555896224571968E17) (the final digit 9 in the math.log2() expression was lost, it's 2^58 not 2^54) The unexpected output (according to the bugreport) arises from javascript, it does NOT round like everything else for reasons I don't understand. It seems to prefer rounding to arbitrary multiples of 10 in my limited testing.

CGamesPlay 11mo ago

Claude Code is simultaneously the most useful and lowest quality app I use. It's filled with little errors and annoyances but succeeds despite them. Not to mention the official documentation is entirely vibe-copywritten and any quality control is cursory at best. It forcibly installs itself to ~/.local/bin. Do you already have a file at that location? Not anymore. When typing into the prompt, EACH KEYSTROKE results in the ENTIRE conversation scrollback being cleared and replayed, meaning 1 byte of new data results in kilobytes of data transferred when using Claude over SSH. The tab completion for @-mentioning is so bad it's worthless, and also async, so not even deterministic. You cannot disable their request for feedback. Apparently it lies in tool output. It truly is a testament to the dangers of vibe coding, proudly displayed for everyone to take an example from.

ec109685 11mo ago
Are you sure about the one char thing? I’d expect a huge flash if that was the case.
- CGamesPlay 11mo ago
  No, I'm not sure about the precise mechanics of it, but I noticed it because of the huge flash when using it over a somewhat laggy SSH connection. It doesn't happy in all contexts. I've definitely seen it when typing into the new-ish Claude "ask questions about the plan" flow, and I've also noticed that it redraws the entire conversation history when each new line of output is presented in a long-running tool call.
- resonious 11mo ago
  It happens over ssh on cellular when the history gets long. Drives me nuts as I'm a heavy claude-over-ssh-on-phone user.
- 1718627440 11mo ago
  Terminal emulators are fast.
DecoPerson 11mo ago

cyral 11mo ago

Typing "348555896224571969" into your browser's dev console will also return 348555896224571970

alyxya 11mo ago
I don't think this is a bug specifically with Claude Code, rather it's due to Claude Code having javascript in the backend. The interesting thing to me is that the numeric string was interpreted as an integer.
- rrwright 11mo ago
  I agree with what I think you meant: it is not a bug in Claude. However "javascript on the backend" is exactly what Claude Code is! It's a node.js implementation of automation components that feed and pull from the Claude model. I have a VERY hard time believing that they only use JSON serialization between the model and the TUI. If they're seriealizing JSON between agents, tools, or other components, then this problem is going to continue to persist for a very long time.

rrwright 11mo ago

Why did Hacker News rename the title of this post? It was originally: "Claude Code Introduces Off-by-One Errors" Original: https://pasteboard.co/xTjaRmnkhRRo.png https://pasteboard.co/xTjaRmnkhRRo.png Unilaterally Edited: https://pasteboard.co/rDPINchmufIF.png https://pasteboard.co/rDPINchmufIF.png

sevg 11mo ago
Looks like mods changed the title to the title of the GitHub Issue. This from HN guidelines is probably why: > Otherwise please use the original title, unless it is misleading or linkbait; don't editorialize.
- rrwright 11mo ago
  Good catch on the guidelines. But that github issue title obviously misses the point. The whole point is that it's a silent error in Claude Code.
  - sevg 11mo ago
    A matter of opinion, but I actually don’t think the current headline is too bad. When I saw the headline and “github.com/anthropic” next to it, my initial assumption was that it must be a problem introduced by Claude Code rather than a bug in bash or something. That said, I don’t think your edited headline was bad either, but perhaps there wasn’t enough reason not to use the original (which is a default I personally appreciate on HN).
dragonwriter 11mo ago
Probably for a couple reasons: First, HN prefers the source title unless that title is misleaing clickbait. Second, the problem is not consistently off-by-one errors, as there is a manifestation shown in the bug of an off-by-much-less-than-one error. The problem looks like a "for some reason it seems to be roundtripping numbers in text through a numeric representation which has about [perhaps exactly] the same precisions issues as float64" issue.

kazinator 11mo ago

Note that the number is 18 digits long. If there is a conversion to IEEE 64 bit double involved, that type is only guaranteed to record 15 decimal digits of precision, so this number cannot be represented with enough precision to recover all of its original digits. In C implementations, this value is represented as DBL_DIG, which is typically 15 on systems with IEEE floating point. (There is also DBL_DECIMAL_DIG which is typically 17; that's the opposite direction: how many decimal digits we need to print a double such that the exact same double can be recovered by parsing the value. DBL_DIG existed in C90, but DBL_DECIMAL_DIG didn't appear until, it looks like, C11.)

amorriscode 11mo ago

We merged a fix for this that'll go out in the ~next release. Also, for clarification, this bug was only impacting the display of numbers in the TUI, not what the model sees. The model sees raw results from bash.

rrwright 11mo ago
This symptom was affecting display of numbers in the TUI, but the real bug/design flaw is using jq or JSON as a transport mechanism between components or between tools. JSON's number range is famously poor and it even architecture-dependent. So if you use jq or JSON to connect components or tools, you will have this same problem silently occurring elsewhere without visibility in the UI. This is a bigger deal than it seems like! A confidence-inducing fix would include a blog post describing a top-to-bottom audit of jq/JSON used as a transport layer between tools and components. Not just a patch to the most visible problem.

dvh 11mo ago

There are 2 hard problems in computer science. Cache invalidation, naming things, and off by 1 errors.

ahazred8ta 11mo ago
You left out exactly-once delivery and exactly-once delivery.