3 ms·

Why is it replacing true/false with T/F? true/false is already 1 token in all tokenizer I've seen. Even worse is replacing null with ∅. ∅ is a special unicode s

by debazel 2mo ago

Why is it replacing true/false with T/F? true/false is already 1 token in all tokenizer I've seen. Even worse is replacing null with ∅. ∅ is a special unicode symbol that takes up 2 tokens compared to the 1 token for null...

AmazingTurtle 2mo ago
↲ is also two tokens instead of a simple \n lmao
- hnlmorg 2mo ago
  How is an LF two tokens? Or were you referring to the Unicode symbol? I took their example to mean an actual LF ASCII character but now Ive read your comment, maybe I was being too charitable?
  - wongarsu 2mo ago
    They literally replace \n with the unicode symbol https://github.com/activeing123/mcptoon/blob/main/src/mcptoon/output.py#L89 https://github.com/activeing123/mcptoon/blob/main/src/mcptoo...
    - hnlmorg 2mo ago
      Wow. Just “wow”.
cedws 2mo ago
Brand new GitHub account, brand new HN account. I stay far away from projects like this these days, they can easily be malicious. GitHub needs some kind of indicator for projects authored by tenured developers with a real identity.
- plufz 2mo ago
  And they need some kind of downvote. Projects needs to be able to lose a star.
- mcptokensaver 2mo ago
  [flagged]
fallingbananna 2mo ago
What's even weirder is that the substitution is explained in "How TOON works" section. Yet TOON never describes such behavior anywhere in its spec.
wehnsdaefflae 2mo ago
Maybe they want to be tokenizer agnostic? Then they would need to go by character count, right? Despite these inconsistencies, has anyone actually verified their promise? 350 vs. 10.000 tokens would still be very valuable even if they mess up some edge cases
- wannabe44 2mo ago
  Oh it is not returning full tool schema. You can't cut down that much by serialisation alone.
vunderba 2mo ago
Agreed. This is what happens when you confuse token counts with byte counts. A trivial test through tiktoken [1] (though technically you really have to match the tokenizer to the specific LLM) would have shown them that ∅ was a poor choice. Even from the perspective of learned training data, you can probably just intuit that from a frequency standpoint alone the empty-set symbol ∅ can’t possibly have appeared that often outside of things like set theory and logic. [1] - https://github.com/openai/tiktoken https://github.com/openai/tiktoken
- mcptokensaver 2mo ago
  [flagged]
mcptokensaver 2mo ago
[flagged]
mcptokensaver 2mo ago
[dead]
mcptokensaver 2mo ago
[flagged]
scripthound841 2mo ago
[flagged]

by debazel 2mo ago

AmazingTurtle 2mo ago

↲ is also two tokens instead of a simple \n lmao

hnlmorg 2mo ago
How is an LF two tokens? Or were you referring to the Unicode symbol? I took their example to mean an actual LF ASCII character but now Ive read your comment, maybe I was being too charitable?
- wongarsu 2mo ago
  They literally replace \n with the unicode symbol https://github.com/activeing123/mcptoon/blob/main/src/mcptoon/output.py#L89 https://github.com/activeing123/mcptoon/blob/main/src/mcptoo...
  - hnlmorg 2mo ago
    Wow. Just “wow”.

cedws 2mo ago

Brand new GitHub account, brand new HN account. I stay far away from projects like this these days, they can easily be malicious. GitHub needs some kind of indicator for projects authored by tenured developers with a real identity.

plufz 2mo ago
And they need some kind of downvote. Projects needs to be able to lose a star.
mcptokensaver 2mo ago
[flagged]

fallingbananna 2mo ago

What's even weirder is that the substitution is explained in "How TOON works" section. Yet TOON never describes such behavior anywhere in its spec.

wehnsdaefflae 2mo ago

Maybe they want to be tokenizer agnostic? Then they would need to go by character count, right? Despite these inconsistencies, has anyone actually verified their promise? 350 vs. 10.000 tokens would still be very valuable even if they mess up some edge cases

wannabe44 2mo ago
Oh it is not returning full tool schema. You can't cut down that much by serialisation alone.

vunderba 2mo ago

Agreed. This is what happens when you confuse token counts with byte counts. A trivial test through tiktoken [1] (though technically you really have to match the tokenizer to the specific LLM) would have shown them that ∅ was a poor choice. Even from the perspective of learned training data, you can probably just intuit that from a frequency standpoint alone the empty-set symbol ∅ can’t possibly have appeared that often outside of things like set theory and logic. [1] - https://github.com/openai/tiktoken https://github.com/openai/tiktoken

mcptokensaver 2mo ago
[flagged]

mcptokensaver 2mo ago

[flagged]

mcptokensaver 2mo ago

[dead]

mcptokensaver 2mo ago

[flagged]

scripthound841 2mo ago

[flagged]