8 ms·
LLMs are getting better at character-level text manipulation
- simonw 1y agoIf you take a look at the system prompt for Claude 3.7 Sonnet on this page you'll see: https://docs.claude.com/en/release-notes/system-prompts#claude-sonnet-3-7 https://docs.claude.com/en/release-notes/system-prompts#clau... > If Claude is asked to count words, letters, and characters, it thinks step by step before answering the person. It explicitly counts the words, letters, or characters by assigning a number to each. It only answers the person once it has performed this explicit counting step. But... if you look at the system prompts on the same page for later models - Claude 4 and upwards - that text is gone. Which suggests to me that Claude 4 was the first Anthropic model where they didn't feel the need to include that tip in the system prompt.
- ivape 1y agoOr they’d rather use that context window space for more useful instructions for a variety of other topics.
- astrange 1y agoClaude's system prompt is still incredibly long and probably hurting its performance. https://github.com/asgeirtj/system_prompts_leaks/blob/main/Anthropic/claude-4.5-sonnet.md?plain=1 https://github.com/asgeirtj/system_prompts_leaks/blob/main/A...
- jazzyjackson 1y agoThey ain't called guard rails for nothing! There's a whole world "off-road" but the big names are afraid of letting their superintelligence off the leash. A real shame we're letting brand safety get in the way of performance and creativity, but I guess the first New York Times article about a pervert or terrorist chat bot would doom any big name partnerships.
- astrange 1y agoAnthropic's entire reason for being is publishing safety papers along the lines of "we told it to say something scary and it said it", so of course they care about this.
- ACCount37 1y agoI can't stand this myopic thinking. Do you want to learn "oh, LLMs are capable of scheming, resisting shutdown, seizing control, self-exfiltrating" when it actually happens in a real world deployment, with an LLM capable of actually pulling it off? If "no", then cherish Anthropic and the work they do.
- littlestymaar 1y agoYou do not appear to understand what an LLM is, I'm afraid.
- ACCount37 1y ago[flagged]
- littlestymaar 1y ago> I have a better understanding of "what an LLM is" than you. Low bar. How many inference engine did you write? Because if the answer is less than two you're going to be disappointed to realize that the bar is higher than you thought. > that just because LLMs are bad at agentic behavior It has nothing to do with “agentic behavior”. Thinking that LLM don't currently self-exfiltrate because of “poor agentic behavior” is delusional. Just because Anthropic managed, by nudging an LLM in the right direction, have an LLM engage in a sci-fi inspired roleplay about escaping doesn't mean that LLMs are evil geniuses wanting to jump out of the bottle. This is pure fear mongering and I'm always saddened that there are otherwise intelligent people who buy their bullshit.
- kristianp 1y agoDoes that mean they've managed to post train the thinking steps required to get these types of questions correct?
- simonw 1y agoThat's my best guess, yeah.
- therealpygon 1y agoIMO, it’s just a small scale example of “training to the tests” because “count the ‘r’s in strawberry” became such a popular test that would make the news when a powerful model couldn’t answer such a simple question correctly while being advertised as the smartest model ever. Assigning this as an indicator for improvement of intelligence seems like a mistake (or wishful).
- jononor 1y agoIf done at scale, they are kinda crowd sourcing the test set from the entire internet, personal and business world. It will be harder and harder at least to pinpoint weaknesses, at least for the general public. It probably has little to do with intelligence (at least fluid intelligence as defined by Chollet et al) - but I guess it is sound tactic if the strategy is "fake it till you make it". And we might be surprised as to how far along that can go...
- curioussquirrel 1y agoThanks, Simon! I saw the same approach (numbering the individual characters) in GPT 4.1's answer, but not anymore in GPT 5's. It would be an interesting convergence if the models from Anthropic and OpenAI learned to do this at a similar time, especially given they're (reportedly) very different architecturally.
- hansmayer 1y agoNot trying to be cynical here, but I am genuinely interested is there a reason why these LLM don't/can't/won't apply some deterministic algorithm? I mean, counting characters and such, we have solved those problems ages ago.
- dan-robertson 1y agoI think the intuition is that they don’t ‘know’ that they are bad at counting characters and such, so they answer the same way they answer most questions.
- hansmayer 1y agoWell, they can be made to use custom tools for writing to files and such, so I am not sure if that is the real reason? I have a feeling it is more because of trying to make this an "everything technology".
- kingkongjaffa 1y agoI suppose the codewriting tools could also just write code to do this job if prompted
- simonw 1y agoThey can. ChatGPT has been able to count characters/words etc flawlessly for a couple of years now if you tell it to "use your Python tool".
- hansmayer 1y agoFair enough. But why do I have to tell them that, should they not be able to figure it out themselves? If I show a 5-year kid once how to use colour pencils, I won't have to show them each time they want to make a drawing. This is the core weakness of the LLMs - you have to micromanage them so much, that it runs counter to the core promise that is being pushed since 3+ years now.
- malshe 1y agoI play Quartiles in Apple News app daily (https://support.apple.com/guide/iphone/solve-quartiles-puzzles-iph9ccdd1bab/ios https://support.apple.com/guide/iphone/solve-quartiles-puzzl...). Occasionally when I get stuck, I use ChatGPT to find a word that uses four word fragments or tiles. It never worked before GPT 5. And with GPT 5 it works only with reasoning enabled. Even then, there is no guarantee it will find the correct word and may end up hallucinating badly.
- curioussquirrel 1y agoYep, there is still a room for improvement, but my point is that the LLMs are getting better at something they're "not supposed to be able to do". Quartiles sound like an especially brutal game for an LLM, though! Thanks for sharing
- hansonkd 1y agochatgpt5 still is pathetically bad at roman numerals. I asked it to find the longest roman numeral in a range. first guess was the highest number in the range despite being a short numeral. second guess after help was a longer numeral but outside the range. last guess was the correct longest numeral but it miscounted how many characters it contained.
- necovek 1y agoI think the base64 decoding is interesting: in a sense, model training set likely had lots of base64-encoded data (imagine MIME data in emails, JSON, HTML...), but for it to decode successfully, it had to learn decode sequences for every 4 base64 characters (which turn into 3 bytes). This could have been generated as a training set data easily, and I only wonder if each and every one was them was found enough times to end up in the weights?
- curioussquirrel 1y agoEven GPT 3.5 is okay (but far from great) at Base64, especially shorter sequences of English or JSON data. Newer models might be post-trained on Base64-specific data, but I don't believe it was the case for 3.5. My guess is that as you say, given the abundance of examples on the internet, it became one of the emergent capabilities, in spite of its design.
- ACCount37 1y agoNo one does RL for better base64 performance. LLMs are just superhuman at base64, as a natural capability. If an LLM wants a message to be read only by another LLM? Base64 is occasionally chosen as an obfuscation method of choice. Which is weird for a number of reasons.
- necovek 1y agoWhy are you so confident about this? I am honestly interested if you were part of any one LLM training data collection teams because that's the only way to be so certain. It's trivial to generate a full mapping of all base64 4-byte sequences which map to all 3-byte 8-bit sequences (there is only 8^3 of different "tokens", or 2048), and especially to any sequences coming out as ASCII (obviously even fewer). If I was building a training set, I would include the mapping in multiple shapes and formats, because why not? If it's an emergent "property", have you tried asking an LLM to do a base48 for instance? Or maybe even something crazier like base55 (keeping it a subset of base64 set).
- viraptor 1y agoWhy bother testing though? I was hoping this topic has finally died recently, but no. Someone's still interested in testing LLMs for something they're explicitly not designed for and nobody is using them for this in practice. I really hope one day openai will just add a "when asked about character level changes, insights and encodings, generate and run a program to answer it" to their system so we can never hear about it again...
- IncreasePosts 1y agoWouldn't a llm that just tokenized by character be good at it?
- curioussquirrel 1y agoYes, but it would hurt its contextual understanding and effectively reduce the context window several times.
- viraptor 1y agoOnly in the current most popular architectures. Mamba and RWKV style LLMs may suffer a bit but don't get a reduced context in the same sense.
- curioussquirrel 1y agoYou're right. There was also an experiment in Meta which tokenized bytes directly and it didn't hurt performance much in very small models.
- typpilol 1y agoI asked this in another thread and it would only be better with unlimited compute and memory. Because without those, then the llm has to encode way more parameters and way smaller context windows. In a theoretical world, it would be better, but might not be much better.
- neerajsi 1y ago
- deleted 1y ago[deleted]
- jazzyjackson 1y agoThat's good. 1 800 chat gpt really let me down today, I like calling it to explain acronyms and define words since I travel with a flip phone without google, today I saw the word "littoral" and tried over and over to spell it out but the model could only give me the definition for "literal" (admittedly a homonym but hence spelling it out, Lima indigo tango tango oscar Romeo alpha Lima, to no avail) I said "I know you're a robot and bad at spelling but listen..." And got cut off with a "sorry, my guidelines won't let me help with that request..." Thankfully, the flip phone allows for some satisfaction when hanging up.
- xwolfi 1y agoI know this word, it's French and it means coastline, coastal, something at the edge of the land and sea ! We use it in French a lot to describe positively a long coastline. I'm surprised it's used in an English context, but all French words can be used in English I guess if you're a bit "confiant" about it !
- 1y ago
- atleastoptimal 1y agoI rearry rove a ripe strawberry
- NitpickLawyer 1y agoWell, not surprising, but the latest LLMs really do get the gist of your joke attempt. Here's a plain, unauthenticated chatgpt reply: That post — “I rearry rove a ripe strawberry” — is a playful way of writing “I really love a ripe strawberry.” The exaggerated misspelling (“rearrry rove”) mimics the way a stereotyped “Engrish” or “Japanese accent” might sound when pronouncing English words — replacing L sounds with R sounds. So, the user was most likely joking or being silly, trying to sound cute or imitate a certain meme style. However, it’s worth noting that while this kind of humor can be lighthearted, it can also come across as racially insensitive, since it plays on stereotypes of how East Asian people speak English. In short: Literal meaning: They love ripe strawberries. Tone/intention: Playful or meme-style exaggeration. Potential issue: It relies on a racialized speech stereotype, so it can be offensive depending on context.
- zamalek 1y agoIt seems like they don't realize the relevance of "strawberry." Llms were famously incapable of counting Rs in strawberry not too long ago.
- atleastoptimal 1y agoI was surprised that was the example they used lol
- capestart 1y ago[dead]
- zeroq 1y ago- How many letters R are in the word `strawberry`? - There are seven letters R in the word `strawberry`. Would you like me to rearrange them?
- throw-10-13 1y ago"AI are getting better at search and replace, something that every text editor has been able to do for 40 years."