5 ms·
Does this mean Claude no longer outputs the full raw reasoning, only summaries? At one point, exposing the LLM's full CoT was considered a core safety tenet.
by puppystench 6mo ago
Does this mean Claude no longer outputs the full raw reasoning, only summaries? At one point, exposing the LLM's full CoT was considered a core safety tenet.
- andrepd 6mo agoCoT is basically bullshit, entirely confabulated and not related to any "thought process"...
- clbrmbr 6mo agoBut still CoT distillation WORKS. See the DeepSeek R1 paper.
- whattheheckheck 6mo agoTokens relate to each other. More tokens more compute
- fasterthanlime 6mo agoI don't think it ever has. For a very long time now, the reasoning of Claude has been summarized by Haiku. You can tell because a lot of the times it fails, saying, "I don't see any thought needing to be summarised."
- fmbb 6mo agoMaybe there was no thinking.
- derrida 6mo agoNot a haiku, more a koan.
- astrange 6mo agoIt also gets confused if the entire prompt is in a text file attachment. And the summarizer shows the safety classifier's thinking for a second before the model thinking, so every question starts off with "thinking about the ethics of this request".
- FeepingCreature 6mo agoI'd get confused if I was a LLM and you put my entire prompt in a text file attachment. I'd be like, "is this the user or is this a prompt injection??"
- astrange 6mo agoIf you paste a long enough prompt into either GPT or Claude they turn it into an attachment, so it can happen. I think it's invisible to the model, but somehow not to the summarizer.
- DrammBA 6mo agoAnthropic always summarizes the reasoning output to prevent some distillation attacks
- nyc_data_geek1 6mo agoVery cool that these companies can scrape basically all extant human knowledge, utterly disregard IP/copyright/etc, and they cry foul when the tables turn.
- stavros 6mo agoYep, that is exactly what happens. It's a disgrace that their models aren't open, after training on everything humanity has preserved. They should at least release the weights of their old/deprecated models, but no, that would be losing money.
- copperx 6mo agoWe should treat LLM somewhat like patents or drugs. After 5 years or so, the models should become open source. Or at very least the weights. To compensate for the distilling of human knowledge.
- butlike 6mo agoAll extant human knowledge SO FAR. Remember, by the nature of the beast, the companies will always be operating in hindsight with outdated human knowledge.
- MasterScrat 6mo agoand so does OpenAI
- vintermann 6mo agoAttacks? That's a choice of words.
- DrammBA 6mo ago
- blazespin 6mo agoSafety versus Distillation, guess we see what's more important.
- MarkMarine 6mo agoAnthropic was chirping about Chinese model companies distilling Claude with the thinking traces, and then the thinking traces started to disappear. Looks like the output product and our understanding has been negatively affected but that pales in comparison with protecting the IP of the model I guess.
- andai 6mo agoWhen Gemini Pro came out, I found the thinking traces to be extremely valuable. Ironically, I found them much more readable than the final output. They were a structured, logical breakdown of the problem. The final output was a big blob of prose. They removed the traces a few weeks later.
- axpy906 6mo agoThat’s kind of funny since a Chinese model started the thinking chains being visible in Claude and OA in the first place.
- einrealist 6mo agoThey are trying to optimize the circus trick that 'reasoning' is. The economics still do not favor a viable business at these valuations or levels of cost subsidization. The amount of compute required to make 'reasoning' work or to have these incremental improvements is increasingly obfuscated in light of the IPO.