kouteiheika

7 ms·

91.
by kouteiheika 6mo ago
> This is almost certainly wrong. So how would you explain the increase in token usage, considering the fact that conventionally tokenizers are trained to minimize the token usage within a given vocabulary budget? > Putting an inducti
92.
by kouteiheika 6mo ago
I'd guess it's because they don't want people to reverse engineer it. Note that they're the only provider which doesn't make their tokenizer available offline as a library (i.e. the only provider whose tokenizer is
93.
by kouteiheika 6mo ago
> Opus 4.7 tokenizer used 1.46x the number of tokens as Opus 4.6 Interesting. Unfortunately Anthropic doesn't actually share their tokenizer, but my educated guess is that they might have made the tokenizer more semantically aware t
94.
by kouteiheika 7mo ago
> Activation would still require gigabytes for a few kb context. For that you use activation checkpointing, and you can also offload that to the CPU in a smart way to hide the latency. Although, yes, for long context training the activat
95.
by kouteiheika 7mo ago
I have no idea; you have to check their docs. AFAIK what they do is that they calculate a hash of the true thinking trace, save it into a database, and only send those hashes back to you (try to man-in-the-middle Claude Code and you'll
96.
by kouteiheika 7mo ago
Yes. Although this wasn't integrated into PyTorch itself (but to torchtune, which is a different thing). If you're writing your own training loop you need to use a third-party kernel, e.g. the Liger kernel mentioned in the article
97.
by kouteiheika 7mo ago
> You can only give it a try, but don't get your hopes high on a large context. You may or may not know this, but: when training off-the-shelf LLMs (i.e. ones which have a huge vocabulary) what consumes a huge amount of memory usa
98.
by kouteiheika 7mo ago
This isn't really anything new; I've been doing something like this for quite a while, I just haven't bothered writing a paper. (: Probably anyone who would seriously tackle the problem of "how do I train a huge model on
99.
by kouteiheika 7mo ago
> Can I just see the actual thinking (not summarized) so that I can see the actual thinking without a latency cost? You can't, and Anthropic will never allow it since it allows others to more easily distill Claude (i.e. "distil
100.
by kouteiheika 7mo ago
You can, but you lose access to anything that was associated with your old account. Another fun thing Google did is to automatically (without my consent) add a required second-factor authentication to my current Google account. I have thi
101.
by kouteiheika 7mo ago
This is not a hypothetical problem and you don't need to be deliberately targeted. It actually happens to normal people. And if it does you have absolutely zero recourse. Source: I have a banned Google account (it's over 20 year
102.
by kouteiheika 7mo ago
> Are they actually doing this? The stuff that Anthropic has been saying about the deliberate use of XML-style markup makes me wonder a bit. Yes. The XML-style markup are not special tokens, and are usually not even single-token; usually
103.
by kouteiheika 7mo ago
In theory you still use the same blob (i.e. the prompt) to tell the model what to do, but practically it pretty much stops becoming an in-band signal, so no. As I said, the best way to do this is to inject a brand new special token into the
104.
by kouteiheika 7mo ago
There is one way to practically guarantee than no prompt injection is possible, but it's somewhat situational - by finetuning the model on your specific, single task. For example, let's say you want to use an LLM for machine trans
105.
by kouteiheika 7mo ago
What OP's saying is fundamentally true though? Unfortunately most people don't really care about privacy, regardless of whether it's going to an American company or a Chinese one.
106.
by kouteiheika 7mo ago
Although it might not satisfy FSF there is a very simple way to do it - commit to release your models for free X months after they're first made available.
107.
by kouteiheika 8mo ago
> Anthropic were very vocal, well before this happened, that they were against the use case. > I don't blame them. These use cases are like blaming MySQL for storing the lat/long of the school. AI can't be held accounta
108.
by kouteiheika 8mo ago
> Which is why I was proposing a middle-ground where an agreement is setup between the model training company and the collective of developers/artists et all and come up with a license agreement where they are rewarded for their ori
109.
by kouteiheika 8mo ago
> I would go so far as to say the most restrictive license that the model is trained on should be applied to all model generated code. That license is called "All Rights Reserved", in which case you wouldn't be able to leg
110.
by kouteiheika 8mo ago
Help kill people[1]? [1] -- https://edition.cnn.com/videos/business/2020/07/24/thiel-pal...
111.
by kouteiheika 8mo ago
Direct quote from their CEO: > Our product is used on occasion to kill people. Doesn't get any more clear than this.
112.
by kouteiheika 8mo ago
Ah, right, by being actually good, as in - being okay with mass surveillance as long as it isn't being done in the US, being okay with Claude assisting in killing people as long as it isn't fully autonomous, and being actively hos
113.
by kouteiheika 8mo ago
> There are several open source models with no built in (or trivial to ecape) safeguards. You are underestimating this. It's almost trivial to remove the safeguards for any open-weight model currently available. I myself (a random
114.
by kouteiheika 8mo ago
> The main explanation of bottleneck is wrong: it’s the model weights which dominate memory bandwidth (and hence why batching multiple requests in a single pass increases total throughput). If copy user tokens was the bottle neck, batchi
115.
by kouteiheika 8mo ago
I would give my left kidney for either a continuation or a reboot of Babylon 5 under the helm of J. Michael Straczynski with full creative freedom. Or hell, even an entirely different show. In my opinion he's one of the few people in t
116.
by kouteiheika 9mo ago
No. I'm just saying that people should be consistent and if they apply a certain standard to Grok then they should also apply the same standard to other things. Be consistent. Meanwhile what I commonly see is people dunking on anything
117.
by kouteiheika 9mo ago
I'm not talking about video editing software; that's a different class of software. I'm talking about other generative AI models, which you can download today onto your computer, and have it do the same thing as Grok does. &g
118.
by kouteiheika 9mo ago
This is really amusing to watch, because everything that Grok is accused of is something which you can also trigger in currently available open-weight models (if you know what you're doing). There's nothing special about Grok in
119.
by kouteiheika 9mo ago
> I don't really get how VSCode got so popular. You can use a language server perfectly easily with Vim, Emacs, Helix, Sublime, etc. You open it. It just works. And the learning curve is smooth. Compare this to Vim where, if it&#x27
120.
by kouteiheika 9mo ago
> This is why my heart sank this morning. I have spent over a year training 4.0 to just about be helpful enough to get me an extra 1-2 hours a day of productivity. Maybe you should consider basing your workflows on open-weight models ins

More ›