jychang

6 ms·

61.
by jychang 7mo ago
The free version of ChatGPT is insanely crippled, so that's not surprising.
62.
by jychang 7mo ago
That's completely not true. LLM on device would use MORE electricity. Service providers that do batch>1 inference are a lot more efficient per watt. Local inference can only do batch=1 inference, which is very inefficient.
63.
by jychang 7mo ago
Great! That's a good thing. Embrace being human sometimes. Plus, "lazy" would actually be just using AI to edit the writing.
64.
by jychang 7mo ago
Just wait until your ISP puts you behind a CGNAT. Or if you ever need to travel a lot and tether off your phone. Most mobile devices are IPV6 only (via 464XLAT) behind a CGNAT these days.
65.
by jychang 7mo ago
Being removed for versions 25 to 38… honestly confirms the feminist narrative of some people being idiots, though. Like, imagine documentation on object oriented programming being removed because it offended some functional programming folk
66.
by jychang 7mo ago
Most people are on a CGNAT these days, drowning in captchas is the new normal. You’re at the mercy of one of your neighbors not hosting a botnet from their home computer.
67.
by jychang 7mo ago
Throw it in a Thermoplastic Polyurethane (TPU) bag and it'll be a pretty good long term solution.
68.
by jychang 7mo ago
That's a foolish take. That's like gamers thinking most of Nvidia's revenue coming from gaming GPUs, so Nvidia should prioritize gamers. Inference is ruled by inference providers, not local. Local inference is a rounding erro
69.
by jychang 7mo ago
Correct, but note that's exactly what inference providers do. https://arxiv.org/pdf/2412.19437 > The minimum deployment unit of the decoding stage consists of 40 nodes with 320 GPUs. The attention part employs
70.
by jychang 7mo ago
Well, the actual inference providers put each expert on its own single GPU. Deepseek explicitly does this. Read-only parameters is also usually the majority of space. Deepseek is 700GB of params. Meanwhile kv cache is small (Deepseek is abo
71.
by jychang 7mo ago
Well, Deepseek batch sizes are something like 8192, so 128 isn't much. https://arxiv.org/html/2412.19437v1 "the batch size per expert is relatively small (usually within 256 tokens)"
72.
by jychang 7mo ago
I'm not sure that I buy their conclusion that more compute during inference is good. Yes, batch=1 inference is mostly memory bandwidth bound, not GPU compute bound. But no provider does batch=1 inference. Everyone groups all the reques
73.
by jychang 7mo ago
That's completely different. That's like saying you want to compare the Nvidia 5090 GPU to the latest Call of Duty.
74.
by jychang 7mo ago
1. Claiming that gpt-4o and gpt-4.5 came from the same training run is ridiculous, gpt-4.5 was not distilled from the same pretrain as 4o. - Mark Chen has literally publicly said as much, it's a completely different pretrain run. - And
75.
by jychang 7mo ago
The comment is phrased in the greater context of the public discussion about housing, in general. Not the specific information of the article. You know, like how a discussion about war might reference the various recent wars that everyone k
76.
by jychang 7mo ago
We already have this concept. It’s called user accounts. Your Gmail account vs my Gmail account. Your macOS account vs my macOS account. Yes, I can spam you from my Gmail. Yes, I can use sudo on my Mac and damage your account. But the impac
77.
Estimating the Size of Claude Opus 4.5/4.6(unexcitedneurons.substack.com)
1 points by jychang 7mo ago|0 comments
78.
by jychang 7mo ago
The numbers look about right. Opus 4.5 is about 1.5x the size of Sonnet 4.6, and Opus 4/4.1 is about 5x the size of Sonnet 4.5/4.6. Note that Opus 4.5 is about 1/3 the size of Opus 4/4.1 (and 1/3 the price in the AP
79.
by jychang 7mo ago
Well, for one, Anthropic mostly uses Google TPUs and Amazon Inferentia2 chips, not Nvidia NVL72s. That's because... Google and Amazon are major investors in Anthropic. Secondly, you missed out the entire AI industry trend in 2024-2025,
80.
by jychang 7mo ago
Yes, but I highly doubt they would increase sparsity much vs the chinese models. That's how you get Llama 4. Pretty much every major lab settled on ~3-5% sparsity for a reason.
81.
by jychang 7mo ago
No, Opus cannot be 10x larger than the chinese models. If Opus was 10x larger than the chinese models, then Google Vertex/Amazon Bedrock would serve it 10x slower than Deepseek/Kimi/etc. That's not the case. They're
82.
by jychang 7mo ago
Except there are providers that serve both chinese models AND opus as well. On the same hardware. Namely, Amazon Bedrock and Google Vertex. That means normalized infrastructure costs, normalized electricity costs, and normalized hardware pe
83.
by jychang 7mo ago
That doesn't work. Think about it a bit more. Hint: what's in the kv cache when you start processing the 2nd token? And that's called layer parallelism (as opposed to tensor parallelism). It allows you to run larger models (p
84.
by jychang 7mo ago
Nobody is running 10s of trillion param models in 2026. That's ridiculous. Opus is 2T-3T in size at most .
85.
by jychang 7mo ago
In practice, tps is a reflection of vram memory bandwidth during inference. So the tps tells you a lot about the hardware you're running on. Comparing tps ratios- by saying a model is roughly 2x faster or slower than another model- can
86.
by jychang 7mo ago
> but not sure how to figure out what it would cost and I'm sure as hell not going to try. Ask Opus to figure out how much it would cost. Lol.
87.
by jychang 7mo ago
Yep, you can also get similar analysis from Amazon Bedrock, which serves Opus as well. I'd say Opus is roughly 2x to 3x the price of the top Chinese models to serve, in reality.
88.
by jychang 7mo ago
> I find it likely Opus is larger. Unlikely. Amazon Bedrock serves Opus at 120tokens/sec. If you want to estimate "the actual price to serve Opus", a good rough estimate is to find the price max(Deepseek, Qwen, Kimi, GLM)
89.
by jychang 7mo ago
That's a tautology. People think chinese models are 10x more efficient because they're 10x cheaper, and then you use that to claim that they're 10x more efficient. Opus isn't that expensive to host. Look at Amazon Bedroc
90.
by jychang 7mo ago
Nope. Where do they say something along the lines of "we had MXFP4 tensors in our previous upload" or "that's why we re-uploaded new versions"? This is a famous non-apology non-explanation of what actually happened.

More ›