qeternity

8 ms·

91.
by qeternity 1y ago
Second line of the post: > The main objective is to learn writing attention in CUDA C++, since many features are not available in Triton, such as MXFP8 / NVFP4 MMA for sm120.
92.
by qeternity 1y ago
FAANG has been replaced by Mag7: Alphabet, Amazon, Apple, Broadcom, Meta, Microsoft, and Nvidia.
93.
by qeternity 1y ago
> Claude Code Max is affordable with Opus Because Anthropic is presumably massively subsidizing the usage.
94.
by qeternity 1y ago
> but forget cost if we want to compare solely the quality I think this is the whole reason not to compare it to Opus...
95.
by qeternity 1y ago
120B MoE. The 20B is dense. As far as dense models go, it’s larger than many but Mistral has released multiple 120B dense models, not to mention Llama3 405B.
96.
by qeternity 1y ago
I think this is overselling most professors.
97.
by qeternity 1y ago
Take the API and assume 24/7 usage (or whatever working hours are). That’s your fixed cost. It’s more likely that this sum is higher than they want. So really it’s not about predictability.
98.
by qeternity 1y ago
In shambles compared to when? Quality of life is the highest it's ever been across socioeconomic strata. It's just our expectations outpace reality.
99.
by qeternity 1y ago
> I don’t know how you got to the conclusion that they only used SRAM. Because they are doing 1,500 tokens per second.
100.
by qeternity 1y ago
> if they result in a greater return. Greater return than what and to whom? We already have existing labor markets that are very capable of determining returns.
101.
by qeternity 1y ago
You can rent H100s for $1.50/gpu/hr these days.
102.
by qeternity 1y ago
This is not a result of income inequality. This is a result of measures intended to reduce income inequality, which of course almost universally make things worse. The other lending that the US (and Western governments in general) massive
103.
by qeternity 1y ago
Ok? I agree with you. It doesn't change anything I said.
104.
by qeternity 1y ago
It's almost as if geopolitical environments evolve over time. I don't think anyone today would feel any differently about preventing Pakistan from attaining nuclear weapons. Unfortunately it's very difficult to take nukes a
105.
by qeternity 1y ago
People really need to stop glossing over the very real differences between state controlled media, and media that you think is aligned with a certain political group. You can believe Fox News is the worst entity in human history, but Fox Ne
106.
by qeternity 1y ago
> "Fine-tuning LLMs for knowledge injection is a waste of time" is true, but IDK who's trying to do that. Have people who say this ever actually done it? It works. It works pretty well. I have no clue why this bad advice i
107.
by qeternity 1y ago
> How is power discarded It isn’t, not at scale in any traditional sense.
108.
by qeternity 1y ago
> LLMs are text in -> text out, and you can drop in a new LLM and replace them without changing anything else. If anything, newer LLMs will just have more capabilities. I don't mean to be too pointed here, but it doesn't sou
109.
by qeternity 1y ago
> the old one is completely valueless to them This is of course untrue for the same reason that people are still running Windows 2000.
110.
by qeternity 1y ago
I think you mean non-deterministic, instead of probabilistic. And there is no reason that these models need to be non-deterministic.
111.
by qeternity 1y ago
Append-only would imply yes. There is no overwriting in append-only. There is only truncate and append.
112.
by qeternity 1y ago
Ok. It's more revealing that the rest of you are so offended by the truth.
113.
by qeternity 1y ago
It's the same process as regular training, but instead of a single token for your cost function, it's top-k logits.
114.
by qeternity 1y ago
It also appears that this was a sampling benchmark...which is not representative. Generation benchmark was 5% faster than SGLang.
115.
by qeternity 1y ago
> At even a $300B valuation a typical wall street analysts would want to value them at 2x sales which would mean they'd expect OpenAI to have $600B in annual sales to account for this valuation when they go public. Lmfao where did y
116.
by qeternity 1y ago
I know. That was my point. Throughput doesn't scale on CPU as well as it does on GPU.
117.
by qeternity 1y ago
It's not impressive nor efficient when you consider batch sizes > 1.
118.
by qeternity 1y ago
Attention is not the primary inference bottleneck. For each token you have to load all of the weights (or activated weights) from memory. This is why Cerebras is fast: they have huge memory bandwidth.
119.
by qeternity 1y ago
Mistral have a long history of open weight models...
120.
by qeternity 1y ago
The model is the abstraction.

More ›