yorwba

16 ms·

361.
by yorwba 6mo ago
I think it's a solid theoretical contribution, but it might nonetheless fail to have practical relevance if some of their assumptions and approximations turn out to be too unrealistic. One way this could happen, for example, would be i
362.
by yorwba 6mo ago
You don't need to understand that part of the derivation to implement it. You just need Algorithm 1 on page 33 of the paper. Or look at the author's implementation: https://github.com/elonlit/PopRiskMinimizati
363.
by yorwba 6mo ago
The handwaving required is just to assume a diagonal preconditioner, and the optimal preconditioner under that constraint corresponds to the new update rule. (See section F of the paper.) And of course a diagonal preconditioner works on the
364.
by yorwba 6mo ago
To get pure grokking, you need a model large enough to easily memorize the entire training data and keep training for a long time after memorization. In practice, you'll probably use a more realistically-sized model that might grok on
365.
by yorwba 6mo ago
What is classic about "skip updating parameters with high gradient/loss variance in multiple batches/samples"? Do you have a particular algorithm in mind that uses this heuristic?
366.
by yorwba 6mo ago
There are similar unfounded complaints about US companies. It's just what happens with a product that doesn't work 100% of the time even under ideal conditions. Some people are bound to get unlucky but blame it on deliberate actio
367.
by yorwba 6mo ago
I don't think anyone in this thread made that particular argument.
368.
by yorwba 6mo ago
Not every site, just the ones using DNSSEC. Clearly, denic.de was online, for instance.
369.
by yorwba 6mo ago
From [0]: "In fiscal year 2017, there were 80 copyright/trademark infringement offenders who accounted for 0.1% of all offenders sentenced under the guidelines." This is such a low number that I assume most prosecuted cases a
370.
by yorwba 6mo ago
None of this has anything to do with the paper, which is concerned with theoretical computer science and constructs artificial "languages" that have a small representation as a(n idealized theoretical) transformer but whose smalle
371.
by yorwba 6mo ago
Even if you spread out reformatting over multiple commits in different PRs, you can still make use of blame.ignoreRevsFile as long as your pull request workflow doesn't enforce squash merges even when somebody took extra care to produc
372.
by yorwba 6mo ago
The attention residuals paper uses attention across layers for the same token, in addition to the usual case of attention across tokens within the same layer, but it doesn't do anything to address the "lost in too much context&quo
373.
by yorwba 6mo ago
You might also be interested in the writeup by the person who reverse-engineered the original dictionary file format: https://digitalnk.com/blog/2020/05/08/porting-north-korean-d...
374.
by yorwba 6mo ago
FWIW, North Koreans also use lots of English-derived loanwords for computer topics, but they may be different from those in the south. Some time ago I made a search interface for a North Korean English dictionary if you want to poke around:
375.
by yorwba 6mo ago
FYI, towardsdatascience has been banned on HN since about 2020 due to its general spamminess: https://news.ycombinator.com/from?site=towardsdatascience.co... (turn on "showdead" in your settings if you don't
376.
by yorwba 6mo ago
The reason nobody is pricing Kimi K2.6 at half the cost of the best closed source models is that there are too many providers of the same model, so the competition drives prices down and they have to charge far less than that. https:/
377.
by yorwba 6mo ago
There are objective ways to compare models. They involve repeated sampling and statistical analysis to determine whether the results are likely to hold up in the future or whether they're just a fluke. If you fine-tune each model to ac
378.
by yorwba 6mo ago
Your source cites https://www.washingtonpost.com/technology/2024/09/18/energy-... which in turn claims to be based on https://arxiv.org/abs/2304.03271 but uses 0.14 kWh as the energ
379.
by yorwba 6mo ago
The actual source https://www.theinformation.com/newsletters/applied-ai/uber-c... says "about 11% of real, live updates to the code in its backend systems are being written by AI agents built primarily with C
380.
by yorwba 6mo ago
It wasn't flagged. Compare to this comment by the same user that was actually flagged: https://news.ycombinator.com/item?id=47954834 Note the part where it says [flagged] [dead] instead of just [dead].
381.
by yorwba 6mo ago
Everybody training models on large amounts of lightly filtered internet text is partially distilling every other model that had its output posted verbatim to the internet.
382.
by yorwba 6mo ago
Not owning the rights to some content and somebody else owning those rights are not the same thing. If someone else owns the copyright and you redistribute their stuff without permission, they have grounds to sue you. If nobody owns the c
383.
by yorwba 6mo ago
It's not like Tariq ibn Ziyad invented the concept of intentionally making a retreat impossible in order to compel soldiers to fight. There are proverbs about this kind of thing that predate him by centuries: https://en.wikt
384.
by yorwba 6mo ago
You could try a sequential testing setup, which can let you stop the experiment earlier if the difference is larger than expected. But if the difference is small, there's no way around the fact that reliably detecting small differences
385.
by yorwba 6mo ago
To be useful for identifying which model is better, benchmark scores only need to correlate with true performance, for which it's enough that the majority of tasks are scored correctly. You could have a terrible benchmark where 49% of
386.
by yorwba 6mo ago
How many samples you need depends on the difference you want to be able to measure (0% to 1% is different from 50% to 51% is different from 0% to 10% is different from 50% to 60%), the significance level at which you will declare a differen
387.
by yorwba 6mo ago
Why 45 times in particular? If you want 80% power to distinguish a model at 50% from a model at 51%, you need 39,440 samples per model, or 329 samples per question per model. But that would just give you a more precise estimate of how well
388.
by yorwba 6mo ago
https://jack-clark.net/
389.
by yorwba 6mo ago
The technical report discussing the why and how is here: https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/blob/main...
390.
by yorwba 6mo ago
Donut's website claims that they will release the next test result in 7 days, so you can check next week whether those people will have something new to talk about: https://idonutbelieve.com

More ›