sanxiyn

54 ms·

511.
by sanxiyn 3y ago
No, distilling step-by-step https://arxiv.org/abs/2305.02301 distills LLM to task specific model. That works, and I know of multiple successes. But it doesn't relate to choice of optimizing training FLOP vs traini
512.
by sanxiyn 3y ago
I agree distillation is the wild card. The question is whether distillation works for LLM. I am not aware of any public report of successful distillation of LLM (I searched quite hard for this; if you know of any and can tell me I would be
513.
by sanxiyn 3y ago
Eh, that's why the second check I mentioned is there... To see what the neuron is doing in relation to the rest of the network.
514.
by sanxiyn 3y ago
Theoretical advances are constant (just check the literature), but two particular advances in the last decade are of practical importance: one for sort and one for hash table. As in: check the implementation you are using, if it doesn'
515.
by sanxiyn 3y ago
> How do we know if the explainer is good? The paper explains this in detail, but here is a summary: an explanation is good if you can recover actual neuron behavior from the explanation. They ask GPT-4 to guess neuron activation given a
516.
by sanxiyn 3y ago
The article says: > The chip is being manufactured by Globalfoundries on the 12-nanometer low-power process. As far as I can tell it is not radiation hardened yet. I think what the article is trying to say is that since it is an open sou
517.
by sanxiyn 3y ago
It is a set of extensions to the RISC-V instruction set. Extensions (Xssr, Xfrep, Xdma) are documented here: https://pulp-platform.github.io/snitch/rm/custom_instruction...
518.
by sanxiyn 3y ago
Yes, TCP header had 16 bits allocated to "Urgent Pointer". It's (unfortunately) still there for compatibility, but RFC 9293, which updated RFC 793 after more than 40 years, has the following to say about it: > As a result
519.
by sanxiyn 3y ago
Maybe the situation has changed recently, but as I understand there is no scholarly consensus on when plate tectonics started and it is the topic of active debate. For example Onset of Plate Tectonics (2011) https://www.science.o
520.
by sanxiyn 3y ago
> Historically, copyright law has not been updated a lot to deal with any kind of technical change. Ok, not a lot, but it has been updated. Chip mask is a good example: https://en.wikipedia.org/wiki/Semiconductor_Chi
521.
by sanxiyn 3y ago
LLMs totally do, and the fact can be confirmed by experiment. See "Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4": https://arxiv.org/abs/2305.00118
522.
by sanxiyn 3y ago
The Stack's removal request form is currently processed manually.
523.
by sanxiyn 3y ago
I think fighting this is very worthwhile. For example, I think the fight is the reason why The Stack, dataset produced from GitHub, has a form to request to remove code. https://www.bigcode-project.org/docs/about/t
524.
by sanxiyn 3y ago
Re: graphite story. Wikipedia citation is to a biography of Szilard, and it says: > Fermi and Szilard met with representatives of National Carbon Company, who manufactured graphite, and Szilard made another important discovery. He asked
525.
by sanxiyn 3y ago
I do not get why you do not get the appeal. Do you want to die?
526.
by sanxiyn 3y ago
Theoretically speaking, not all water use is renewable like rainfall, you can in fact deplete aquifers, and ancient near east in fact desertified due to unsustainable water use. Ditto for land use, you can in fact erode and degrade topsoil,
527.
by sanxiyn 3y ago
> Of the top 20 region-sectors ranked by environmental impacts, none would be profitable if environmental costs were fully integrated. Ponder that for a moment: None of the world's top industrial sectors would be profitable if they
528.
by sanxiyn 3y ago
They are probably not lying, but good performance on benchmarks does not imply good performance on your use cases.
529.
by sanxiyn 3y ago
This is an interesting empirical study indeed. Direct link to the study: https://www.nber.org/papers/w31161
530.
by sanxiyn 4y ago
If you are interested in grokking you should have a look at this: Progress measures for grokking via mechanistic interpretability. https://arxiv.org/abs/2301.05217
531.
by sanxiyn 4y ago
I note that Rust already has been reimplemented (mrustc), mostly by one person.
532.
by sanxiyn 4y ago
It also cannot generate inappropriate or offensive content. https://twitter.com/conspirator0/status/1647671394476478467
533.
by sanxiyn 4y ago
I don't think your "synthetic data on ImageNet" reference shows "synthetic data is already very effective". Since many people won't read the paper, here's what it says: Training ResNet-50 on real ImageNet
534.
by sanxiyn 4y ago
AlphaZero in fact improves based on its own output, but I agree it is a special case and probably not generalizable.
535.
by sanxiyn 4y ago
用于生成式人工智能产品的预训练, 优化训练数据, 应满足以下要求: 不含有侵犯知识产权的内容. China is getting this right.
536.
by sanxiyn 4y ago
I guess asking "can we use an LLM here" is natural, but how often is the answer yes?
537.
by sanxiyn 4y ago
writing-mode support is pretty good these days. Here is another website in the wild using it for vertical text, mixed with horizontal text: http://ko.scp-wiki.net/gumiho
538.
by sanxiyn 4y ago
China is demonstrably capable of training GLM-130B. Skills involved seem comparable to Stability AI. https://keg.cs.tsinghua.edu.cn/glm-130b/
539.
by sanxiyn 4y ago
No. OpenAI haven't disclosed parameter count of GPT-3.5 or GPT-4, which are models used by ChatGPT. You may be thinking of GPT-3, which is indeed a 175B parameter model.
540.
by sanxiyn 4y ago
It doesn't mean anything. The letter says: > Therefore, we call on all AI labs to immediately pause for at least 6 months the training of AI systems more powerful than GPT-4. StableLM is not an AI system more powerful than GPT-4, so

More ›