cs702

61 ms·

781.
by cs702 3y ago
You're right. I changed it. Thanks!
782.
by cs702 3y ago
Wow, 128 experts in a single model. That's a lot more than everyone else. The Snowflake team has a blog post explaining why they did that: https://www.snowflake.com/blog/arctic-open-efficient-foundat... But the mo
783.
by cs702 3y ago
This is alarmist linkbait that fails to provide any context. The follow plot shows interest as a percent of GDP -- that is, in relation to the size of the economy, which is how we should all look at this: https://fred.stlouisfed.
784.
How to fix college finances? Eliminate faculty, then students. (Satire)(washingtonpost.com)
1 points by cs702 3y ago|0 comments
785.
by cs702 3y ago
"A person with a watch knows what time it is. A person with two watches is never sure." Not quite. A more accurate version would be: "A person with a watch thinks they know what time it is, but they are wrong. A person with t
786.
Four of the Five Least Expensive Car Brands to Maintain Are American(consumerreports.org)
2 points by cs702 3y ago|0 comments
787.
by cs702 3y ago
Moravec's plots (from 1998!) are looking more and more like prophecies: https://www.jetpress.org/volume1/power_075.jpg https://www.jetpress.org/volume1/All_things_075.jpg
788.
When will computer hardware match the human brain? (1998)(jetpress.org)
101 points by cs702 3y ago|126 comments
789.
by cs702 3y ago
> The best is not always popular. I agree. Network effects routinely overpower better technology.
790.
by cs702 3y ago
That's my experience as well. PyTorch dominates the ecosystem. Which is a shame, because JAX's approach is superior.[a] --- [a] In my experience, anytime I've have to do anything in PyTorch that isn't well supported ou
791.
by cs702 3y ago
He's certainly smart enough to contribute to the work, if he had the time and wanted to do it. But in all likelihood his contribution to Llama 3 consisted of... approving Llama 3.
792.
by cs702 3y ago
Interesting. There's a lot of preprocessing involved in getting these toy models to work, but the approach seems promising. Is there code available, so others can examine it and replicate the results? I couldn't find a link.
793.
by cs702 3y ago
> Wasn't your question about how independent the data is? No. My original (top) comment was about how the number of training tokens required per parameter slowly declines as models become larger. dzdt suggested it could be because
794.
by cs702 3y ago
Thank you for sharing this. I upvoted the OP so your comment would get more views!
795.
by cs702 3y ago
And it's only going to get faster, better, easier, cheaper.[a] Meanwhile, yesterday my credit card company asked me if I wanted to use voice authentication for verifying my identity "more securely" on the phone. Surely the co
796.
by cs702 3y ago
Thanks for that. I laughed out loud at "far above or below us".
797.
by cs702 3y ago
See also https://news.ycombinator.com/item?id=40040100 To repeat what I wrote on that post, this is all 100% explainable by Tesla's sacred five-step algorithm for manufacturing[a]: 1. Make the requirements less dumb: &
798.
by cs702 3y ago
I agree, but my question was not whether distance between data points tends to decrease as dataset size grows, but whether that is the reason why the number of training tokens required per parameter declines. It could be, but proving it w
799.
by cs702 3y ago
The authors claim to have reduced the number of iterations necessary for sampling from ~25 to 3, while improving image quality. The sample images look amazing. I've added this to my (very long) reading list.
800.
by cs702 3y ago
Yes, could be. Not sure how or even if anyone could prove it, though.
801.
by cs702 3y ago
Interesting! If the authors are right, it seems that the number of training tokens required per parameter (slowly) declines as models become larger (Figure 5). That's good news. I think it deserves wider dissemination, so I'm up
802.
by cs702 3y ago
Yes, but the claim is about "unlimited context length." I doubt attention over each segment can be as good at recall as attention over the full input context.
803.
by cs702 3y ago
Those benchmarks are about long context. The question isn't about long context. It's about recall , or the ability to fetch and repeat parts of the input context. There's nothing about recall on section 4.3.
804.
by cs702 3y ago
The two issues I see with this work: * There's no mention of model performance on recall tasks . Models with attention do well on recall tasks, but models without it, like this one, tend to do poorly.[a] What is the performance of thi
805.
by cs702 3y ago
Fantastic work by Grant Sanderson, as usual. Attention has won .[a] It deserves to be more widely understood. --- [a] Nothing has outperformed attention so far, not even Mamba: https://arxiv.org/abs/2402.01032
806.
by cs702 3y ago
100% explainable by Tesla's five principles of manufacturing: 1. Make the requirements less dumb: "All designs are wrong, it’s just a matter of how wrong." - Musk 2. Try and delete parts (that seem unnecessary): "If part
807.
by cs702 3y ago
I'd be less sure, because China has been moving towards a more centrally planned approach to managing winners and losers in the economy. Central planning has a lousy track record.
808.
by cs702 3y ago
Mamba came out of the same research group, Hazy Research, led by Chris Ré. This new "Jamba" model incorporating Mamba and dot-product attention layers has ~8x more parameters than the largest open Striped Hyena, and appears to wor
809.
by cs702 3y ago
Please link to the original post: https://www.ai21.com/blog/announcing-jamba Jamba looks fabulous . Good performance for its size and much more efficient than the available open alternatives. The key idea: One of ou
810.
by cs702 3y ago
TL;DR: The authors embed task instructions in a vector space with a language model, and train a sensorimotor-controlling model on top to perform tasks given the instruction embeddings. The authors find that the models generalize to previous

More ›