cs702

60 ms·

811.
by cs702 3y ago
Yes, that sounds right, but it doesn't make the work less worthwhile or less interesting. The authors do interesting things with the NFM, including explaining why pruning should even be possible and why we see grokking during learning.
812.
by cs702 3y ago
Interesting: Given an input x to a layer f(x)=Wσ(x), where σ is an activation function and W is a weight matrix, the authors define the layer's "neural feature matrix" (NFM) as WᵀW, and show that throughout training, it remai
813.
by cs702 3y ago
Thank you for that ! I needed the laugh!
814.
by cs702 3y ago
Saudi Aramco's CEO sure seems afraid of the future. Alas, exhortations almost never work. If other sources of energy, like solar, become superior to and cheaper than oil, society will choose them over oil. Superior products at cheape
815.
by cs702 3y ago
TL;DR: * Scale the time series data and quantize the floating point values into B bins. * Each bin becomes a corresponding token id in a vocabulary of B embeddings. * Train a small LLM to predict the next token id given a sequence of token
816.
by cs702 3y ago
The most familiar version is from Pulp Fiction: https://www.youtube.com/watch?v=1hLIXrlpRe8 Brilliantly transformed by the Black Eyed Peas: https://www.youtube.com/watch?v=ZaI2IlHwmgQ Both are based on this
817.
by cs702 3y ago
According to the company, the new chip will enable training of AI models with up to 24 trillion parameters. Let me repeat that, in case you're as excited as I am: 24. Trillion. Parameters. For comparison, the largest AI models curren
818.
by cs702 3y ago
Thank you for the context. I will hold off on passing judgment for now, even though I too share the popular opinion of Boeing's current (lack of) engineering culture.
819.
by cs702 3y ago
...and most of those startups are hoping they won't have to raise cash at today's lower valuations.
820.
by cs702 3y ago
You're right. I edited my comment to qualify it with "when". Thank you!
821.
by cs702 3y ago
It's still significant. When the Softmax head is the transpose of the embedding matrix, the proposed method enables extraction of the entire matrix of pretrained token embeddings from a black-box model at a shockingly low cost. If I
822.
by cs702 3y ago
Looks great on paper, but several things give me pause: * Rivian expects deliveries to start in 2026, two years from now! * Rivian expects the R2 will start at $45K. Usually, starting prices are for vehicles with less range, slower acce
823.
by cs702 3y ago
I barely made it through the abstract and introduction before realizing that I too don't have enough "brain jam to spread" on this at the moment. This paper isn't exactly, you know, light reading.
824.
by cs702 3y ago
The largest AI models currently in use have around 0.5 trillion connections between artificial neurons. Each connection is represented and quantified by a weight parameter, i.e., a single floating-point number that is learnable in training.
825.
by cs702 3y ago
Company execs don't want to raise money at today's lower valuations, because it signals desperation and hurts employee morale. Execs would rather borrow money in the form of "bridge financing." Many execs are crossing th
826.
Tech writer Kara Swisher has a new book. Enter the AI-generated scams(washingtonpost.com)
3 points by cs702 3y ago|0 comments
827.
by cs702 3y ago
I still cannot believe the last photo in the article, showing him in his current disguise as an orthodox priest.
828.
by cs702 3y ago
Because it's not just the use ternary values. It's also that there are no dot-products; there are only additions. And when we apply both changes to existing LLMs, there's no performance degradation (as tested by the authors).
829.
by cs702 3y ago
The domain fees now exceed $30M/year and population is under 16K, so call it $2K/year per capita. Assuming average family size of 3.5, it's $7K/year per family. That's real money, in exchange for doing nothing. Angu
830.
by cs702 3y ago
> I don't see custom hardware for this as a short term thing - they need to prove out the quantization concept more first. Yes, I agree. This still needs to be more extensively tested.
831.
by cs702 3y ago
EDIT: Embarrassingly, on the last paragraph I got the number of possible input pairs wrong: > only three possible input pairs, (a,b), (a,c), or (b,c) (within each pair, order doesn't matter) The correct number, ignoring order, is
832.
by cs702 3y ago
Yes. More generally, this will enable implementation via crazy-cheap bit-wise ops in binary hardware, and possibly, maybe, via crazy-cheap trit-wise ops in ternary hardware that manipulates ternary digits, or trits. Note that any binary o
833.
by cs702 3y ago
Thank you. Others on this thread have addressed the citation-trail issues you raise. I just want to tell you how helpful I find your comment about why ternary weights ought to work at all without degrading performance: > My best guess
834.
by cs702 3y ago
You're right. Thank you for pointing that out!
835.
by cs702 3y ago
You're right. Thank you for pointing that out.
836.
by cs702 3y ago
Thank you. Your key point -- that so far all models with the proposed methods may have been only "grossly trained" -- is compelling. If I understand the authors correctly, they trained the compared models on only 100B tokens, all
837.
by cs702 3y ago
...and 8x where it was 72 hours ago.
838.
by cs702 3y ago
Bitcoin's price jumped past $60K today. I'd imagine traffic to Coinbase's servers looks like this: | | | | _/\ /\_/\ _| \/\&
839.
by cs702 3y ago
> So from the perspective that it's all just bits in the end the only thing that is interesting is how useful it is to arrange those bits into trits for this particular algorithm, and that the algorithm seems to be able to use thin
840.
by cs702 3y ago
Others have already made the same point in this thread. See my response here: https://news.ycombinator.com/item?id=39539508

More ›