cs702

64 ms·

841.
by cs702 3y ago
You're talking about mapping floating-point vector representations, i.e., embeddings, computed by a pretrained LLM to binary vector representations, right? And you're talking about doing this by first having someone else 's
842.
by cs702 3y ago
It means "we don't see price gouging." I assume absence of cartel-like price coordination.
843.
by cs702 3y ago
"It Is Difficult to Get a Man to Understand Something When His Salary Depends Upon His Not Understanding It"[a] [a] https://quoteinvestigator.com/2017/11/30/salary/
844.
by cs702 3y ago
If I understand the authors correctly, they trained the compared models on only 100B tokens, all drawn from RedPajama, to make the comparisons apples-to-apples. That's sensible. It allows for easier replication of the results. Otherwis
845.
by cs702 3y ago
The activations are in (-1, 1), so they're also representable by (-1, 0, 1).
846.
by cs702 3y ago
Thank you. I think the paper as it is provides enough evidence to support the claims. If I understand the authors correctly, they trained the compared models on only 100B tokens, all drawn from RedPajama, to make the comparisons apples-to-a
847.
by cs702 3y ago
> Why is this so shocking? Quantization has been widely explored, driving that to its extreme (and blowing up parameter count to make up for it) just seems like a natural extension of that. I find it shocking that we don't even ne
848.
by cs702 3y ago
There are two findings I find shocking in this work: * In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1). * In matrix multiplications (e.g., weights
849.
by cs702 3y ago
Surge pricing works well only if there is real competition. Think: multiple competing machines selling bottled water/soda within a short walking distance. I'm not sure it would work well for consumers in a monopoly or duopoly. Thi
850.
by cs702 3y ago
Yes. I edited my comment to replace "never" with "rarely," which more closely reflects what I meant to write the first time around. Thank you!
851.
by cs702 3y ago
As this brilliant cartoon points out, we've had 128 years of warnings about human-induced climate change, and over all those years, humanity did not change course. That's because warnings and exhortations rarely accomplish anyth
852.
by cs702 3y ago
Great question, very much aligned with my thinking! Yes, assume they maintain their current experience, education, and general attitude towards each other.
853.
by cs702 3y ago
Whenever I read sad stories about poor countries like Haiti[a], I like to ponder a hypothetical scenario: If one were a god who could, with the snap of a finger, replace the country's entire population with an equal number of people, r
854.
by cs702 3y ago
...because solving the problem would make them unnecessary.
855.
by cs702 3y ago
True. In many cases they're acting like patsies... because they believe they have no other choice!
856.
What Texas can teach San Francisco and London about building houses(ft.com)
4 points by cs702 3y ago|3 comments
857.
by cs702 3y ago
Not necessarily: https://www.reddit.com/r/MachineLearning/comments/1amb3xu/d_...
858.
by cs702 3y ago
LoL. I never thought someone would pay to create and show an ad acknowledging that almost no one in the corporate world has a clue about any of it! If you remember the name of the company that paid for that ad, please share it!
859.
by cs702 3y ago
I never said most companies are buying GPU's from Nvidia. What I said is that they're spending money, for a variety of AI and AI-related services provided by third parties. Those third parties, in turn, are buying GPU's.
860.
by cs702 3y ago
Thank you for that. I needed the laugh :-)
861.
by cs702 3y ago
My post was not meant to be patronizing. Sorry if it came across as such! I have no doubt that society will benefit from all that spending... but I'm much less certain about the fate of all the companies incurring all that spending!
862.
by cs702 3y ago
Every non-tech company in the world is trying to figure out an "AI strategy," but most executives have no clue as to what they're doing or ought to be doing. They're like patsies with money joining a game of poker they d
863.
by cs702 3y ago
> I'm not sure I understand the intended interpretation of this. Concretely speaking, if it cost CoolAI 100k seconds of compute to process a sequence of length 100k, it would not take them 1 second now. Cost would be lower by a fact
864.
by cs702 3y ago
I'm the author of the parent comment, and just saw your response. First of all, by "a factor of 100,000×" I mean a model-specific factor , call it c , of 100,000x, but looking at what I wrote I can see how it could be misin
865.
by cs702 3y ago
Hmm... You're right: It's quite possible that the new crop of linear RNNs may not be able to solve the same problems as Transformers. It could also be that we will have to increase the depth of these linear RNNs (possibly even
866.
by cs702 3y ago
Thank you. Right now, my puny little brain is searching through what I would describe as "those distant and slightly vague memories" that make up my poor man's knowledge of computational complexity theory. > Attention, wh
867.
by cs702 3y ago
Well, it would depend on how many tokens from the vocabulary are required, in the worst case, to solve NP problems. Think of the tokens as symbols, and of the vocabulary as a finite alphabet of symbols, in a formal system. In other words, n
868.
by cs702 3y ago
I would add to the list a few questions about the evolution of cost dynamics going forward, given the advent of new sequence modeling architectures. As we all know, Transformers are very expensive to train and run because their compute co
869.
by cs702 3y ago
The key takeaway for me: Whether data is continuous or discrete, no matter its modality (text, video, music, etc.), we now have an array of proven methods for representing it with discrete tokens, enabling us to use existing sequence mode
870.
by cs702 3y ago
I'm partial to Large World Model (LWM). Large Sequence Model (LSM) is OK too.

More ›