spindump8930

7 ms·

61.
by spindump8930 1y ago
This paper is very cool, comes from respected authors, and is a very nice idea with good experiments (flop controlled for compute). It shouldn't be seen as a wall-breaking innovation though. From the paper: > Existing transformer li
62.
by spindump8930 1y ago
It's not clear how your perception of quantum computing would lead to 'faster horses' in the current view of NN architectures - keep mind that the common view of 'exploring many paths simultaneously' is at best an o
63.
by spindump8930 1y ago
You might be thinking of this article by Sebastian Ruder: https://www.ruder.io/nlp-imagenet/ Note that the author has a background spanning a lot of the timespans/topics discussed - much work in multilingual NLP,
64.
by spindump8930 1y ago
Yes, it could be stacked on quants. It might be that quantized activations already are more "dense" and so they can't be compressed as much (from 16 -> ~11 bits), but certainly possible.
65.
by spindump8930 1y ago
bf16 is the defacto default datatype and distribution type for LLMs, which are then often eagerly quantized by users with more limited hardware. See the recent Llama releases and e.g. the H100 spec sheet (advertised flops and metrics target
66.
by spindump8930 3y ago
Those sizes refer to the data before processing and filtering. The actual training size was about 3 TB: The Stack v2 is ten times larger than its predecessor, yielding a raw dataset of 67.5 TB. Through extensive cleaning, filtering, an
67.
by spindump8930 3y ago
Also good to note that that the Pile contains lots of curated sources and recent trends have been to take curated data sources and combine them with filtered webcrawls (i.e. commoncrawl with heavy processing). See dolma or the stack v2 (for