mmoskal

17 ms·

91.
by mmoskal 2y ago
LLMs used in writing of 1/3 of scientific papers. On the one hand scary, on the other it seems people find them useful. Maybe this AI thing is not like crypto after all...
92.
by mmoskal 2y ago
Not quite. Assume 1kW power consumption (with cooling). At $0.08/kWh (avarage US industrial rate) this is $700 per year. Adjust for more cooling etc and for say 5 years of usage but you still won't be anywhere near the $25k MSRP f
93.
by mmoskal 2y ago
It's funny because it's doubly exponential WRT code size. However it's also almost linear (n log*(n) due to union-find) WRT the size of the type and sane humans don't write programs with huge types.
94.
by mmoskal 2y ago
Definitely not. My understanding is they got significant market share on lunch - since it was during the pandemic chip shortage and they were the only ones with sub year lead times.
95.
by mmoskal 2y ago
The FPGA would be only to prove the transistor efficiency. If this works people will eventually do ASICs (possibly inside the GPUs a few generations ahead).
96.
by mmoskal 2y ago
Mobile comment view is messed up on Safari. The top level view seems to leave a bit too much margins on mobile.
97.
by mmoskal 2y ago
Assuming $30k GPU with 3yr deprecation, it's additionally $1.14/h. Much more than energy.
98.
by mmoskal 2y ago
I guess the short way to say it is that "undecidable" doesn't mean "it can't ever be decided", just not always. And of course all programs of practical significance are finite state machines (since there is onl
99.
by mmoskal 2y ago
One sentence from The Economist seems to explain more than TFA: "Microsoft reported a 31% increase in its indirect (Scope 3) emissions last year from building more data centres (including the carbon found in construction materials) as
100.
by mmoskal 2y ago
Projects change license for new code going forward. The old code remains available under the previous license (and sometimes new). Here, they are able to change the conditions for existing weights.
101.
by mmoskal 2y ago
From model card: Falcon2-11B was trained on 1024 A100 40GB GPUs for the majority of the training, using a 3D parallelism strategy (TP=8, PP=1, DP=128) combined with ZeRO and Flash-Attention 2. Doesn't say how long though.
102.
by mmoskal 2y ago
The post seems to be about execution speed though. However, even there it's definitely not #1 factor as witnessed by popularity of CPython...
103.
by mmoskal 2y ago
Survivor bias? maybe they just don't catch the ones that only had two suitcases of cash.
104.
by mmoskal 2y ago
TFA says you can teach it new facts, but it's very slow and makes the model hallucinate more.
105.
by mmoskal 2y ago
Yes. It's speculative decoding but instead of generating just a few sequential tokens with the draft model they generate a whole tree of some sort of optimal shape with hundreds of possible sequences. It ends up being somewhat faster
106.
by mmoskal 2y ago
A problem with this is that the inference has to be 100% deterministic. For example of you change tailing order of matmul you may get slightly different results. It's easy to imagine people changing tailing in llama.cpp or other librar
107.
by mmoskal 2y ago
Model checkers typically only explore a bounded number of states which is efficient at bug finding and often doesn't require additional annotations in the program. Automatic (SMT-based) verifiers like Verus, Dafny, F* (and my VCC :) re
108.
by mmoskal 2y ago
Every token generates a KV cache entry based on this token and all previous KV cache entries. This happens in every layer. KV cache is 100k-1MB per token so quite a bit. Edit: also you can forward generate 5 or 10 dots in batch without muc
109.
by mmoskal 2y ago
I guess easier to make the url be githubusercontent.com/whatever without mentioning repo name (or use uuid of repo).
110.
by mmoskal 3y ago
Interesting, the 8B model was trained for 1.3M hours, while the 70B for 6.4M hours at 700W. Assuming $0.05/kWh (WA price) it's $46k and $224k. Even allowing for cooling, CPUs, and more expensive power wherever they are running thi
111.
by mmoskal 3y ago
You can ask the model sth like: is xyz correct, answer with one word, either Yes or No. The log probs of the two tokens should represent how certain it is. However, apparently RLHF tuned models are worse at this than base models.
112.
by mmoskal 3y ago
I think it's a bit like saying microcode in CPU underpins high-level programming languages - kind of, but good luck understanding what a program does based on its microcode translation. OTOH you can mess with microcode and maybe get be
113.
by mmoskal 3y ago
It seems the trick here is they first quantize it to 1- or 2-bit, and then they fine-tune the quantization bias parameters (the parameters that dequantize from 1-2 to 16 bit) via LoRA. Then they have specialized kernels to do matrix multipl
114.
by mmoskal 3y ago
Looks like 1/10 of TOPS and 1/3 of mem bandwidth of Jetson Orin Nano (the TI chip seems able to run 3733MT/s at 32 bit, so 15GB/s) but much cheaper.
115.
by mmoskal 3y ago
Apparently this works very well, except for a single letter, Turkish I. Turkish has two version of 'i' and Unicode folks decided to use the Latin 'i' for lowercase dotted i, and Latin 'I' for uppercase dot-less
116.
by mmoskal 3y ago
This year 18/28 European NATO members will hit the 2% target. They will have to hit 3% or so even if Ukraine holds Russia off [0]. If the tanks roll to Polish and Baltic states borders it will be much more expensive. [0] https:/&
117.
by mmoskal 3y ago
> JIT compilers often use techniques, such as speculative optimizations, that make it difficult to reuse the compiler outside its original context, since they encode so many assumptions about the specific language for which they were des
118.
by mmoskal 3y ago
The most interesting bit is that this is RWKV model, meaning constant size state (no quadratic attention). AFAIK the biggest open weights non-transformer model.
119.
by mmoskal 3y ago
Oh come on! No one would ever do that!... Oh wait
120.
by mmoskal 3y ago
They say 200GB/s of mem bandwidth; Mixtral uses 13B parameters for inference; they claim 22t/s, so 22 * 13B parameters per second, so (200 * 8) bits / (22 * 13) around 5.6 bits / parameter max. With overheads, it's

More ›