liuliu

14 ms·

211.
by liuliu 2y ago
> I think they could argue for greater security architecture (i.e. part of NATO but no nukes) to prevent repeat. But otherwise I can appreciate why the current administration is where its at. The current administration is redundantly cle
212.
by liuliu 2y ago
It looks pretty obvious (I think): 1. S2 is a 7B VLM, it is responsible for taken in camera streams (from however many of them), run through prompt guided text generation, and before the lm_head (or a few layers leading to it), directly tak
213.
by liuliu 2y ago
So, I am not doing big development with hundreds / thousands engineers for quite some time. So I started to have sympathy to the other side of the story. The problem with feature branches, especially for frontend development, is that i
214.
by liuliu 2y ago
Yes, the Llava model can encode image, and you can encode image into 3D vae space. Without fine-tune the model though, you are not going to have fidelity to original (if only use Llava's SigLIP to encode), or end up with image with lim
215.
by liuliu 2y ago
The training process in OmniHuman-1 seems to be straightforward to replicate once Tencent releases their image-to-video model too.
216.
by liuliu 2y ago
ByteDance didn't release their text-to-video model, which is the base of this work, so I would think unlikely.
217.
by liuliu 2y ago
I don't think anyone uses MTP for inference right now. Even if you use MTP for drafting, you need to batching in the next round to "verify" it is the right token, if that happens you need to activate more experts. DELETED: If
218.
by liuliu 2y ago
Yeah, if you look DeepSeek v3 paper deeper, each saving on each axis is understandable. Combined, they reach some magic number people can talk about (10x!): FP8: ~1.6 to 2x faster than BF16 / FP16; MLA: cut KV cache size by 4x (I think
219.
by liuliu 2y ago
This is a humble and informed acrticle (comparing to others written by financial analysts the past a few days). But still have the flaw of over-estimating efficiency of deploying a 687B MoE model on commodity hardware (to use locally, cloud
220.
by liuliu 2y ago
It does. They have 256 experts per MLP layer, and some shared ones. The minimal deployment for decoding (aka. token generation) they recommend is 320 GPUs (H800). It is all in the DeepSeek v3 paper that everyone should read rather than spec
221.
by liuliu 2y ago
KV cache shifting is interesting! Just curious: how much of your code nowadays completed by LLM?
222.
by liuliu 2y ago
I don't think this is useful distinction nowadays. If Willow demonstrated RCS within 100-qubit, I don't really think there are physical limitations for them to implement Shor for 50-bit integer numbers. It is just less impressive
223.
by liuliu 2y ago
What about the Bitcoin argument? If CRQC is built, given that you only needs ~1000 logic qubits (and probably fewer gates than RSA) for ECDSA, you would expect big financial incentive to just use the compute a little bit to crack a few cold
224.
by liuliu 2y ago
The router is manually designed (see their cem function). Also, the experts are not separate weights, just different scales of it's singular values.
225.
by liuliu 2y ago
One weakness of this method is the storage of decomposed UV from W. My linear algebra is rusty, but it seems required if you want to scale in that U projected subspace, hence double your weight memory footprint (that has been said, U /
226.
by liuliu 2y ago
Anyone made a X to Bluesky importer yet? It is pretty difficult to discover the users back on Bluesky, and I thought with the Bluesky hose, it should be possible to matching content v.s. user?
227.
Metal FlashAttention 2.0: Pushing Forward Inference & Training on Apple Silicon(engineering.drawthings.ai)
2 points by liuliu 2y ago|0 comments
228.
by liuliu 2y ago
Genuinely asking, are we still trying to "stop" the climate change any more or are we in the "easing the incoming pain" phase already?
229.
On the Measure of Intelligence(arxiv.org)
2 points by liuliu 2y ago|0 comments
230.
by liuliu 2y ago
I think many of replies here to you missing is the word he uses is "unpredictable". It is not "surprising", "unverifiable" or "unreasonable". "Prediction" associated in this particular talk
231.
by liuliu 2y ago
BTW, people found that in-conext instruction is useful for these (for example, directly using the last hidden layer to condition a diffusion model is much worse than encoder-decoder model, but you can add instruction prefix "try to ima
232.
by liuliu 2y ago
Not really. See the literature on sharing lm_head (last matrix multiplication) with the input embedding dict. Basically, the lm_head (a MxN matrix where M is the dictionary size and N is the internal dimension) can be seen as the dictionary
233.
by liuliu 2y ago
If you have mixed-language needs, yes. But if not, standardized setup is far better with IDE integration (mostly, VS code LSP etc).
234.
by liuliu 2y ago
They leaked an API access key (I think) and some generated videos not reviewed by OpenAI potentially (they say that all Sora videos posted online has to be reviewed by OpenAI).
235.
by liuliu 2y ago
I did try, but in a wrong way (try to SVD quantization error to recover quality (I.e. SVD(W - Q(W)))). The lightbulb moment in this paper is to do SVD on W and then quantize the remaining.
236.
by liuliu 2y ago
For more complex tasks, asking it to write in a more comfortable language for that task and then translate to target language helps. Example: if you ask Claude to write code that generates UDP datagram for mDNS in Swift, it will fail flat.
237.
by liuliu 2y ago
Yeah, honestly not too surprising. Happy someone made the experiments though. I think we know that NN with limited data tends to over-fitting, so to train LoRA you need stronger regularization mechanism, that including: * Fixing A as proj
238.
by liuliu 2y ago
Yeah, I believe this intuition first introduced by the Neural Turing Machine line-of-work and later simplified into AIAYN paper (NTM maintains "external memory" a.k.a. weight_keys, weight_values here). Disclaimer: these are from m
239.
by liuliu 2y ago
Yeah. I guess that I misspoke a little bit. I am not even criticize actor (as in Go, there is a relatively "heavy" runtime to implement its green thread too). I was originally thinking about distributed actor proposal (and impleme
240.
by liuliu 2y ago
Yeah. I think "keyword" is less of an issue personally. At the end of the day, if we want to approach things like async / await and ownership, I would prefer "consume / borrow" than some particular symbols. Tha

More ›