valine

6 ms·

31.
by valine 2y ago
Probably helps that models like deepseek are mixture of expert. Having all weights in VRAM means you don’t have to unlod/reload. Memory bandwidth usage should be limited to the 37B active parameters.
32.
by valine 2y ago
There is no alternative. Europe tried to peacefully coexist with Germany in the 1930s. Standing up to imperialist dictators is the least bloody option.
33.
by valine 2y ago
My impression is that it’s a massive increase in the parameter count. This is likely the spiritual successor to GPT4 and would have been called GPT5 if not for the lackluster performance. The speculation is that there simply isn’t enough da
34.
by valine 2y ago
Lets just call every turning complete system AI and be done with it.
35.
by valine 2y ago
Optimization can be a tool used in the creation of AI. I'm taking issue with people who say their optimizer is an AI. We don't need to personify every technology that can be used to automate complex tasks. All that does is furth
36.
by valine 2y ago
My point is that it’s equally ridiculous to call either AI. If our chip here is not the AI then the AI has to be the optimizer. By extension that means AdamW is more of an AI than ChatGPT.
37.
by valine 2y ago
I strongly dislike when people say AI when they actually mean optimizer. Calling the product of an optimizer “AI” is more defensible, you optimized an MLP and now it writes poetry. Fine. Is the chip itself the AI here? That’s the product of
38.
by valine 2y ago
The RL is done on problems with verifiable answers. I’m not sure how o1 slop would be at all useful in that respect.
39.
by valine 2y ago
The existence of R1-zero is evidence against any sort of theft of OpenAI's internal COT data. The model sometimes outputs illegible text that's useful only to R1. You can't do distillation without a shared vocabulary. The onl
40.
by valine 2y ago
Results of their clinical trial look positive. https://pubmed.ncbi.nlm.nih.gov/39289373/
41.
by valine 2y ago
Even if everything you said is true it doesn’t mean general AI is 20 years away. Computers are many orders of magnitude faster at non-parallel tasks. Who’s to say raw clock speeds won’t easily compensate for any deficiencies vs the brain. Y
42.
by valine 2y ago
And human reasoning is somehow more magical? Really struggling to understand the distinction between searching through a latent space and “reasoning”.
43.
by valine 2y ago
You probably had your render loop set to poll rather than wait for events. If you don't configure wait events you'd be re-rendering your UI as fast as your monitor refreshes.
44.
by valine 2y ago
How did you create the rotated labels? That's been a major pain point for me using ImGui, I wish there was native support for that.
45.
by valine 2y ago
You can induce an activation potential with an external EM field. That’s the entire basis for deep brain stimulation.
46.
by valine 2y ago
No, and that’s because we are talking about relative positions. Every query can have its own set of keys. From the perspective of token 100 token 3 would be squished down, but from the perspective of token 3 it is still at position 3 and ca
47.
by valine 2y ago
They don’t need rapid reuse for Artemis. It would certainly help but Starship can just hang out in orbit for a few weeks while they do what they need to do to launch all their rockets.
48.
by valine 2y ago
One strategy I’ve been playing around with is to take an instruction I want the model to follow and squish the positional encodings for the keys down to position zero, and the new queries out slightly further in the window. The model will s
49.
by valine 2y ago
You have to be careful to keep the relative positions for adjacent and nearby tokens intact. The relative positions of distant tokens are less brittle.
50.
by valine 2y ago
Maybe this is wishful thinking but could something like this be used to make star wars style holograms? Strategically placed laser shadows to block light sounds like a useful tool in that direction.
51.
by valine 2y ago
Some people are really great at reading code, and they’re the best equipped to take advantage of LLMs in their current slightly unreliable state. If you can read and fix the LLMs code faster than you can write it from scratch then it’s a ne
52.
by valine 2y ago
One of the things I really love about rope is that it allows for a lot of interesting encoding schemes at inference time without model retraining. I’ve had a lot of fun playing with different relative positions. You can elicit a lot of inte
53.
by valine 2y ago
LEO is the exact place you need to be careful. Higher orbits have more space and have fewer satellites overall so it's less of a concern. Obviously not a problem for IFT6 since it's sub-orbital, but the original comment was about
54.
by valine 2y ago
Detonating something in orbit could trigger Kessler's Syndrome.
55.
by valine 2y ago
Unless I’m reading it wrong I don’t think rows matter. Attention doesn’t take into account sequence position natively, that’s why positional encodings exist.
56.
by valine 2y ago
I would like to see a comparison for the inference time compute between a regular transformer and this. I’m assuming token/s is lower since you need to compute the weights of the model for each token prior to the actual attention calcu
57.
by valine 2y ago
Completely agree. It’s frustrating how every thread on self-driving devolves into the same argument about cameras vs. LIDAR. Everyone points to Tesla FSD disengagements as evidence, when the vast majority of disengagements are caused by the
58.
by valine 2y ago
People get hung up on the vision stack and mapping. Obviously, any self-driving car needs a way to perceive the world around it, and Tesla’s approach is very different from Waymo. The software that actually does the driving, ie turns the wh
59.
by valine 2y ago
It gets progressively bigger and louder the longer you ignore it. After 30ish seconds it sounds an alarm and kicks you out.
60.
by valine 2y ago
All of it can be reused. Their plan is for the tower to set the booster back down and immediately relaunch it.

More ›