sailingparrot

7 ms·

91.
by sailingparrot 1y ago
Isn't the ability to store past reasoning in an external system to avoid having to do the computation all over again precisely what a memory is though? But mathematically KV-caching, instead of doing prefilling at every token is equiva
92.
by sailingparrot 1y ago
You are forgetting about attention on the kv-cache, which is the mechanism that allows LLM to not start anew everytime.
93.
by sailingparrot 1y ago
But you are missing the causal attention from your analysis. The output is not the only thing that is preserved, there is also the KV-cache. At token 1, the model goes through, say, 28 transformer blocks, for each one of those block we save
94.
by sailingparrot 1y ago
Indeed what I meant. The LLM isn’t a blank slate at the beginning of each new token during autoregression as the kv cache is there.
95.
by sailingparrot 1y ago
I don't remember any paper looking at this specific question (thought it might be out there), but in general Anthropic's circuit threads series of article is very good on the broader subject: https://transformer-circuit
96.
by sailingparrot 1y ago
I think a better mental framework of how those model work is that they keep an history of the state of their "memory" across time. Where humans have a single evolving state of our memory LLMs have access to all the states of their
97.
by sailingparrot 1y ago
I don't follow how this relates to what we are discussing. Autoregressive LLMs are able to plan within a single forward pass and are able to look back at their previous reasoning and do not start anew at each token like you said. If yo
98.
by sailingparrot 1y ago
> They rebuild the "long term plan" anew for every token Well no, there is attention in the LLM which allows it to look back at it's "internal thought" during the previous tokens. Token T at layer L, can attend t
99.
by sailingparrot 1y ago
I’m not arguing about efficiency though ? Simply saying next token predictors cannot be thought of as actually just thinking about the next token with no long term plan.
100.
by sailingparrot 1y ago
> the token-at-a-time approach of the in-vogue LLMs. Speaking for myself, I don't generate words one a time based on previously spoken words Autoregressive LLMs don't do that either actually. Sure with one forward pass you only
101.
by sailingparrot 1y ago
The issue isn’t so much scheduling as it is stability. More clusters means one more layer of things that can crash your (very expensive) training. You also then still need to write tooling to manage cross cluster trainings correctly just as
102.
by sailingparrot 1y ago
> Quite a wide variety of people find AI deeply ego threatening to the point of being brainwormed into spouting absolute nonsense, but why? He is not brainwashed, this just happens to be his business. What happens to Gary Marcus if Gary
103.
by sailingparrot 1y ago
What laws do you suppose they have to follow, exactly? Cops need a warrant to track your phone, check which tower it connected to or tail your car for extended period of time. Cops do not need a warrant to use Flock system. They have an app
104.
by sailingparrot 1y ago
Its a DGX dev box, for those (not consumers) that will ultimately need to run their code on large DGX clusters where a failure or a ~3% slowdown of training ends up costing tens of thousands of dollars. That's the use case, not running
105.
by sailingparrot 1y ago
> This voice standardization model is an in-house accent-preserving voice conversion model. Not sure this model works really well. As a french/spanish native speaker, I can immediately recognize an actual French or Spanish person sp
106.
by sailingparrot 1y ago
Oh in that case there is definitely a top-k or top-p behind the scene, it might just not be exposed to the user as a param they can change through their API. I haven’t heard of anyone running a LLM in prod with actual pure sampling
107.
by sailingparrot 1y ago
> The TLDR/key (from my understanding) is that verifying N tokens can be faster than generating N tokens. Yes. This is because to generate token n+1 you need token n etc. So generating from scratch is a sequential (thus slow) proces
108.
by sailingparrot 1y ago
Not sure exactly what setup you are running, in theory yes, higher temperature for both model means higher chance of overlap and thus less rejections -> faster sampling (but worse quality overall). However, if you have higher temperature
109.
by sailingparrot 1y ago
> the verifier is checking if the tokens are good-enough, not if they're the exact same tokens it would have selected That's up to you, depends on how you implement it and how much you want to prioritize speed at the expense of
110.
by sailingparrot 1y ago
I don't know anything about Together quality in general, but the specific technique discussed here (speculative decoding) has no impact on the quality of generations. So you should be able to apply it to whichever model you want, and s
111.
by sailingparrot 1y ago
Definitely a better high level metric than nvidia-smi, and probably fine if you just want to get a very coarse idea of whether or not your are using the GPUs reasonably at all. But when you get to the point where you care about a few percen
112.
by sailingparrot 1y ago
For kernel-level performance tuning you can use the occupancy calculator as pointed out by jplusqualt or you can profile your kernel with Nsight compute which will give you a ton of info. But for model-wide performance, you basically have t
113.
by sailingparrot 1y ago
"Utilization" tells you the percentage of your GPU's SM that currently have at least one thread assigned to them. It does not at all take into count how much that thread is actually using the core to it's capacity. So if
114.
by sailingparrot 1y ago
Fair, didn’t mean to strawman.
115.
by sailingparrot 1y ago
> Odds are that shirt had already sailed around the world once or twice by the time you buy it, That's like... 40 grams of CO2 for those 2 trips for a t-shirt. Aka, a 2 min uber ride. Shipping containers are incredibly efficient. Mo
116.
by sailingparrot 1y ago
Your argument boils down to "people already said some things like this in the past" (ok, and?) and "some people I don't like agree with part of it", which is very weak and doesn't address anything in substance.
117.
by sailingparrot 1y ago
This is basically the entire theme of "Abundance" by Klein and Thompson, for those looking for a longer read on this. [1]: https://en.wikipedia.org/wiki/Abundance_(Klein_and_Thompson_...
118.
by sailingparrot 1y ago
> they set their hiring bar for engineers too high Not sure I agree, if you look at the head count growth of companies like OpenAI, Anthropic etc, it is super fast, its already pretty hard to keep everything working smoothly with that ra
119.
by sailingparrot 1y ago
> I can only assume all these ideas have been widely implemented at all significant labs for years, right? Nope. I was also surprised, when joining such significant labs at how much relatively-low hanging fruits were still available to w
120.
by sailingparrot 1y ago
> What about for inference? Same techniques apply. > but did not dispute that they store the entire model on SRAM. No idea what they did or did not do for that specific test (which was about delivering 1800 tokens/sec though, not

More ›