15 ms·
Non-determinism in GPT-4 is caused by Sparse MoE
- osmarks 3y agoI feel like this introduces the potential for weird and hard-to-implement side channel attacks, if the sequences in a batch can affect the routing of others.
- tehsauce 3y agoI think you’re right. Would be very hard to exploit I imagine though.
- derwiki 3y agoHard like building a virtual machine in an image decoder? If there’s a way there’s a will.
- adql 3y agoSame thing was said about Spectre-like bugs
- catchnear4321 3y agothe tools available to imagine such things are limited today. the language models in our heads have not caught up to the ones in our browsers. as the similarities and associations crystallize a bit better, it won’t look so hard. bookmark this if you think it bullshit. eight months.
- osmarks 3y agoI don't expect LLMs to be good enough at engineering to trivialize this kind of thing for a while - possibly never, if something else comes along and outcompetes them.
- catchnear4321 3y agonot models. monkeys.
- refulgentis 3y agoThis is _excellent_ work, I've been adamantly against MoE for a set of reasons, this is the first compelling evidence I've seen that hasn't been on Substack or a bare repeating of rumor. I had absolutely no idea GPT4 was nondeterministic and I use it about 2 hours a day. I can see why a cursory looking wasn't cutting it, they "feel" the same in your memory, a lot of similar vocab usage, but are formatted entirely differently, and have sort of a synonym-phrase thing going where some of the key words are the same.
- derwiki 3y agoGPT4 web chat for two hours a day? I buy that. Using the API repeatedly for the same inputs, eg developing a program, and the non-determinism is hard to miss.
- sebzim4500 3y agoI would imagine that most people use nonzero temperature, so they won't need to look for any explanation for non-determinism.
- dekhn 3y agoLiterally the first thing I did when I had llama.cpp working was set the temperature to 0 and repeat queries. (but that's mainly because I'm a weird old scientist with lots of experience with nondeterminism in software).
- refulgentis 3y agoI did too, Kmeans broke me a couple years ago: but, never temperature at 0 with long length, and trusted my instinct instead of actual diffs. This is was the first time I actually diffed
- phillipcarter 3y agoYeah, it's one of the first things you notice when trying to do some kind of "feed GPT some data and get it to produce a novel answer to a question" task with the API.
- pazimzadeh 3y agoMixture of Experts
- airstrike 3y agoThank you! I knew it couldn't mean "Merger of Equals"... but then again, if those experts are equals, then maybe that acronym also works ;-)
- TechBro8615 3y agoThanks. I assumed it was Margin of Error. The article doesn't expand the acronym until midway through the post, where it appears almost accidentally. Perhaps the intended audience is a mixture of experts, of which I'm not a part.
- mst 3y agoI suspect the article is written primarily to be clear to people sufficiently immersed in the relevant areas to be able to have a concrete opinion on the theory. Also I strongly suspect that at least in the case of -me-, an article that was easier for me to understand wouldn't make the underlying theory any easier for me to judge. (on the upside, at least I -did- understand and appreciate your self deprecating pun :)
- alpark3 3y ago_If_ 3.5 is a MoE model, doesn't that give a lot of hope to open source movements? Once a good open source MoE model comes out, maybe even some type of variation of the decoder models available(I don't know whether MoE models have to be trained from scratch), that implies a lot more can be done with a lot less.
- 152334H 3y agoI agree, and really hope that Meta is doing something in that vein. Reducing the FLOPs:Memory ratio (as in Soft MoE) could also open the door to CPU (or at least Apple Silicon) inference becoming more relevant.
- osmarks 3y agoIt would be bad for single-consumer-GPU inference setups.
- worldsayshi 3y agoCould this work well with distributed solutions like petals? https://github.com/bigscience-workshop/petals https://github.com/bigscience-workshop/petals I don't understand how petals can work though. I thought LLMs were typically quite monolithic.
- osmarks 3y agoPetals does a layerwise split I think. You could probably run separate experts on each system. I don't think this sort of tech is very promising so I haven't looked.
- Me1000 3y agoNot an expert (no pun intended), but MoE where each expert is actually just a LoRA adaptor on top of the base model gets me pretty excited. Since LoRA adaptors can be swapped in and out at runtime, it might be possible to get decent performance without a lot of extra memory pressure.
- dudus 3y agoOff topic > 3 months later, reading a paper while on board a boring flight home, I have my answer. I noticed people from hacker news routinely read scientific papers. This is a habit I envy but don't share. Any tips or sites for someone interested in picking up more science papers to read.
- dylan604 3y agoI want to know what a non-boring flight would be like
- nerdponx 3y agoHigh turbulence definitely makes it less boring. So will a crying baby, disruptive passenger, or someone getting sick. After a few of those, you'll prefer the boring flights.
- 152334H 3y agohttps://www.youtube.com/watch?v=iFImKMjM-q4 https://www.youtube.com/watch?v=iFImKMjM-q4
- LordShredda 3y agoSnakes on a plane
- ineedasername 3y agoAirforce One
- goatlover 3y agoThe Langoliers
- ineedasername 3y agoAirplane!
- deleted 3y ago[deleted]
- throwawayadvsec 3y ago"these tokens often compete against each other for available spots in expert buffers. " So is this also why ChatGPT is often just writing placeholders in place of functions when I ask him for some long code?
- gojomo 3y agoNot sure I understand the excerpt from the referenced paper. Is it saying that part of its more-efficient inferencing relies on mixing tokens from completely-separate inputs – eg, from other users? And then, depending on what other inputs chance into the same grouping, the relative assignment-to-'experts' varies, and thus the eventual completions? If so, I'd see that as not just introducing non-determinism, but also potentially making the quality of your responses dependent on how-many-concurrent-requests are fighting for the same expert-allocations. (For example, maybe the parts of the system best at translating/interpreting Hindi give worse results during peak usage hours-of-the-day in India, when the most concurrent inputs are competing for that same competence.) Perhaps also, this is another possible explanation for perceived quality-degradation over time. When certain tests were reliably succeeding earlier, there was less congestion for the relevant 'experts'. Now, with more concurrent use, those same tests aren't as reliably winning as much of relevant 'experts' effort. This may also suggest a bit of a quagmire: on whatever domains some sub-experts seem impressively good, initially, even more proportionate use will be attracted. But such new congestion means all the copycat use no longer gets the same expert allocations – and thus the initially-impressive performance degrades. (And if the effect is strong, & known-but-undisclosed-by-OpenAI, does it amount to a bait-and-switch? Attract users with unrepresentative excellence on an initially-uncongested Mixture-of-Experts system, but then offer them the lower-quality results from a more-congested system.)
- albystein 3y agothis seems like a plausible outcome, and if true could spell disaster for OpenAI models relative to the competition and open source models. Currently, reliability is one of the core obstacles preventing widespread adoption of LLMs in many business critical workflows. And if these rumors, that GPT-4 is inherently un-deterministic and unreliable, are true then most enterprises are better off finetuning open source LLMs—which are just as capable—for their specific domains. they stand to gain better performance that way anyways, as domain-specific models will always outperform generalist ones
- geysersam 3y ago> domain-specific models will always outperform generalist ones That's only true assuming you habe enough data to train a domain-specific model / expertise to train it and test it correctly. I've encountered cases where an image recognition task could be accomplished well with a very general model like CLIP, but people still fine-tuned another model on their own small data set because that's considered better. A domain specific model might be more likely to fail on weird outliers not present in the small domain specific training data. > could spell disaster for OpenAI Nah I don't think so. They are not all in on one specific model architecture. If the current architecture is found to have serious unfixable flaws then they'll just change architecture.
- albystein 3y agothis hypothesis makes a lot of sense. if indeed gpt-4 is a sparse MoE—which i believe it is—then OpenAI must have tested and proved their initial idea of a large capacity MoE LLM model first training/building a smaller one. this smaller test model might be gpt-3.5-turbo.
- crazypython 3y agoThe GPT-3.0 "davinci-instruct-beta" models have been returning non-deterministic logprobs as early as early 2021. This is speculation. CUDA itself often has nondeterminism bugs. text-davinci-001 and text-davinci-002 were trained through FeedMe and SFT, while text-davinci-003 was RLHF; the models themselves have more variance at high temperature.
- cubefox 3y agoWhat about the foundation models, i.e. davinci and code-davinci-002?
- jiggawatts 3y agoFloating point inaccuracies are generally deterministic - running the same calculations twice ought to yield the same results, down to the bit. You only get divergent results if there is some other source of state or entropy: not zeroing buffers correctly, race conditions, not setting rounding mode flags consistently, etc… From the quality of the code I’ve seen being cobbled together in the AI/ML ecosystem I would assume all three of those issues going on, and maybe more.
- n2d4 3y agoNo, this is not true for GPUs. https://www.twosigma.com/articles/a-workaround-for-non-determinism-in-tensorflow/ https://www.twosigma.com/articles/a-workaround-for-non-deter... (In this particular case, the order in which the numbers are summed up is non-deterministic due to GPU parallelism, which may change the result slightly.) I would generally refrain from insulting other people's code if you don't know much about the system it's written on. . Editing here since all the replies to this are mostly saying the same thing: Yes, CPUs can also be parallel and it can happen there as well, but unlike a CPU where most instructions on their own are deterministic, CUDA provides primitives that aren't. This is very much by design (as they're faster than their deterministic counterparts), and I mostly just take issue with how parent phrased this as a bug caused by bad code.
- Tunabrain 3y agoGPUs are deterministic machines, even for floating point. The behavior in the linked article has to do with the use of atomic adds to reduce sums in parallel. Floating point addition is not associative, so the order in which addition occurs matters. When using atomic adds this way, you get slightly different results depending on the order in which threads arrive at the atomic add call. It's a simple race condition, although one which is usually deemed acceptable.
- n2d4 3y agoI just edited my comment while you were writing your comment to add an explanation. The point here is that some primitives in eg. cudNN are non-deterministic. Whether you classify that as a race condition or not is a different question; but it's intended behaviour.
- rvcdbn 3y agoI wonder if there’s a side channel attack in there waiting to happen..
- icelancer 3y agoHow interesting. I was just discussing this last night with our analysts after I experimentally noticed that temp=0.0 (and all penalties/top_p set accordingly) still showed non-determinate behavior. Wasn't sure why this was, and now this article comes about. The explanation makes quite a bit of sense.
- hyperthesis 3y agoMoE: Mixture of Experts
- ShamelessC 3y agoThere’s a comment that’s 3 hours older than yours that clarifies this.
- hyperthesis 3y agoI searched for MoE in the comments and didn't see it. ah, you must mean this one https://news.ycombinator.com/item?id=37006549 https://news.ycombinator.com/item?id=37006549, which doesn't include "MoE", so that's why I didn't find it. Still, my comment's upvotes show it was helpful to some - maybe they searched for "MoE" too, instead of "mixture of experts".
- afro88 3y ago> these tokens often compete against each other for available spots in expert buffers. Hold up, does this mean that under heavy load the results change? Does this explain why it sometimes feels like the output quality changes?
- cainxinth 3y agoI asked GPT to explain this: >In the MoE approach, different "experts" or portions of the model are selected for different parts of the input data. The selection of which experts to use can be influenced by several factors, including the specific content of the input data, the order in which data is processed in a batch, and possibly even minor variations in the internal state of the model. >This "expert selection" process introduces a level of stochasticity, or randomness, into the model's operation. For example, if you process the same input data twice in slightly different contexts (e.g., as part of different batches), you might end up consulting slightly different sets of experts, leading to slightly different outputs.
- deleted 3y ago[deleted]
- cratermoon 3y ago> It’s well-known at this point that GPT-4/GPT-3.5-turbo is non-deterministic, even at temperature=0.0 Interestingly, on another discussion there was a claim that setting the temperature to 0.0 made gpt-4 deterministic: https://news.ycombinator.com/item?id=36503146 https://news.ycombinator.com/item?id=36503146
- moonchrome 3y agoThis guy probably never did anything nontrivial with the API - you notice almost instantly that the chat models (both 3.5 and 4) are nondeterministic at 0 temperature. Source - built a documentation search bot and had it crap out on me on copy pasted prompts when I was demoing it.
- keskival 3y agoIf you want to make it deterministic, just cache the responses keyed by queries.
- cratermoon 3y agoApparently, and I haven't tested this, just from what I read, the simpler GPT-2 models are deterministic at 0 temperature.
- f1shy 3y agoI see in the comments it seems to be a huge miss understanding between 2 uses of “non-deterministic”: 1) from normal English: cannot be determined beforehand (results may vary) 2) from theory of computation: loosely “parallel computation” (unknown path to the solution)
- PeterisP 3y agoFor floating point math, there's no distinction, as "parralel computation with unknown path to the solution" inherently implies "results will vary", as (a+b)+c != a+(b+c).
- DeathArrow 3y agoWell, a colleague of mine managed to build a non deterministic GET REST API endpoint. :D
- heroku 3y agocan somebody make some quantum AI, that's super deterministic.
- pmarreck 3y agoDeterminism should always be an option in any system.
- rgoldste 3y agoThis is a plausible hypothesis. I’m curious whether OpenAI has considered this already and examined it I feel like an average senior eng could eval this in under two focused days, but maybe OpenAI has less unit-testing than I expect.