25 ms·
Mixtral of experts
- jstummbillig 3y agoIs there a non-obvious reason that models keep getting compared to GPT-3.5 instead of 4?
- seydor 3y agonobody can run gpt4 on their machine
- jstummbillig 3y agoI am not sure what that means. How can they run GPT 3.5 any more or less?
- deleted 3y ago[deleted]
- simonvc 3y agoGPT-3.5 (unfinetuned) has been matched by many OW (open weights) models now, and with fine tuning to a specific task (coding, customer care, health advice etc) can exceed it. It's still useful as a well known model to compare with, since it's the model the most people have experience with.
- stsquad 3y agoI think they mean the assumed parameter size of GPT-4 is so large you couldn't run it on commodity hardware even if you could get hold of the model.
- karmasimida 3y agoThe rumored GPT4 model size is 1.8 trillion parameters
- baq 3y agonon-obvious? don't think so. the obvious is GPT-4 blows them all out of the water in quality and is completely trounced on quality/inference cost.
- deleted 3y ago[deleted]
- opdahl 3y agoMaybe because GPT-3.5 is a free model so it's basically comparing free models with each other.
- skerit 3y agoMost people I know that actually use ChatGPT, use the free (GPT3.5) version. Not a lot of people are willing to pay the extra 20 euros per month.
- deleted 3y ago[deleted]
- viraptor 3y agoYet, if you want to go cheaper, you totally can by paying for the API access. Gpt4 is accessible there and you get to use your own app. $20 will last you way longer then a month if you're not a heavy user.
- skerit 3y agoTrue. Though I myself still use ChatGPT a lot more as I can quickly reach the $20 threshold via the API
- rrrrrrrrrrrryan 3y agoIt really depends on usage. If you need to have long conversations, or are sending huge texts, the all-you-can-eat for $20 plan will almost certainly be cheaper than API access. If you're doing lots of smaller one shot stuff without a lot of back and forth, the API will be cheaper.
- criddell 3y agoDo you think it's getting close to the point where it adds a euro of value per workday (on average) for them?
- skerit 3y ago
- deleted 3y ago[deleted]
- dcastm 3y agoGPT-3.5 is probably the most popular model used in applications (due to price point vs. GPT-4, and quality of results vs. open-weight models). So I guess they're trying to say now it's a no-brainer to switch to open-weight models.
- tyfon 3y agoI second this. For some type of applications, the 4 model can quickly ramp up costs, especially with large contexts and 3.5 often does the job just fine. So for many applications it's the real competitor.
- anonylizard 3y agoToo bad GPT3.5 Turbo is dirt cheap. Open source models are substantially more expensive when you factor in operating costs. There is no mature ecosystem where you can just plug in a model and spin up a robust infrastructure to run a local LLM at scale, aka you need infrastructure/ML engineers to do it, aka extremely expensive unless you are using LLMs at extremely large scales.
- dcastm 3y agoI think we'll start seeing a lot more services like https://www.together.ai https://www.together.ai soon. Having open-weight models better than gpt-3.5 will drive a lot of competition on the LLM infra.
- happycube 3y agoMistral's endpoint for mistral-small is slightly cheaper.
- Der_Einzige 3y agoThe additional control/features (support for grammars, constraints, fine-tuning, etc) far o/w the cost savings.
- anonyfox 3y agonot only price but also speed and API limits. I always ask myself the following pseudo-question: "for this geneneration/classification task, do I need to be more intelligent than an average highschool student?" Almost always in business tasks, the answer is a no. Therefore I go with GPT3.5. Its much quicker and good enough to accomplish the task usually. And then I need to run this task thousands of times, so the API limits are the most limiting factor, which are much higher in GPT3.5 variants, whereas when using GPT4 I have to be more careful with limiting/queueing requests. I patiently wait for a efficient enough model that only needs to be on a GPT3.5 level I can self-host alongside my applications with reasonably low server requirements. No need for GPT-5 for now, for business automations the lower end of "intelligence" is more than enough, but efficiency/scaling is the real deal.
- 0x008 3y agoGPT3.5 is the most popular MoE model probably.
- orbital-decay 3y agoBecause GPT-4 is multimodal, which puts it into the entirely different class. GPT-3.5 was trained on text, same as this model.
- deleted 3y ago[deleted]
- reqo 3y agoCan someone explain why MoE works? Is there any downside to MoE compared to a regular model?
- deleted 3y ago[deleted]
- gorbypark 3y agoI'm still new to most of this (so please take this with a grain of salt/correct me), but it seems that in this specific model, there's eight separate 7B models. There is also a ~2B "model" that acts as a router, in a way, where it picks the best two 7B models for the next token. Those two models then generate the next token and somehow they are added together. Upside: much more efficient to run versus a single larger model. The press release states 45B total parameters across the 8x7B models, but it only takes 12B parameters worth of RAM to run. Downside: since the models are still "only" 7B, the output in theory would be not as good as a single 45B param model. However, how much less so is probably open for discussion/testing. No one knows (outside of OpenAI) for sure the size/architecture of GPT-4, but it's rumoured to have a similar architecture, but much larger. 1.8 trillion total params, but split up into 16 experts at around 111B params each is what some are guessing/was leaked.
- sebzim4500 3y agoYou are almost right: * The routing happens in every feedforward layer (32 of these iirc). Each of these layers has it's own 'gate' network which picks which of the 8 experts are most promising. It runs the two most promising and interpolates between them. * In practice, all parameters still need to be in VRAM so this is a bad architecture if you are VRAM constrained. The benefit is you need less compute per token.
- miven 3y agoI wonder what would be the most efficient tactic for offloading select layers of such a model to a GPU within a memory-constrained system As far as I understand usually layer offloading in something like llama.cpp loads the first few consecutive layers to VRAM (the remainder being processed in the CPU) such that you don't have too much back and forth between the CPU and GPU. I feel like such an approach would lead to too much wasted potential in terms of GPU work when applied to a SMoE model, but on the other hand offloading non-consecutive layers and bouncing between the two processing units too often may be even slower...
- deleted 3y ago[deleted]
- seydor 3y ago> A proper preference tuning can also serve this purpose. Bear in mind that without such a prompt, the model will just follow whatever instructions are given. Mistral does not censor its models and is committed to a hands free approach, according to their CEO https://www.youtube.com/watch?v=EMOFRDOMIiU https://www.youtube.com/watch?v=EMOFRDOMIiU > Mixtral 8x7B masters French, German, Spanish, Italian, and English. EU budget cut by half
- gwervc 3y ago> Mistral does not censor its models and is committed to a hands free approach This will change really fast. I highly doubt AI will have free speech in France when citizens don't.
- seydor 3y agoIf they play the "French product" card well, La france will change her laws
- thrance 3y agoCan you please expand on how french citizens don't have free speech ? And how ensuring minimum decency in the output of a computer program would impact free speech ?
- anonyfox 3y agoYou can say anything you want in european countries. Some things might backfire in various ways, though. IE if you megaphone that some person group X should be killed right away, that usually is a criminal offense and punished according to law.
- pugio 3y ago> We’re currently using Mixtral 8x7B behind our endpoint mistral-small... So 45 billion parameters is what they consider their "small" model? I'm excited to see what/if their larger models will be.
- miven 3y agoThere seems to be an experimental Mistral Medium model listed among other available model endpoints on [1], the comparison table they give shows that it outmatches 8x7B by a few percent on every benchmark listed [1] https://mistral.ai/news/la-plateforme/ https://mistral.ai/news/la-plateforme/
- mebazaa 3y agoIt apparently outperforms GPT-4 at WinoGrande as well…
- jug 3y agoNote that it processes tokens at speed and cost of a 12B model though.
- michaelt 3y ago> So 45 billion parameters is what they consider their "small" model? According to Wikipedia: Rumors claim that GPT-4 has 1.76 trillion parameters, which was first estimated by the speed it was running and by George Hotz. [1] [1] https://the-decoder.com/gpt-4-architecture-datasets-costs-and-more-leaked/ https://the-decoder.com/gpt-4-architecture-datasets-costs-an...
- maelito 3y agoJust tried to register, haven't received the confirmation email.
- mfi 3y agoSame here.
- anonzzzies 3y agoSame.
- kosolam 3y agoSame.
- kosolam 3y agoNow works. I’m on the waitlist.
- AngaraliTurk 3y agoSame.
- saliagato 3y agoSame. Tip: if you put a random number it will get you through regardless
- Semaphor 3y agoHah, it works, that’s hilarious
- mijoharas 3y ago> It is the strongest open-weight model with a permissive license and the best model overall regarding cost/performance trade-offs. Is there any link to the model and weights? I don't see it if so.
- M4v3R 3y agohttps://nitter.net/MistralAI/status/1733150512395038967 https://nitter.net/MistralAI/status/1733150512395038967
- world2vec 3y agoThey released the weights as a torrent [0] but you can easily find it on Huggingface [1][2]. [0] https://twitter.com/MistralAI/status/1733150512395038967 https://twitter.com/MistralAI/status/1733150512395038967 [1] https://huggingface.co/search/full-text?q=mixtral https://huggingface.co/search/full-text?q=mixtral [2] https://huggingface.co/mistralai https://huggingface.co/mistralai
- mkesper 3y agoAlso here graphs are confusing as they don't show the full y axis (please don't do that!) https://towardsdatascience.com/misleading-graphs-e86c8df8c5de https://towardsdatascience.com/misleading-graphs-e86c8df8c5d...
- mr_mitm 3y agoDisagree, at least when stated this generally. This article is explaining it thorougly: http://www.chadskelton.com/2018/06/bar-charts-should-always-start-at-zero.html http://www.chadskelton.com/2018/06/bar-charts-should-always-... (The article you linked is not accessible to non-medium users, by the way. Apologies if it covers caveats.) For bar charts it's a good rule of thumb. For line charts, not necessarily. Scroll down to the cases where it makes sense to zoom in. Imagine you plot the global average temperature of the last 200 years including the zero. You could barely see the changes, but they've been dramatic. Use Kelvin to make this effect even stronger. Which brings up another point: The zero is sometimes arbitrary. If instead of a quantity you only plot its difference to some baseline, all that is changing is the numbers of the y-axis, but the actual plot stays the same. Is it now less misleading? I say no, because the reader must look at the axes and the title either way. So please do zoom in to frame the data neatly if it makes sense to do so.
- causal 3y agoFor headline news that's probably a good rule of thumb. But in an academic context, especially data science, anyone who is data literate should know to check the axes.
- dizzydes 3y agoHonest question: if they're only beating GPT 3.5 with their latest model (not GPT 4) and OpenAI/Google have infrastructure on tap and a huge distribution advantage via existing products - what chance do they stand? How do people see things going in the future?
- spacebanana7 3y agoPerhaps they’re hoping some enterprises will be willing to pay extra for a 3.5 grade model that can run on prem? A niche market but I can imagine some demand there. Biggest challenge would be Llama models.
- anentropic 3y agoand according to the article this model behaves like a 12B model in terms of speed and cost while matching or outperforming Llama 2 70B in output
- viraptor 3y agoIn terms of speed per token. What they don't say explicitly is that choosing the mix per token means you may need to reload the active model multiple times in a single sentence. If you don't have memory available for all the experts at the same time, that's a lot of memory swapping time.
- anon373839 3y agoTim Dettmers stated that he thinks this one could be compressed down to a 4GB memory footprint, due to the ability of MoE layers to be sparsified with almost no loss of quality.
- jlokier 3y agoIf your motivation is to be able to run the model on-prem, with parallelism for API service throughput (rather than on a single device), you don't need large memory GPUs or intensive memory swapping. You can architect it as cheaper, low-memory GPUs, one expert submodel per GPU, transferring state over the network between the GPUs for each token. They run in parallel by overlapping API calls (and in future by other model architecture changes). Th MoE model reduces inter-GPU communication requirements for splitting the model, in an addition to reducing GPU processing requirements, compared with a non-MoE model with the same number of weights. There are pros and cons to this splitting, but you can see the general trend.
- jurmous 3y agoAre there any scores on Dutch support? Is it totally not supported or not benchmarked?
- trash_cat 3y agoThis is very exciting and I think this is the future of AI until we get another, next-gen architecture beyond transformers. I don't think we will get a lot better until that happens and the effort will go into making the models a lot cheaper to run without sacrificing too much accuracy. MoD is a viable solution.
- cuuupid 3y agoMamba has shown SSMs are highly likely to be contenders for the replacement to transformers in a really similar fashion to when transformers were first introduced as enc-dec models. I’m personally very excited for those models as they’re also built for even faster inference (a major feature of transformers being wildly faster inference than with LSTMs)
- miven 3y ago>a major feature of transformers being wildly faster inference than with LSTM Wasn't the main issue with RNNs the fact that inference during training can't be efficiently parallelized? The inference itself normally should be faster for an RNN than for a transformer since the former works in linear time in terms of input size while the latter is quadratic
- visarga 3y agoMamba has dual view - you can use it both as CNN and RNN. The first is used for pre-training and for preloading the prompt because it can process all tokens at once. The second is used for token generation because it is O(1) per token. Basically two models in one, inheriting both advantages. This is possible because the Structured State Space layer is linear, so you can reshape some sums and unroll recursion into a convolution the size of the input, which can be further sped up with FFT.
- jimmySixDOF 3y agoand which is why the speed up is proportional to context length so starting near parity then, theoretically, see 100x at 100k tokens
- gardenhedge 3y agoThey're only paying 80,000€ for a full stack dev and want the candidate to have a "portfolio of successful projects"
- ranguna 3y agoWelcome to Europe
- j-v-w 3y ago[flagged]
- speedylight 3y agoThat’s ~$86,000 not bad at all imo and probably comes with other benefits as well. Not really sure how economical that is in the EU but it can’t be worse than the U.S where this is still pretty good compensation.
- mr_mitm 3y agoOpenAI is known to pay up to 10 times that. It's a different world. $86k in Europe is good (about 90th percentile of earners in Germany), but not as fantastical as some salaries in the US. Plus, Paris is probably expensive.
- mjirv 3y agoCurrent OpenAI job postings top out at $385,000 for full stack engineers which is 98th percentile in the US (and less for San Francisco). The €80k is the start of the Mistral salary range. €100k is the top, which would put you in the top 1% of earners in France.
- ukuina 3y agoThe $385k is base salary, not total comp, right?
- gardenhedge 3y ago
- epups 3y agoI think comparisons to base LLaMA are not so interesting, as almost no one is using those models. The most informative comparison is between Mistral 7B and 8x7B, provided in this picture: https://mistral.ai/images/news/mixtral-of-experts/open_models.png https://mistral.ai/images/news/mixtral-of-experts/open_model... The key takeway for me is that there is a decent improvement in all categories - about 10% on average with a few outliers. However, the footprint of this model is much larger so the performance bump ends up being underwhelming in my opinion. I would expect about the same performance improvement if they released a 13B version without the MoE. May be too early to definitely say that MoE is not the whole secret sauce behind GPT4, but at least with this implementation it does not seem to lift performance dramatically.
- ilaksh 3y agoHow does it compare to existing 13b models on benchmarks?
- epups 3y agoGood question. If you believe the results on the HuggingFace leaderboard (https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...), which I find very hard to navigate, we find that Mistral was not even the best 7B model in there, and there is a huge variance as well. I prefer to rely on benchmarks done by the same group of known individuals over time for comparisons, as I think it's still too easy to game benchmark results - especially you are just releasing something anonymously.
- gorbypark 3y agoMost of the top 7B models on the leaderboard are finetuned Mistral 7B models.
- epups 3y agoYou are right - upon closer inspection, even models that were not previously Mistral finetunes are now using Mistral in their later versions. I wasn't aware of it before as I could not filter results in the leaderboard (it doesn't even load at all for me now).
- DeathArrow 3y agoWhat kind of training data did they use? Are the training data and replies censored in any way?
- GaggiX 3y agoAre the training data and replies censored in any way? It doesn't seem so.
- donatzsky 3y agoFrom the link: > Mixtral is pre-trained on data extracted from the open Web – we train experts and routers simultaneously. > Note: Mixtral can be gracefully prompted to ban some outputs from constructing applications that require a strong level of moderation, as exemplified here. A proper preference tuning can also serve this purpose. Bear in mind that without such a prompt, the model will just follow whatever instructions are given.
- ensocode 3y agoIs anyone using this models self-hosted in production? What cloud hosting provider/plan do you use and how is it performance wise?
- rybosome 3y agoVery interested to hear a similar answer, but if anyone has tried running it locally on high-end consumer grade hardware, eg an Nvidia 4090?
- mdcurrent 3y agostumbled on this thread looking for tips- so far i haven't had success. i can load the model into memory by lowering the precision, I was OOMing on my 128GB RAM, now im OOMing on loading the model into the 4090 on the 17th shard at 4 bit qtization so i would imagine theres another knob or two im missing to get this to run
- rybosome 3y agoThanks for your insight. Sounds like it’s quite a challenge, 128GB of RAM and a 4090 are pretty beefy for a consumer PC.
- reqo 3y agoConsidering that this model has basically zero censorship I would say it is going to take a while before companies have fine-tuned this enough before putting it in production.
- dannyw 3y agoThis seems like grounds for an interesting and sustainable business model. * Offer uncensored, pure-instruction-following open source models. * Offer censored, 'corporate-values' models commercially.
- anonyfox 3y agoTo the contrary, I want need things locally/cheap/fast for generic internal business automation, so it will never generate content any outsider would ever see. Basically glue code between services with classification of data built-in.
- HarHarVeryFunny 3y agoInteresting to see Mistral in the news raising EUR 450M at a EUR 2B valuation. tbh I'd not even heard of them before this Mixtral release. Amazing how fast this field is developing!
- korvalds 3y agoMore models available at Huggingface now: https://huggingface.co/search/full-text?q=mixtral https://huggingface.co/search/full-text?q=mixtral Already available from both Mistralai and TheBloke https://huggingface.co/mistralai/Mixtral-8x7B-v0.1 https://huggingface.co/mistralai/Mixtral-8x7B-v0.1 https://huggingface.co/TheBloke/Mixtral-8x7B-v0.1-GGUF https://huggingface.co/TheBloke/Mixtral-8x7B-v0.1-GGUF
- xena 3y agoThe GGUF variant looks promising because then I can run it on my MacBook (barely)
- loudmax 3y agoNote that running these GGUF models currently requires a forked version of llama.cpp: https://github.com/ggerganov/llama.cpp/pull/4406 https://github.com/ggerganov/llama.cpp/pull/4406 The GGUF handling for Mistral's mixture of experts hasn't been finalized yet. TheBloke and ggerganov and friends are still figuring out what works best. The Q5_K_M gguf model is about 32GB. That's not going to fit into any consumer grade GPU, but it should be possible to run on a reasonably powerful workstation or gaming rig. Maybe not fast enough to be useful for everyday productivity, but it should run well enough to get a sense of what's possible. Sort of a glimpse into the future.
- suslik 3y agoThis can fit into a Macbook Pro with integrated memory. With all the recent development in the world of local llms I regret I settled for only 24Gb RAM on my laptop - but the 13b models work great.
- ComputerGuru 3y agoRecent CUDA releases let you use shared GPU memory instead of (only) dedicated vram, but the pci-e bandwidth constrains the inference speed significantly. How much faster is GPU access to the unified memory model on the new Macs/how much less of a hit do you take? Also, given the insane cost premium apple charges per extra GB of RAM (at least when I was last shopping for a device), do you come out ahead?
- bilsbie 3y agoI wonder if the brain uses a mixture of experts?
- MVissers 3y agoWe have regions that specialize in specific tasks. Information gets routed there for recognition and understanding. Eg: Faces are processed in the fusiform area. And if you play pokemon obsessively as a kid, you’ll even create an expert pokemon region: https://news.stanford.edu/2019/05/06/regular-pokemon-players-pikachu-brain/ https://news.stanford.edu/2019/05/06/regular-pokemon-players...
- i8comments 3y agoThe claim that it is better than GPT3.5, in practice, should be taken with a grain of salt, since the benchmarks themselves aren't... ideal. Despite the questionable marketing claim, it is a great LLM for other reasons.
- moffkalast 3y agoPlus benchmarks are typically ran from the API, using a tagged version of 3.5-turbo and not the whatever extra tuning and prompt magic ChatGPT as a frontend has to actually get the better results that people see in practice. On the other hand, if the base instruct model is good enough to roughly match it then the fine tunes will be interesting for sure.
- chandureddyvari 3y agoSorry if this is a dumb question. Can someone explain why it’s called 8x7B(56B) but it has only 46.7B params? and it uses 12.9B params per token generation but there are 2 experts(2x7B) chosen by a 2B model? I’m finding it difficult to wrap my head around this.
- pilotneko 3y agoI haven’t looked at the structure carefully, but It’s hard to guess there are shared layers between models. Likely the input layers for sure, since there is no need to tokenize separately for each model (unless different models have specialized vocabulary).
- lordswork 3y agoThis is my understanding as well. Also includes the parameters of the expert-routing gating network.
- brrrrrm 3y agomixture of experts gates on the feed forward network only. the shared weights are the KQV projections for the attention mechanism of each layer.
- shekhar101 3y agoExplanation from Andrej karpathy makes sense on why: ''' "8x7B" name is a bit misleading because it is not all 7B params that are being 8x'd, only the FeedForward blocks in the Transformer are 8x'd, everything else stays the same. Hence also why total number of params is not 56B but only 46.7B. '''
- deleted 3y ago[deleted]
- matrix2596 3y agothe model says 8x7B model, so its a 56B model. what is the GPU memory requirements to run this model for a 512 context size? are there any feasible quantization models of this available? I want to know if my 16GB VRAM GPU can run this model? Thanks
- Jackson__ 3y agoAccording to https://huggingface.co/TheBloke/Mixtral-8x7B-v0.1-GGUF https://huggingface.co/TheBloke/Mixtral-8x7B-v0.1-GGUF : 18.14GB in 2bit, which is still too high for your GPU, and most likely borders on unusable in terms of quality. You could probably split it between CPU and GPU, if you don't mind the slowdown.
- _giorgio_ 3y agoCan mixtral be fine tuned in any way?
- roguas 3y agoyes, they dropped instruct model already and i think people are churning the variety as we speak
- ComputerGuru 3y agoI’m surprised no one has commented on the context size limitations of these offerings when comparing to the other models. The sliding window technique really does effectively cripple its recall to approximately just 8k tokens which is just plain insufficient for a lot of tasks. All these llama2 derivatives are only effective if you fine tune them, not just because of the parameter count as people keep harping but perhaps even more so because of the tiny context available. A lot of my GPT3.5/4 usage involves “one offs” where it would be faster to do the thing by hand than to train/fine-tune first, made possible because of the generous context window and some amount of modest context stuffing (drives up input token costs but still a big win).
- tbalsam 3y agoA lot of attention window stuff is fluff IMPE, just gotta look at the benchmarks regardless of what the raw numbers sat.
- ComputerGuru 3y agoBenchmarks are, by definition, artificial. I'm speaking from real-world experience, not from "based off the architecture, here's my guess" and lack of context or poor recall within the "supported" context window is a real problem.
- tbalsam 3y agoYes, this is partially due, among other reasons, to the fact that inference induces a domain shift compared to training due to the use of teacher forcing.
- potatoman22 3y ago8k tokens is about 6000 words, which is more than enough for most classification tasks. Maybe it's not enough for something like story writing, but I feel like it's enough context for most business use cases.
- 3y ago
- inChargeOfIT 3y agoIt sounds like the same requirements as a 70b+ model, but if someone manages to get inference running locally on a single rtx4090 (AMD 7950x3D w/ 64GB ddr5) reasonably well, please let me know.
- BryanLegend 3y agoAndrej Karpathy's Take: Official post on Mixtral 8x7B: https://mistral.ai/news/mixtral-of-experts/ https://mistral.ai/news/mixtral-of-experts/ Official PR into vLLM shows the inference code: https://github.com/vllm-project/vllm/commit/b5f882cc98e2c9c6dde7357dbac2ec0c2c57d8cd https://github.com/vllm-project/vllm/commit/b5f882cc98e2c9c6... New HuggingFace explainer on MoE very nice: https://huggingface.co/blog/moe https://huggingface.co/blog/moe In naive decoding, performance of a bit above 70B (Llama 2), at inference speed of ~12.9B dense model (out of total 46.7B params). Notes: - Glad they refer to it as "open weights" release instead of "open source", which would imo require the training code, dataset and docs. - "8x7B" name is a bit misleading because it is not all 7B params that are being 8x'd, only the FeedForward blocks in the Transformer are 8x'd, everything else stays the same. Hence also why total number of params is not 56B but only 46.7B. - More confusion I see is around expert choice, note that each token and also each layer selects 2 different experts (out of 8). - Mistral-medium Source: https://twitter.com/karpathy/status/1734251375163511203 https://twitter.com/karpathy/status/1734251375163511203
- nojvek 3y agoAnyone have a feeling karpathy may leave openAI to join an actual Open AI startup where he can openly speak about training tweaks, the datasets architecture etc. It seems recently OpenAI is the least open startup. Even Gemini talks more about their architecture. OpenAI still doesn’t openly mention GPT4 is a mixture of experts model.
- mercat 3y agoCan someone please explain how this works to a software engineer who used to work with heuristically observable functions and algorithms? I'm having a hard time comprehending how a mix of experts can work. In SE, to me, it would look like (sorting example): - Having 8 functions that do some stuff in parallel - There's 1 function that picks the output of a function that (let's say) did the fastest sorting calculation and takes the result further But how does that work in ML? How can you mix and match what seems like simple matrix transformations in a way that resembles if/else flowchart logic?
- namibj 3y agoThe feed forward layer is essentially a differentiable key-value store. Similar to the attention layer, actually. So it just uses an attention mechanism like pre-selector to attend to only some experts. During inference, this cutoff is made a hard cutoff.
- mercat 3y agoThis is a very interesting approach. I know it may be too much to ask, but would you suggest any actual practical and hands-on workshops, playgrounds, or courses where I could practice using NN layers for stuff like that? For example, conditional/weighted selection of previous inputs, etc. It feels like I'm looking at ML programming from another angle.
- legel 3y agoThe Sparse Mixture of Experts neural network architecture is actually an absolutely brilliant move here. It scales fantastically, when you consider that (1) GPU RAM is way too expensive, in financial dollars, (2) SSD / CPU RAM are relatively cheap, and (3) you can have "experts" running on their own computers, i.e. it's a natural distributed computing partitioning strategy for neural networks. I did my M.S. thesis on large-scale distributed deep neural networks in 2013 and can say that I'm delighted to point our where this came from. In 2017, it emerged from a Geoffrey Hinton / Jeff Dean / Quoc Le publication called "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer". Here is the abstract: "The capacity of a neural network to absorb information is limited by its number of parameters. Conditional computation, where parts of the network are active on a per-example basis, has been proposed in theory as a way of dramatically increasing model capacity without a proportional increase in computation. In practice, however, there are significant algorithmic and performance challenges. In this work, we address these challenges and finally realize the promise of conditional computation, achieving greater than 1000x improvements in model capacity with only minor losses in computational efficiency on modern GPU clusters. We introduce a Sparsely-Gated Mixture-of-Experts layer (MoE), consisting of up to thousands of feed-forward sub-networks. A trainable gating network determines a sparse combination of these experts to use for each example. We apply the MoE to the tasks of language modeling and machine translation, where model capacity is critical for absorbing the vast quantities of knowledge available in the training corpora. We present model architectures in which a MoE with up to 137 billion parameters is applied convolutionally between stacked LSTM layers. On large language modeling and machine translation benchmarks, these models achieve significantly better results than state-of-the-art at lower computational cost." So, here's a big A.I. idea for you: what if we all get one of these sparse Mixture of Experts (MoEs) that's a 100 GB on our SSDs, contains all of the "outrageously large" neural network insights that would otherwise take specialized computers, and is designed to run effectively on a normal GPU or even smaller (e.g. smartphone)? Source: https://arxiv.org/abs/1701.06538 https://arxiv.org/abs/1701.06538