4 ms·
blog post: https://x.ai/blog/grok-os https://x.ai/blog/grok-os * 314B parameters (86B active at a time) * mixture of experts 8 (2 active at a time) * wei
by tosh 3y ago
blog post: https://x.ai/blog/grok-os https://x.ai/blog/grok-os
* 314B parameters (86B active at a time)
* mixture of experts 8 (2 active at a time)
* weights and architecture licensed under Apache 2.0
(edit:) announcement blog post from last year
with benchmarks compared to Claude 2, GPT-3.5 and GPT-4: https://x.ai/blog/grok https://x.ai/blog/grok
(edit2:)TL;DR: somewhat comparable to GPT-3.5, Mixtral and Qwen-1.5-72B in capability but way larger than the open weight models
- TOMDM 3y agoMixtral is also comparable to gpt 3.5 and open. At 8x7B it's also a fraction of the size. Are there any benchmarks comparing Mixtral to Grok?
- tosh 3y agoMixtral announcement is here: https://mistral.ai/news/mixtral-of-experts/ https://mistral.ai/news/mixtral-of-experts/ Mixtral looks more economical @ capability to size (similar also for Qwen 1.5 72b)
- OkGoDoIt 3y agoIs a model so huge that’s only at the level of GPT 3.5 actually good? That seems incredibly inefficient to me.
- fwlr 3y agoOpenAI is valued at 90 billion and all they do is make GPT; Twitter is valued at 40 billion and this was essentially a vanity side-project by a cowboy CEO. Presuming that benchmarks and general “it’s about the level of 3.5” is accurate, it’s inefficient, but not incredibly inefficient imho
- pelorat 3y ago> Twitter is valued at 40 billion WAS vaulued at 44B. Now? Maybe 5 billion.
- wongarsu 3y agoLast I heard they lost 15% of their users, so let's call it 36 billion.
- mceachen 3y agoMore like $13b. https://arstechnica.com/tech-policy/2024/01/since-elon-musks-twitter-purchase-firm-reportedly-lost-72-of-its-value/ https://arstechnica.com/tech-policy/2024/01/since-elon-musks...
- wraptile 3y agoTwitter didn't have direct competitors other than Mastodon when it was taken at 44B. Now there's Threads, Bluesky and bigger Mastodon.
- _ea1k 3y agoHonestly, none of those look like meaningful competitors at the moment.
- squigglydonut 3y agoNone of these matter
- dilyevsky 3y agoThey weren't even 44B when elon took the keys - he specifically tried to back out of the deal because 44B was insane peak '21 asset bubble price. In truth they were probably like 10-15B at that moment. And now that bunch of advertisers left due to we know who it's probably about 10B
- Lewton 3y agotwitter was valued around 30 billion when musk tried getting out of buying it (then the market cap went up when it became clear that he would be forced to pay full price)
- thekhatribharat 3y agoxAI is a separate entity, and not a X/Twitter subsidiary.
- xcv123 3y agoAccording to their benchmarks it is superior to GPT-3.5
- cma 3y agoSince it is MoE, quantized it could be able to run on cheaper hardware with just consumer networking inbetween instead of needing epyc/xeon levels of PCI-e lanes, nvlink, or infiniband type networking. Or it could even run with people pooling smaller systems over slow internet links.
- drak0n1c 3y agoIt’s designed to be actively searching real-time posts on X. Apples and oranges.
- hn_20591249 3y agoThe data pipeline isn't included in this release, and we already know it is a pretty simple RAG pipeline using qdrant, https://twitter.com/qdrant_engine/status/1721097971830260030 https://twitter.com/qdrant_engine/status/1721097971830260030. Nothing about using data in "real time" predicates that the model parameters need to be this large, and is likely quite inefficient for their "non-woke" instructional use-case.
- lmeyerov 3y agoAgreed. We have been building our real-time GPT flows for news & social as part of Louie.AI, think monitoring & and investigations... long-term, continuous training will become amazing, but for the next couple of years, most of our users would prefer GPT4 or Groq vs what's here and much smarter RAG. More strongly, the interesting part is how the RAG is done. Qdrant is cool but just a DB w a simple vector index, so nothing in Grok's release is tech we find relevant to our engine. Eg, there is a lot of noise in social data, and worse, misinfo/spam/etc, so we spend a lot of energy on adverserial data integration. Likewise, queries are often neurosymbolic, like on a data range or with inclusion/exclusion criteria. Pulling the top 20 most similar tweets to a query and running through a slow, dumb, & manipulated LLM would be a bad experience. We have been pulling in ideas from agents, knowledge graphs, digital forensics & SNA, code synthesis, GNNS, etc for our roadmap, which feels quite different from what is being shown here. We do have pure LLM work, but more about fine-tuning smaller or smarter models, and we find that to be a tiny % of the part people care about. Ex: Spam classifications flowing into our RAG/KG pipelines or small model training is more important to us than it flowing into a big model training. Long-term, I do expect growing emphasis on the big models we use, but that is a more nuanced discussion. (We have been piloting w gov types and are preparing for next cohorts, in case useful on real problems for anyone.)
- pests 3y agoIsn't that... the same thing as search?
- tootie 3y agoHow is it that OpenAI was touted like it was some massive years-long effort that blew all AI research out of the water and now we have so many competitors popping up one after another?
- ben_w 3y agoEgg of Columbus. Also, the general architecture is well documented, ChatGPT (specifically the chat interface, not GPT-3, not InstructGPT) is what made a lot of people care, and actually reproducing it requires someone wanting to in the first place.
- longdog 3y agoYou don't need to be a cutting edge research scientist to train a SOTA LLM. You just need money for scaling. OpenAI's "secret" was just their willingness to spend tens/hundreds of millions without guaranteed returns, and RLHF/instruct fine tuning, both of which are out of the bag now.
- simonw 3y agoDisagree. It took more than 12 months from the release of GPT-4 to someone else producing a model of equivalent quality, and that definitely wasn't due to a shortage of investment from the competition. There's a huge amount of depth in training a really good LLM. Not helped by the fact that iteration is incredibly expensive - it might take several months (and millions of dollars) before you can tell if your new model is working well or if there was some mistake in the pipeline that lead to a poor quality result. Almost all of the world-class LLMs outside of OpenAI/DeepMind have been trained by people who previously worked at those organizations - giving them invaluable experience such that they could avoid the most expensive mistakes while training their new models.
- barrell 3y agoWhile I do agree there is some amount of secret sauce, keep in mind the training takes several months. So from someone to see the success of GPT4, decide they want to invest that amount of money to train the same, raise the money to train the model, find someone competent to supervise the training, train the model for several months, then test and integrate it could easily be a year long even if there was no secret sauce.
- asciii 3y agoI love the citation for image in the article > The cover image was generated using Midjourney based on the following prompt proposed by Grok: A 3D illustration of a neural network, with transparent nodes and glowing connections, showcasing the varying weights as different thicknesses and colors of the connecting lines.