4 ms·
i've been noticing that there's a divergence in philosophy between Llama style LLMs (Mistral are Meta alums so I'm counting them in tehre) and OpenAI/GPT style
by swyx 2y ago
i've been noticing that there's a divergence in philosophy between Llama style LLMs (Mistral are Meta alums so I'm counting them in tehre) and OpenAI/GPT style LLMs when it comes to code.
GPT3.5+ prioritized code very heavily - there's no CodeGPT, its just GPT4, and every version is better than the last.
Whereas the Llama/Mistral models are now shipping the general language model first, then adding CodeLlama/Codestral with additional pretraining (it seems like we don't know how much more tokens are on this one, but CodeLLama was 500B-1T extra tokens of code).
Zuck has mentioned recently that he doesnt see coding ability as important for his usecases, whereas obviously OpenAI is betting heavily on code as a way to improve LLM reasoning for AGI.
- memothon 2y ago>Zuck has mentioned recently That's a really surprising thing to hear, where did you see that? The only quote I've seen is this one: >“One hypothesis was that coding isn’t that important because it’s not like a lot of people are going to ask coding questions in WhatsApp,” he says. “It turns out that coding is actually really important structurally for having the LLMs be able to understand the rigor and hierarchical structure of knowledge, and just generally have more of an intuitive sense of logic.” https://www.theverge.com/2024/1/18/24042354/mark-zuckerberg-meta-agi-reorg-interview https://www.theverge.com/2024/1/18/24042354/mark-zuckerberg-...
- imachine1980_ 2y agoMake Sense, they want better interaction whit users for Whatsapp, Instagram and Facebook marketers, content creation and moderation,and their glasses(ai /ar) I don't see in that context why the should push more effort into llm coding, is sad anyways
- whoami_nr 2y agoHe mentioned it on the Dwarkesh podcast: https://www.youtube.com/watch?v=bc6uFV9CJGg https://www.youtube.com/watch?v=bc6uFV9CJGg
- Davidzheng 2y agoI watched this podcast and i also remember zuck saying it is important
- tkellogg 2y agoThe OpenAI philosophy is that adding modes improves everything. Sure, it’s astronomically expensive, but I tend to think they’re on to something.
- deleted 2y ago[deleted]
- guyomes 2y ago> OpenAI is betting heavily on code as a way to improve LLM reasoning for AGI. And researchers from Google Deepmind, University of Wisconsin-Madison and Laboratoire de l’Informatique du Parallélisme, University of Lyon, actually publish some of their results in that direction [1,2]. [1]: https://deepmind.google/discover/blog/funsearch-making-new-discoveries-in-mathematical-sciences-using-large-language-models/ https://deepmind.google/discover/blog/funsearch-making-new-d... [2]: https://www.nature.com/articles/s41586-023-06924-6 https://www.nature.com/articles/s41586-023-06924-6
- Rastonbury 2y agoI thought that was the idea, open source small specific models that most people can run vs general purpose ones that require a massive amount of GPUs
- behnamoh 2y ago> Zuck No, if anything he said Meta realized coding abilities make the model overall better, so they focused on those more than before.
- nabakin 2y ago> there's no CodeGPT, its just GPT4 Codex[1] is OpenAI's CodeGPT. It's what powers GitHub Copilot and it is very good but not publicly accessible. Maybe they don't want something else to outcompete Copilot. [1] https://openai.com/index/openai-codex/ https://openai.com/index/openai-codex/