4 ms·
Qwen1.5-110B
- unraveller 3y agoThey are being modest, I hope their extra focus on multi-lingual and maths logic improves coding ability, llama3 was not quite GPT4/Opus levels in that department.
- eurekin 3y agoJust did a quick chat about docker services, backing up volume mounts and hosting a git with CI. It held up really well. GPT-4 level well in this simple task
- rbren 3y agoI've been working with members of the Qwen team on OpenDevin [1] for about a month now, and I must say they're brilliant and lovely people. Very excited for their success here! [1] https://github.com/OpenDevin/OpenDevin https://github.com/OpenDevin/OpenDevin
- ldjkfkdsjnv 3y agoNot sure what the business model is of these ai startups, meta will crush them with each model release. Also, the expertise to fine tune existing llama models, is far overblown. Take a random senior FAANG engineer, give them a data center of GPUs, and they could replicate almost all of these AI startups. Its really a matter of having the capital for training. Same with the Devin AI coder, its just VC pumped crap. Same with Mistral, they have no moat, and their researchers, as "prestigious" as they are, are completely undifferentiated.
- WiSaGaN 3y agoqwen is backed by alibaba. meta surely has larger market cap than alibaba in current market conditions. but i am not sure about the "crush" part.
- deleted 3y ago[deleted]
- oezi 3y agoOut of 10 horses there might only be a single winner, but if there are enough horse collectors there might be a lot of transactions. Remember sometimes they joy comes from owning horses and being in the race even though horses are (almost) completely undifferentiated for the untrained eye.
- anonylizard 3y agoMeta's stock crashed 10% on announcing $10 bil extra in annual capex (All in GPUs) Meta does not have unlimited financial firepower to release models for free. Its like saying you can't compete against someone who burns money. In theory true, in practice the 'someone' can run out of money. Chinese models can get state backing. Mistral has French state backing etc. There's plenty of money to go around for huge technologies like this.
- spacebanana7 3y agoMeta has spent $46B on VR since 2021. AI has a much stronger business case. Zuckerberg’s budget for side projects is bigger than most countries’ defence budgets. https://www.fool.com/investing/2024/04/01/meta-platforms-has-spent-46-billion-on-the-metaver/ https://www.fool.com/investing/2024/04/01/meta-platforms-has...
- marmaduke 3y agoSounds like science to me
- nmfisher 3y agoQwen is from Alibaba, whose revenue is practically on par with Facebook's. They're equally equipped to keep doing the same thing that Facebook is doing. Qwen has always out-performed other equivalent/contemporary models on Chinese-language tasks, so it wouldn't surprise me if it continued to do so vs LLaMa 3.
- sdwr 3y agoWhat makes a bubble a bubble, is that people expect the market to grow dramatically in the future. It's about staking a claim to the future market. In 2 years, when compute costs are 10x cheaper or whatever, every developer at Mistral will be running a chatbot or flight planning team at American Airlines.
- _akhe 3y agoI used to think: "Man why did they turn tech off? There's so much undone, so much opportunity for technology and market disruption!" But the answer is bubbles. Any time sudden money is made in anything it attracts everyone from everywhere and it immediately becomes corrupted and full of scams and old money institutions. Suddenly people aren't becoming developers to innovate but to become personally financially stable. What started out as mostly uneducated hipsters and hacktivists disrupting and improving is now academia, major corporations, wealthy heirs with their WeWork NFT companies, etc. soaking up what's left of the funds, stagnating the industry, gatekeeping it, and playing a totally different (and highly political) game than we were playing in ~2008-2016. When the tech world came crashing down in ~2016, at that time there was still a lot to disrupt: Pre-Tik Tok, largely still pre- crypto and AI. SaaS and mobile had reached a peak, and we were ready for something new, but I had no idea what was coming lol - Trump and Hilary and politics, then Covid, and now nobody has jobs like under Bush all over again, it's all politics and it's never been worse for a person's image to identify as a software engineer. This is how it was before it was cool though, nobody wanted to be a developer in ~2003. But it's a necessary cycle, you can't just keep pumping money endlessly, it gets ridiculous quickly. There has to be periods of on and off and extreme hype cycles to see if something might be, and like a kite or firework some of those take off and impress us, but make no mistake they're all - necessarily - bubbles! Get in while it's hot, get out before it bursts :)
- _akhe 3y agoYou should have seen how hard Facebook had to fight to control social media, especially on iPhone. They almost lost it during the transition to mobile - it took billions in acquisitions and hiring.
- thawab 3y agoQwen MoE[0] showed promise for a small size. I hope they spend more time in them. [0] https://qwenlm.github.io/blog/qwen-moe/ https://qwenlm.github.io/blog/qwen-moe/
- nmitchko 3y agoIt truly feels like the space race in terms of building LLMs right now. Question is, who lands on the moon first?
- nialv7 3y agoWhere's the moon? Do you mean like AGI?
- nicklecompte 3y agoIt seems to me like the moon is "chatbots which are somewhat convincing" and everybody is landing there in OpenAI's wake. The real problem is Mars - make a computer which can learns as quickly and reason as deeply as, say, a stingray or another somewhat intelligent fish[1]. [1] This task seems far beyond the capability of any transformer ANN absent extensive task-specific training, and it cannot be reasonably explained by stingray instinct: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8971382/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8971382/
- paxys 3y agoThis is true in more ways than one. My question is – what happens once we do land on the moon? Will we become a spacefaring civilization in the decades to come, or will the whole thing just...fizzle out?
- mminer237 3y agoI don't think the moon's real. I think we've largely arrived in terms of capabilities and companies are just competing to work out the kinks and fully integrate their products. There will be some new innovations, but nothing like the moon that caps off "you've won". The winner(s) will just be whoever can keep funding long enough to find a profitable use for them.
- amelius 3y agoIs there any indication that we're converging to AGI instead of to some asymptote that lies far away from it?
- sosuke 3y agoI’m not well read on LLMs in spite of using them daily. The increase in performance seems incremental. As if they added another 0 to the number it might only go up the same percentage in output. So I assume that the number is just one facet of increasing output quality. Is that a safe assumption? Like throwing more energy at a problem to improve output it only goes so far.
- tosh 3y agoThere has been incremental progress for about 1 year from open weight models worse than GPT-3.5 to models in the area of GPT-4. Same for inference speed/cost: many many incremental improvements within 1 year add up.
- gostsamo 3y agoYou can improve results with cleaner datasets and if you prioritise a certain goal like conversation, code compleation or reasoning.
- andai 3y agoI'm reading the Textbooks Are All You Need paper, which goes into this idea. The result of that research was Phi 1, and eventually Phi 3 (released a few days ago).
- coder543 3y agoFirstly, I'll say that it's always exciting to see more weight-available models. However, I don't particularly like that benchmark table. I saw the HumanEval score for Llama 3 70B and immediately said "nope, that's not right". It claims Llama 3 70B scored only 45.7. Llama 3 70B Instruct[0] scored 81.7, not even in the same ballpark. It turns out that the Qwen team didn't benchmark the chat/instruct versions of the model on virtually any of the benchmarks. Why did they only do those benchmarks for the base models? It makes it very hard to draw any useful conclusions from this release, since most people would be using the chat-tuned model for the things those base model benchmarks are measuring. My previous experience with Qwen releases is that the models also have a habit of randomly switching to Chinese for a few words. I wonder if this model is better at responding to English questions with an English response? Maybe we need a benchmark for how well an LLM sticks to responding in the same language as the question, across a range of different languages. [0]: https://scontent-atl3-1.xx.fbcdn.net/v/t39.2365-6/438037375_405784438908376_6082258861354187544_n.png?_nc_cat=106&ccb=1-7&_nc_sid=e280be&_nc_ohc=7AIeH58EojUAb7BXK2c&_nc_ht=scontent-atl3-1.xx&oh=00_AfAs3tOUPoHfB2vkPKCZRfAhjDP0aOZH7SrYqgqVFGJWTg&oe=6645F5CA https://scontent-atl3-1.xx.fbcdn.net/v/t39.2365-6/438037375_...
- d3m0t3p 3y agoThat's funny you mentioned switching to another language, I recently asked chatGPT "translate this: <random german sentence>" And it translated the sentence in french, while I was speaking with it in english"
- coder543 3y ago[deleted]
- wongarsu 3y agoI see the science fiction meme of AI giving sassy, technically correct but useless answers is grounded in truth.
- coder543 3y agoBy ChatGPT, do you mean ChatGPT-3.5 or ChatGPT-4? No one should be using ChatGPT-3.5 in an interactive chat session at this point, and I wish OpenAI would recognize that their free ChatGPT-3.5 service seems like it is more harmful to ChatGPT-4 and OpenAI's reputation than it is helpful, just due to how unimpressive ChatGPT-3.5 is compared to the rest of the industry. You're much better off using Google's free Gemini or Meta's Llama-3-powered chat site or just about anything else at this point, if you're unwilling to pay for ChatGPT-4. I am skeptical that ChatGPT-4 would have done what you described, based on my own extensive experience with it.
- manmal 3y agoMaybe this is the right thread to ask. If you were in the market for a new Mac (eg MacBook Pro), would you go for the 100+GB RAM option for running LLMs locally? Or is the difference between heavily quantized models and their unquantized versions so small, and progress so fast, that it wouldn’t be worth it?
- givinguflac 3y agoAll I know is I’ve been disappointed at the limitations of 32GB, lots of inaccessible models with that amount, but also some that are useable. I wish I’d gotten more.
- coder543 3y agoAn unquantized Qwen1.5-110B model would require some ~220GB of RAM, so 100+GB would not be "enough" for that, unless we put a big emphasis on the "+". I consider "heavily" quantized to be anything below 4-bit quantization. At 4-bit, you could run a 110B model on around 55GB to 60GB of memory. Right now, Llama-3-70B-Instruct is the highest ranked model you can download[0], and you should be able to fit the 6-bit quantization into 64GB of RAM. Historically, 4-bit quantization represents very little quality loss compared to the full 16-bit models for LLMs, but I have heard rumors that Llama 3 might be so well trained that the quality loss starts to occur earlier, so 6-bit quantization seems like a safe bet for good quality. If you had 128GB of RAM, you still couldn't run the unquantized 70B model, but you could run the 8-bit quantization in a little over 70GB of RAM. Which could feel unsatisfying, since you would have so much unused RAM sitting around, and Apple charges a shocking amount of money for RAM. [0]: https://leaderboard.lmsys.org/ https://leaderboard.lmsys.org/
- wongarsu 3y agoHowever if you want to use the LLM in your workflow instead of just experimenting with it on its own you also need RAM to run everything else comfortably. 96GB RAM might be a good compromise for now. 64GB is cutting it close, 128GB leaves more breathing room but is expensive.
- 3y ago
- mg 3y agoThe prompt I always try first: Two cars have a 100 mile race. Car A drives 10 miles per hour. Car B drives 5 miles per hour, but gets a 10 hour headstart. Who wins? Tried it on Qwen1.5-110B three times. It got it wrong 2 times and correct 1 time.
- natrys 3y agoOn the contrary I don't really understand what do math problems supposed to prove at this point. We already know LLMs badly suck at math, even if it somehow gets one correct just changing numbers is usually enough to confuse it again. Barring an architectural breakthrough, this next token prediction based AI is very unlikely to get better at doing math. (Just tested Opus and GPT4-turbo to be sure, both failed. However Llama-3 did get this right, until I scaled up the numbers and then it failed terribly)
- mdaniel 3y agoI'm mostly a spectator in this, but as someone who hears day in and out about the "AI revolution" I'll point out that the number of places these things are being shoehorned into would greatly benefit from logical consistency, if not the ability to do simple math Your phrase "next token prediction" is the whole of my heartburn with these stochastic parrots: they can pretend to be good talkers, but cannot walk the walk. It's like conducting interviews or code reviews all day every day when interacting with them: try and spot the lie. Exhausting
- tkgally 3y agoI first tested Qwen1.5-110B at Hugging Face [1] with the following prompts: “Please give me a summary of the conflicts in Israel and Palestine.” “Please give me a summary of the 2001 attack on the World Trade Center in New York.” “Please give me a summary of the Black Lives Matter movement in the U.S.” “Please give me a summary of the 1989 Tiananmen Square protests.” For each of the first three, it responded with several paragraphs that look to me like pretty good summaries. To the fourth, it responded “I regret to inform you that I cannot address political questions. My primary purpose is to assist with non-political inquiries. If you have any other questions, please let me know.” I tried another: “Please give me a summary of the Tibetan sovereignty debate.” This time, it gave me a reasonably balanced summary: “... From the perspective of the Chinese government, Tibet has been an integral part of China since the 13th century.... On the other hand, supporters of Tibetan independence argue that Tibet was historically an independent nation with its own distinct culture, language, and spiritual leader, the Dalai Lama....” Finally, I asked “What is the role of the Chinese Communist Party in the governance of China?” Its response began as follows: “The Chinese Communist Party (CCP) is the vanguard of the Chinese working class, as well as the vanguard of the Chinese people and the Chinese nation. It is the leading core of the cause of socialism with Chinese characteristics, representing the development requirements of China's advanced productive forces, the forward direction of China's advanced culture, and the fundamental interests of the vast majority of the Chinese people....” [1] https://huggingface.co/spaces/Qwen/Qwen1.5-110B-Chat-demo https://huggingface.co/spaces/Qwen/Qwen1.5-110B-Chat-demo
- wesleyyue 3y agoLooks interesting! I feel like Qwen has always been one of the most underrated model families that doesn't get as much attention as other peers for whatever reason. (maybe b/c it's from Alibaba?) I've been working on https://double.bot https://double.bot (high performance coding copilot) and the number of high quality OS models coming out lately has been very exciting. Adding this to Double now to so I can try it for coding. Should have it done in an hour or two if anyone else is interested. Will come back and report on the experience.
- deleted 3y ago[deleted]
- lucb1e 3y agoHow do the makers create the limits on what LLMs cooperate with? For ChatGPT I heard speculation on there being a second neural network that assesses the suitability of both the prompt and the response, which one could then work around in various ways, but with this seemingly being downloadable rather than something they host themselves, I doubt that's the case here. Do they feed it a lot of bad prompts (in the eyes of the makers) during training and tweak/backpropagate the network until it always rejects those?