6 ms·
Qwen2-Math
- deleted 2y ago[deleted]
- Leary 2y agoWhere could I try this?
- currycurry16 2y agohttps://huggingface.co/Qwen/Qwen2-Math-72B-Instruct https://huggingface.co/Qwen/Qwen2-Math-72B-Instruct
- tatsuya4 2y agoModelBox have a playground for Qwen Math 7B: https://model.box/try/playground/qwen/qwen2-math-7b-instruct https://model.box/try/playground/qwen/qwen2-math-7b-instruct 1.5b: https://model.box/try/playground/qwen/qwen2-math-1.5b-instruct https://model.box/try/playground/qwen/qwen2-math-1.5b-instru...
- rahimnathwani 2y agoFYI the model is almost 150GB: https://huggingface.co/Qwen/Qwen2-Math-72B-Instruct/tree/main https://huggingface.co/Qwen/Qwen2-Math-72B-Instruct/tree/mai...
- moffkalast 2y agoNobody runs these things at full precision, a 4 bit quant of 70B range models fits into 64 GB. Though you would need more for long context.
- extheat 2y agoA simple equation to approximate it is `memory_in_gb = parameters_in_billions * (bits/8)` So at 32 bit full precision, 70 * (32 / 8) ~= 280GB fp16, 70 * (16 / 8) ~= 140GB 8 bit, 70 * (8 / 8) ~= 70GB 4 bit, 70 * (4 / 8) ~= 35GB However in things like llama.cpp quants sometimes it's mixed so some of the weights are Q5, some Q4, etc, so you usually want to take the higher number.
- moffkalast 2y agoWell that and you also need a fair bit more space for the KV cache which can be a bit unpredictable. Models without GQA, flash attention or 4 bit cache support are really terrible in that regard, plus it depends on context length. Haven't found a good rule of thumb for that yet.
- lostmsu 2y agoHow much ELO on chatbot arena does 4 bit lose vs full fp16/bf16?
- moffkalast 2y agoI think this is the only known benchmark that actually compares different quants: https://oobabooga.github.io/benchmark.html https://oobabooga.github.io/benchmark.html It's not really all that consistent, but larger models can be compressed more without as much loss.
- vessenes 2y agoSample solution for Balkan MO 2023 seems .. questionable? The problem involves players removing stones sequentially and asking which will win with perfect play: the listed answer definitely doesn’t list all possible types of strategies. The answer it gives may be right; in fact I bet it is correct (the second player), but does the qwen team offer the solution as correct including the logic? And is the solution logic correct?
- pathsjs 2y agoI was going to write the same thing. I checked the first three problems and all solutions are partial at best. Now, don't get me wrong, this is still impressive. But putting the problems there with the implication that qwen solves them correctly when it doesn't does not really inspire trust
- tveita 2y agoSame with the Lusophon Mathematical Olympiad 2023 example. It makes a bunch of specious claims about parity. (Adding 4 to a number changes the parity? Therefore, the parity of each colour always changes for each turn? Therefore, the parity will always be the same as it was initially?) And then it concludes that since the parity is right, 2022 of each colour must be a reachable state. Which, as you say, is quite possible the correct answer, but it's really weird to put it out there as an example with no comment on the reasoning.
- sterlind 2y agoUsing human language models for math problems is a bad idea for exactly this reason: judging correctness of proofs in English is subjective, whereas proofs in Lean are objective and easy to machine-check. Deepmind's AlphaProof was so remarkable precisely because it spoke Lean, so its proofs were verifiable. Why waste time on anything else?
- margorczynski 2y agoI would think of two: 1) Much easier to state the problem and basically all the knowledge we have of math is not in the form of Lean proofs 2) It can be applied to a much broader range of domains, math is kinda unique as in most cases verifying something is 100% correct is impossible (and getting such a signal for RL)
- tempfile 2y agoFirst solution (IMO 2002) is completely wrong. It shows that 1,2,3 cubes are not sufficient, and provide an obstacle that doesn't rule out 4 cubes, but does not prove that there actually are 4 cubes that sum to the given number. This is much harder (and I don't know the true answer)
- kevinventullo 2y agoI think it’s about the same level of difficulty as showing you need at least four, but yeah I agree that what is written is insufficient. One solution is that you can write 2002 = 10^3 + 10^3 + 1^3 + 1^3 Then, multiply through by 2002^2001, which is itself a cube since 2001 is divisible by 3.
- tempfile 2y agoActually, I concede. I just didn't spot that.
- ziofill 2y agoI see that they do some decontamination of the datasets, in the hope that the models won't just recite answers from the training data. But in the recent interview with Subbarao Kambhampati on MLST (https://www.youtube.com/watch?v=y1WnHpedi2A https://www.youtube.com/watch?v=y1WnHpedi2A) they explain that models fail as soon as one slightly rephrases the test problems (indicating that they are indeed mostly reciting). I expect this to be the case with this model too.
- eightysixfour 2y agoThis is just not true. There are plenty of private tests with problems that are not in the training set. GPQA is an excellent example.
- refulgentis 2y agoThere's a lot of bunko 3rd-time-rephrased "wisdom" about AI, and enough interest in it that if you have the right title, you can get away with repeating it. I'm a bit surprised that an HN reader, who presumably has first hand experience with them, isn't sure if its just a hashmap lookup.
- beyondCritics 2y agoIt is obious that all of these problems are still way too hard, although sometimes it has ideas. It flawlessly demonstrates how to simplify (2002^2002) mod 9. I recall that there was once a scandalous university exam for future math teachers in germany, which asked to do tasks like that, but all failed the test. With Qwen-2 at hand this might not have happened.
- next_xibalba 2y agoKind of surprised this was released in English first given it was produced by a Chinese group (Alibaba Cloud). I wonder why that is.
- karmasimida 2y agoI mean I think 90% of AI researchers in China, use English as the first choice for publication ... so not surprised?
- m3kw9 2y agoMaybe Training material also mostly English
- ipnon 2y agoThese solutions aren't perfect, but imagine how many more people can become mathematicians now that the price of an elite IMO medal winning tutor can be quantified as Hugging Face hosting costs!
- Strix97 2y agoI don't think that's this is useful tool to become a mathematician. Becoming one does not necessarily involve solving these kinds of puzzle exercises. It's a useful to hone your creative thinking and learning how to approach a mathematical problem, but it wont make you a mathematician.
- meroes 2y agoA mathematician tutor that doesn’t know how to write proofs though. Or at the most charitable, not how human mathematicians write proofs. I’m not talking about using Lean or similar, but the common parlance and rigor outside of theorem provers.
- azinman2 2y ago> This model mainly supports English. We will release bilingual (English and Chinese) math models soon The irony.
- qrian 2y agoThe solution for IMO 2022 is barely a 1/7 solution. It just says ‘ might not satisfy the inequality for all y’ without a proof. That was the point of the question.
- allanren 2y agoQwen2 has been quite good, but still can't compare 9.9 and 9.11