6 ms·
DeepSeek-Prover-V2
- Alifatisk 1y agoIs this model hosted at Deepseek chat too? Couldn’t find it yesterday and I prefer not to selfhost because lack of good hardware.
- simianwords 1y agorelated: I imagine in the future we might several "expert" LLM's and a wrapper can delegate tasks as needed as if it were a "tool". That way we can have segregation of expertise - each individual model can excel at one single thing. A prover model might be used as a tool in the coming future.
- koakuma-chan 1y ago> related: I imagine in the future we might several "expert" LLM's and a wrapper can delegate tasks as needed as if it were a "tool". That way we can have segregation of expertise - each individual model can excel at one single thing. In the future? I'm pretty sure people do that already.
- energy123 1y agoIt's crudely done though.
- kratom_sandwich 1y agoMistrals model is a mixture-of-experts model
- simianwords 1y agoNo I disagree. I would want ChatGPT to abstract away expert models - biochemistry model, coding model, physics model and maybe O3 would use these models as tools to come up with an answer. The point being that a separate expert model would be better at its own field than a single model that tries to be good at everything. Intuitively it makes sense, in practice I have seen anecdotes where finetuning a small model on domain data makes the model lose coherence on other topics.
- koakuma-chan 1y ago> have seen anecdotes where finetuning a small model on domain data makes the model lose coherence on other topics This is expected behaviour.
- simianwords 1y agoi know. so why don't we have domain specific models as tools in consumer llm products
- deleted 1y ago[deleted]
- Arcuru 1y agoFor a concrete example today, see https://openrouter.ai/openrouter/auto https://openrouter.ai/openrouter/auto
- simianwords 1y agothats nice but imagine first having models that are expert in specific domains. routing seems to be the easy part (just feed the available models as tools to your wrapper LLM)
- someguy101010 1y agoThe No Free Lunch Theorem implies that something like this is inevitable https://en.wikipedia.org/wiki/No_free_lunch_in_search_and_optimization https://en.wikipedia.org/wiki/No_free_lunch_in_search_and_op...
- repsilat 1y agoA system of n experts is no different to a single expert wrt the NFLT. The theorem is entirely indifferent to (ie "equally skeptical of") the idea.
- samvaran 1y agoIs that not what MoE models already do?
- oofbaroomf 1y agoNo. Each expert is not separately trained, and while they may store different concepts, they are not meant to be different experts in specific domains. However, there are certain technologies to route requests to different domain expert LLMs or even fine-tuning adapters, such as RouteLLM.
- retinaros 1y agoThat might already happen behind what they call test time compute
- oofbaroomf 1y agoMany models that use test time compute are MoEs, but test-time compute is generally meant to refer to reasoning about the prompt/problem the model is given, not about reasoning about which model to pick, and I don't think anyone has released an LLM router under that name.
- retinaros 1y agowe dont know what OAI does to find the best answer when reasoning but I am pretty sure that having variations of a same model is part of it.
- woah 1y agoWhy do you think that a hand-configured selection between "different domains" is better than the training-based approach in MoE?
- oofbaroomf 1y agoFirst off, they are basically completely different technologies, so it would be disingenuous to act like it's an apples-to-apples comparison. But a simple way to see it is that when you pick between multiple large models that have different strengths, you have a larger amount of parameters just to work with (e.g. Deepseek R1 + V3 + Qwen + LLaMA ends up being 2 trillion total parameters to pick from), whereas "picking" the experts in an MoE like has a smaller amount of total different parameters you are working with (e.g. R1 is 671 billion, Qwen is 235).
- deleted 1y ago[deleted]
- smusamashah 1y agoThat Putnam bench graph (middle one) is showing 49/658 solve rate. > The resulting model, DeepSeek-Prover-V2-671B, achieves state-of-the-art performance in neural theorem proving, reaching 88.9% pass ratio on the MiniF2F-test and solving 49 out of 658 problems from PutnamBench. Which is 0.07% (edit: 7%) for PutnamBench
- darkmighty 1y ago49/658 is 7%
- smusamashah 1y agoSorry, forgot multiply by 100
- booi 1y agoI bet DeepSeek-Prover-V2 wouldn't have made that mistake
- gallerdude 1y agoclassic human hallucination
- HappyPanacea 1y agoHow likely is it that Putnam answers were in DeepSeek's training data?
- EvgeniyZh 1y agoThe solutions weren't published anywhere. There is also no good automatic way to generate solutions as far as I know, even expensive ones (previous sota was 10 solutions and one before was 8 using pass@3200 for 7b model). Potentially the developers could've paid some people who are good in putnam-level math problems and lean to write solutions for LLMs. It is hard to estimate likelihood of that but it sounds like waste of money given relatively marginal problem/benchmark.
- islewis 1y ago> The cold-start training procedure begins by prompting DeepSeek-V3 to decompose complex problems into a series of subgoals It feels pretty intuitive to me that the ability for an LLM to break a complex problem down into smaller, more easily solvable pieces will unlock the next level of complexity. This pattern feels like a technique often taught to junior engineers- how to break up a multi-week project into bitesized tasks. This model is obviously math focused, but I see no reason why this wouldn't be incredibly powerful for code based problem solving.
- cadamsdotcom 1y agoAnd it should be powerful for breaking down reasoning chains of thought too.
- criley2 1y agoImo current models can already break things up into bite sized pieces. The limiter I've seen is twofold 1) Maintaining context of the overall project and goals while working in the weeds on a subtask of a task on an epic (so to speak) both in terms of what has been accomplished already and what still needs to be accomplished and 2) Getting an agentic coding tool which can actually handle the scale of doing 50 small projects back to back. With these agentic tools I find they start projects off really strong but by task #5 they're just making a mess with every change. I've played with keeping basically a dev-progress.md file and implementation-plan.md file that I keep in context for every request and end each task by updating files. But me manually keeping all this context isn't solving all my problems. And all the while, tools like Cline are gobbling up 2M tokens to make small changes.
- jhrmnn 1y ago> Maintaining context of the overall project and goals while working in the weeds on a subtask of a task on an epic (so to speak) both in terms of what has been accomplished already and what still needs to be accomplished This is a struggle for every human I’ve ever worked with
- 1y ago
- jasonjmcghee 1y agoSuper interesting that they chose 671B and 7B. no like 32B which feels like a "sweet spot"
- bredren 1y agoAlso notable is the earliest planning for a positive reception release of a new model might include both parameter-based and skill type market segmentation. --> "In an increasingly crowded field of LLMs, how will our (costly to produce) model stand out?"
- SweetSoftPillow 1y agoI feel like this is very logical way to do things. Test hypothesis on small model, play around, get it working, apply findings to big model.
- ddlsmurf 1y agoor they did and it wasn't sweet ? (no idea but seems they would before redacting a publication)
- versteegen 1y agoLikely because they haven't got their own suitable SoTA base models of any other size to build on. DeepSeek V3 is 671B, and DeepSeek-Prover-v1.5 [1] is 7B only, based on DeepSeekMath which is 7B, which is based on DeepSeekCoder-Base-7B-v1.5. Maybe DeepSeek-Coder-V2 (16B and 236B) would be a good start but it's merged into DeepSeek V2.5, and V2.5 is inferior to V3. Or some version of Qwen. [1] https://github.com/deepseek-ai/DeepSeek-Prover-V1.5 https://github.com/deepseek-ai/DeepSeek-Prover-V1.5
- qoez 1y agoThe best part about these is that I know the weights are static so I know I won't have to deal with a sassy unusable update for a week suddenly.
- Implicated 1y agoOr, like with Claude, it being effectively lobotomized during north american 'business' hours. 3am PST? Cracked. 8am PST? ... mentally challenged.
- navanchauhan 1y agoThis is pretty interesting. Do you have more information about this?
- devoutsalsa 1y agoI’m pretty sure the parent comment is referring to capacity constraints. When the Americans come online in the morning, Claude frequently can’t keep up with demand and error messages saying the system is at capacity are common.
- aitchnyu 1y agoI used Openrouter for Claude and Deepseek which chooses between alternative hosts for a given model. I excluded Deepseek provider as its underperforming. https://openrouter.ai/deepseek/deepseek-chat-v3-0324 https://openrouter.ai/deepseek/deepseek-chat-v3-0324
- whatshisface 1y agoI've noticed it too. When it started the overcapacity messages went away. I think they are switching to models with fewer parameters during oversubscribed hours.
- ekez 1y agoI wonder if the authors have tried incorporating error feedback from Lean into their models. Work from 2023 [1] showed general purpose models did better when they were able to incorporate error feedback, humans incorporate error feedback, but none of the SOTA models on minif2f seem to. [1]: https://arxiv.org/abs/2310.04353 https://arxiv.org/abs/2310.04353
- johnmcd3 1y agoIn a way, DeepSeek Prover's subgoal decomposition is a partial-step towards error/proof-state feedback. (DS Prover breaks down a proof into subgoals and attacks each subgoal separately with batched sampling, then puts the pieces back together.) This is distinct from the approach of the previous SOTA for an open-weights model (Kimina Prover) which generated at the full-proof level. While it was very impressive to see Kimina's ability to generate medium-length proofs (think AIME-level problems) without sub-goals or feedback at intermediate steps, it's likely that at least subgoal decomposition will be required for longer proofs (think IMO-level problems.) I certainly agree that where and how error/proof state feedback is best incorporated (training data synthesis / reward function / CoT during inference / etc.) is a fascinating area of research. (It's rumored that GDM's AlphaProof does use proof state / lean feedback already.)
- MJGrzymek 1y agoThat's surprising to learn. I'm surprised those even use actual lean code instead of like raw type theory.
- revskill 1y agoThe way intelligence works to me, is more about: - Making correct and smart assumption. Currently all LLM bots are too stupid at making good assumptions. I don't want to explicitly repeat and repeat again my own assumptions while the context is clear enough. Hey bots, try harder. - LLM bot needs to bring their own secondary and contextual memory in reasoning, i don't want to do it for you, ok ? You're the bot. - Thinking out of the box. This is the final stage of intelligence. Adapt old technique to make your own technique to solve non-existing before problems.
- szundi 1y ago[dead]
- nthingtohide 1y agoI propose human-AI interaction data must be made public. This is our collective wikipedia of AI era. Otherwise our progress will be blank line after 2022. Just as Egyptians didn't write down process to move giant rocks.
- whatshisface 1y agoHow much education would a human need to perform at this level on the benchmarks?
- pama 1y agoLearning to formalize math to then prove Putnam competition problems rigorously in Lean would require you to have mid-to-advanced college level math and CS background. (Learning to do a small fraction of the Putnam competition without using Lean probably only needs strong highschool math and early undergrad math, with training for competitions a strong bonus.)
- Fokamul 1y ago>"an open-source large language model" Is it really opensource? Something changed?
- eightysixfour 1y agoEveryone calls "open weight" models "open source" at this point, it is wrong, but we'll have to find another way to fight this fight. Maybe "open data and pipeline" or something.
- mcshicks 1y agoYou can run the model for free on openrouter.ai. I have played around with lean, slowly working my way through mathmatics in lean. I was stuck on this problem in section 3.6 theorem convergesTo_unique {s : ℕ → ℝ} {a b : ℝ} (sa : ConvergesTo s a) (sb : ConvergesTo s b) : For fun I tried it on the free model on openrouter.ai. Got the answer the first time. https://leanprover-community.github.io/mathematics_in_lean/mathematics_in_lean.pdf https://leanprover-community.github.io/mathematics_in_lean/m... Here's the answer just to give you a feel. by_contra h have h₁ : a ≠ b := h have h₂ : |a - b| > 0 := by apply abs_pos.mpr exact sub_ne_zero.mpr h₁ -- Use the definition of convergence to find N₁ and N₂ have h₃ := sa (|a - b| / 2) (by linarith) have h₄ := sb (|a - b| / 2) (by linarith) cases' h₃ with N₁ h₃ cases' h₄ with N₂ h₄ -- Choose N to be the maximum of N₁ and N₂ let N := max N₁ N₂ have h₅ := h₃ N (by simp [N, le_max_left]) have h₆ := h₄ N (by simp [N, le_max_right]) -- Derive a contradiction using the triangle inequality have h₇ : |s N - a| < |a - b| / 2 := by simpa using h₅ have h₈ : |s N - b| < |a - b| / 2 := by simpa using h₆ have h₉ : |a - b| < |a - b| := by calc |a - b| = |a - s N + (s N - b)| := by ring_nf _ ≤ |a - s N| + |s N - b| := by apply abs_add _ = |s N - a| + |s N - b| := by rw [abs_sub_comm] _ < |a - b| / 2 + |a - b| / 2 := by linarith _ = |a - b| := by ring linarith
- amelius 1y agoIs someone working on similar ideas for compiler back-ends?
- hartator 1y agoAny way to install via ollama? Like ollama run deepseek-ai/DeepSeek-Prover-V2-7B
- good-luck86523 1y ago[flagged]
- yahaya12 1y agoprediction for astonvilla vs fulham