7 ms·

Over the last two weeks, I ran several unsystematic comparisons of three reasoning models: ChatGPT o1, DeepSeek’s then-current DeepThink, and Gemini 2.0 Flash T

Over the last two weeks, I ran several unsystematic comparisons of three reasoning models: ChatGPT o1, DeepSeek’s then-current DeepThink, and Gemini 2.0 Flash Thinking Experimental. My tests involved natural-language problems: grammatical analysis of long texts in Japanese, New York Times Connections puzzles, and suggesting further improvements to an already-polished 500-word text in English. ChatGPT o1 was, in my judgment, clearly better than the other two, and DeepSeek was the weakest. I tried the same tests on DeepSeek-R1 just now, and it did much better. While still not as good as o1, its answers no longer contained obviously misguided analyses or hallucinated solutions. (I recognize that my data set is small and that my ratings of the responses are somewhat subjective.) By the way, ever since o1 came out, I have been struggling to come up with applications of reasoning models that are useful for me. I rarely write code or do mathematical reasoning. Instead, I have found LLMs most useful for interactive back-and-forth: brainstorming, getting explanations of difficult parts of texts, etc. That kind of interaction is not feasible with reasoning models, which can take a minute or more to respond. I’m just beginning to find applications where o1, at least, is superior to regular LLMs for tasks I am interested in.

synergy20 2y ago
a dumb question,how did you use deepseek,e.g r1?
- tkgally 2y ago
  I use it at https://chat.deepseek.com/ https://chat.deepseek.com/ . It’s free but requires a log-in. Now, when I hover over the “DeepThink” button below the prompt field, a pop-up appears saying “Use DeepSeek-R1 to solve reasoning problems.”
- deleted 2y ago
  [deleted]
torginus 2y ago
o1 is impressive, I tried feeding it some of the trickier problems I have solved (that involved nontrivial algorithmic challenges) over the past few months, and it managed to solve all of them, and usually came up with slightly different solutions than I did, which was great. However what I've found odd was the way it formulated the solution was in excessively dry and obtuse mathematical language, like something you'd publish in an academic paper. Once I managed to follow along its reasoning, I understood what it came up with could essentially be explain in 2 sentences of plain english. On the other hand, o1 is amazing at coding, being able to turn an A4 sheet full of dozens of separate requirements into an actual working application.
- manmal 2y ago
  Do you mean o1-preview or the current o1? I rarely get anything really useful out of the current one ($20 subscription, not the 200 one). They seem to have seriously nerfed it.
  - torginus 2y ago
    o1, not a big user, but haven't used a big model before, only Sonnet and GPT4 so this all seems new and wonderful to me
  - BoorishBears 2y ago
    o1 has a parameter that affects how long it's willing to think for, whereas o1-preview did not It's likely o1-preview was permanently pinned at max thinking, and o1 is not
- joshstrange 2y ago
  > actual working application Working != maintainable The things that ChatGPT or Claude spit out are impressive one-shots but hard to iterate on or integrate with other code. And you can’t just throw Aider/Cursor/Copilot/etc at the original output without quickly making a mess. At least not unless you are nudging it in the right directions at every step, occasionally jumping in and writing code yourself, fixing/refactoring the LLM code to fit style/need, etc.
  - WesleyJohnson 2y ago
    This is how I use Cursor Composer Agents. Detailed outline up front and see what it comes up with. I then use it to iterate on that idea. Sometimes it breaks things, so I'll have to reject/revert the change and then ask it again, but tell it not to change XYZ. If it starts going down the wrong path, I'll step it and code it myself. But I've ran into cases where the next question I ask it seems to be based on the state of the code form its last change, not the current state as I have changed it. So that can be frustrating. I've really only done greenfield hobby projects with it so far. Hesitant to throw larger things at it that have been growing for 8/9 years. But, there's always undo or `git reset`. :P
starfezzy 2y ago
Can it solve easy problems yet? Weirdly, I think that's an important milestone. Prompts like, "Give me five odd numbers that don't have the letter 'e' in their spelling," or "How many 'r's are in the word strawberry?" I suspect the breakthrough won't be trivial that enables solving trivial questions.
- msoad 2y ago
  > Give me five odd numbers that don't have the letter 'e' in their spelling Compare the reasoning times!!! 84s vs 342s R1 (Thought for 84 seconds) No odd number in English avoids the letter 'e' in its spelling. The request for five such numbers cannot be fulfilled. o1 Pro (Thought for 5 minutes and 42 seconds) No standard English spelling of an odd number can avoid “e.” Every odd digit (one, three, five, seven, nine) already includes “e,” so once you build any odd number out of those digits, the spelled‐out form will contain “e.” As a result, there are no such odd numbers in standard English without an “e” in their name.
  - coffeebeqn 2y ago
    Took 1m 36s for me. My default prompt is a bit different “think from first principles”. It’s pretty verbose but I enjoyed looking through all the work it did. Pretty impressive !
  - mattkrause 2y ago
    Even after five minutes, the logic has a (small) hole. Not all odd spell out the digit: thirteen, fifteen (and maybe eleven?) are odd but don’t have an ‘e’ in the word’s “digit” part, though they obviously do elsewhere.
    - ben_w 2y ago
      If the answer is supposed to be logically rigorous, o1 is incomplete for the reason you say. If I was treating this as a lateral thinking puzzle (does anyone still do those for interview questions?) rather than a sensible request, I'd give numbers in a non-Latin character set, at which point it's easy: 一三五七九 (But even this only works for silly games, IMO).
rcpt 2y ago
I found that reasoning models were good for CAD. I can ask for openSCAD code to produce and then add to some kind of shape
- coderenegade 2y ago
  I think this is a massive dark horse application of LLMs. Something like DeepSeek that is able to produce OpenSCAD code for complex parts is a game changer for a lot of industries and individuals. Effectively free part design, in a way that libraries of standard parts can rapidly be built, and the code version-controlled using standard software dev tooling.
deleted 2y ago
[deleted]
thefourthchime 2y ago
I completely agree, for my day-to-day use o1 isn't needed. I only use it for complicated solutions involving code.

synergy20 2y ago

a dumb question,how did you use deepseek,e.g r1?

tkgally 2y ago
I use it at https://chat.deepseek.com/ https://chat.deepseek.com/ . It’s free but requires a log-in. Now, when I hover over the “DeepThink” button below the prompt field, a pop-up appears saying “Use DeepSeek-R1 to solve reasoning problems.”
deleted 2y ago
[deleted]

torginus 2y ago

o1 is impressive, I tried feeding it some of the trickier problems I have solved (that involved nontrivial algorithmic challenges) over the past few months, and it managed to solve all of them, and usually came up with slightly different solutions than I did, which was great. However what I've found odd was the way it formulated the solution was in excessively dry and obtuse mathematical language, like something you'd publish in an academic paper. Once I managed to follow along its reasoning, I understood what it came up with could essentially be explain in 2 sentences of plain english. On the other hand, o1 is amazing at coding, being able to turn an A4 sheet full of dozens of separate requirements into an actual working application.

manmal 2y ago
Do you mean o1-preview or the current o1? I rarely get anything really useful out of the current one ($20 subscription, not the 200 one). They seem to have seriously nerfed it.
- torginus 2y ago
  o1, not a big user, but haven't used a big model before, only Sonnet and GPT4 so this all seems new and wonderful to me
- BoorishBears 2y ago
  o1 has a parameter that affects how long it's willing to think for, whereas o1-preview did not It's likely o1-preview was permanently pinned at max thinking, and o1 is not
joshstrange 2y ago
> actual working application Working != maintainable The things that ChatGPT or Claude spit out are impressive one-shots but hard to iterate on or integrate with other code. And you can’t just throw Aider/Cursor/Copilot/etc at the original output without quickly making a mess. At least not unless you are nudging it in the right directions at every step, occasionally jumping in and writing code yourself, fixing/refactoring the LLM code to fit style/need, etc.
- WesleyJohnson 2y ago
  This is how I use Cursor Composer Agents. Detailed outline up front and see what it comes up with. I then use it to iterate on that idea. Sometimes it breaks things, so I'll have to reject/revert the change and then ask it again, but tell it not to change XYZ. If it starts going down the wrong path, I'll step it and code it myself. But I've ran into cases where the next question I ask it seems to be based on the state of the code form its last change, not the current state as I have changed it. So that can be frustrating. I've really only done greenfield hobby projects with it so far. Hesitant to throw larger things at it that have been growing for 8/9 years. But, there's always undo or `git reset`. :P

starfezzy 2y ago

Can it solve easy problems yet? Weirdly, I think that's an important milestone. Prompts like, "Give me five odd numbers that don't have the letter 'e' in their spelling," or "How many 'r's are in the word strawberry?" I suspect the breakthrough won't be trivial that enables solving trivial questions.

msoad 2y ago
> Give me five odd numbers that don't have the letter 'e' in their spelling Compare the reasoning times!!! 84s vs 342s R1 (Thought for 84 seconds) No odd number in English avoids the letter 'e' in its spelling. The request for five such numbers cannot be fulfilled. o1 Pro (Thought for 5 minutes and 42 seconds) No standard English spelling of an odd number can avoid “e.” Every odd digit (one, three, five, seven, nine) already includes “e,” so once you build any odd number out of those digits, the spelled‐out form will contain “e.” As a result, there are no such odd numbers in standard English without an “e” in their name.
- coffeebeqn 2y ago
  Took 1m 36s for me. My default prompt is a bit different “think from first principles”. It’s pretty verbose but I enjoyed looking through all the work it did. Pretty impressive !
- mattkrause 2y ago
  Even after five minutes, the logic has a (small) hole. Not all odd spell out the digit: thirteen, fifteen (and maybe eleven?) are odd but don’t have an ‘e’ in the word’s “digit” part, though they obviously do elsewhere.
  - ben_w 2y ago
    If the answer is supposed to be logically rigorous, o1 is incomplete for the reason you say. If I was treating this as a lateral thinking puzzle (does anyone still do those for interview questions?) rather than a sensible request, I'd give numbers in a non-Latin character set, at which point it's easy: 一三五七九 (But even this only works for silly games, IMO).

rcpt 2y ago

I found that reasoning models were good for CAD. I can ask for openSCAD code to produce and then add to some kind of shape

coderenegade 2y ago
I think this is a massive dark horse application of LLMs. Something like DeepSeek that is able to produce OpenSCAD code for complex parts is a game changer for a lot of industries and individuals. Effectively free part design, in a way that libraries of standard parts can rapidly be built, and the code version-controlled using standard software dev tooling.

deleted 2y ago

[deleted]

thefourthchime 2y ago

I completely agree, for my day-to-day use o1 isn't needed. I only use it for complicated solutions involving code.