6 ms·

I've been using cursor since it launched, sticking almost exclusively to claude-3.5-sonnet because it is incredibly consistent, and rarely loses the plot. As s

by mvkel 2y ago

I've been using cursor since it launched, sticking almost exclusively to claude-3.5-sonnet because it is incredibly consistent, and rarely loses the plot. As subsequent models have been released, most of which claim to be better at coding, I've switched cursor to it to give them a try. o1, o1-pro, deepseek-r1, and the now o3-mini. All of these models suffer from the exact same "adhd." As an example, in a NextJS app, if I do a composer prompt like "on page.tsx [15 LOC], using shadcn components wherever possible, update this page to have a better visual hierarchy." sonnet nails it almost perfectly every time, but suffers from some date cutoff issues like thinking that shadcn-ui@latest is the repo name. Every single other model, doesn't matter which, does the following: it starts writing (from scratch), radix-ui components. I will interrupt it and say "DO NOT use radix-ui, use shadcn!" -- it will respond with "ok!" then begin writing its own components from scratch, again not using shadcn. This is still problematic with o3-mini. I can't believe it's the models. It must be the instruction-set that cursor is giving it behind the scenes, right? No amount of .cursorrules, or other instruction, seems to get cursor "locked in" the way sonnet just seems to be naturally. It sucks being stuck on the (now ancient) sonnet, but inexplicably, it remains the only viable coding option for me. Has anyone found a workaround?

jvanderbot 2y ago
I've found cursor to be too thin a wrapper. Aider is somehow significantly more functional. Try that.
- dhc02 2y ago
  Aider, with o1 or R1 as the architect and Claude 3.5 as the implementer, is so much better than anything you can accomplish with a single model. It's pretty amazing. Aider is at least one order of magnitude more effective for me than using the chat interface in Cursor. (I still use Cursor for quick edits and tab completions, to be clear).
  - ChadNauseam 2y ago
    I normally use aider by just typing in what I want and it magically does it. How do I use o1 or R1 to play the role of the "architect"?
    - macNchz 2y ago
      You can start it with something like: aider --architect --model o1 --editor-model sonnet Then you'll be in "architect" mode, which first prompts o1 to design the solution, then you can accept it and allow sonnet to actually create the diffs. Most of the time your way works well—I use sonnet alone 90% of the time, but the architect mode is really great at getting it unstuck when it can't seem to implement what I want correctly, or keeps fixing its mistakes by making things worse.
      - cruffle_duffle 2y ago
        I really want to see how apps created this way scale to large codebases. I’m very skeptical they don’t turn into spaghetti messes. Coding is basically just about the most precise way to encapsulate a problem as a solution possible. Taking a loose English description and expanding it into piles of code is always going to be pretty leaky no matter how much these models spit out working code. In my experience you have to pay a lot of attention to every single line these things write because they’ll often change stuff or more often make wrong assumptions that you didn’t articulate. And in my experience they never ask you questions unless you specifically prompt them to (and keep reminding them to), which means they are doing a hell of a lot of design and implementation that unless carefully looked over will ultimately be wrong. It really reminds me a bit of when Ruby on Rails came out and the blogosphere was full of gushing “I’ve never been more productive in my life” posts. And then you find out they were basically writing a TODO app and their previous development experience was doing enterprise Java for some massive non-tech company. Of course RoR will be a breath of fresh air for those people. Don’t get me wrong I use cursor as my daily driver but I am starting to find the limits for what these things can do. And the idea of having two of these LLM’s taking some paragraph long feature description and somehow chatting with each other to create a scalable bit of code that fits into a large or growing codebase… well I find that kind of impossible. Sure the code compiles and conforms to whatever best practices are out there but there will be absolutely no constancy across the app—especially at the UX level. These things simply cannot hold that kind of complexity in their head and even if they could part of a developers job is to translate loose English into code. And there is much, much, much, much more to that than simply writing code.
  - dwaltrip 2y ago
    I haven't tried aider in quite a while, what does it mean to use one model as an architect and another as the implementer?
    - Terretta 2y ago
      Aider now has experimental support for using two models to complete each coding task: - An Architect model is asked to describe how to solve the coding problem. - An Editor model is given the Architect’s solution and asked to produce specific code editing instructions to apply those changes to existing source files. Splitting up “code reasoning” and “code editing” in this manner has produced SOTA results on aider’s code editing benchmark. Using o1-preview as the Architect with either DeepSeek or o1-mini as the Editor produced the SOTA score of 85%. Using the Architect/Editor approach also significantly improved the benchmark scores of many models, compared to their previous “solo” baseline scores (striped bars). https://aider.chat/2024/09/26/architect.html https://aider.chat/2024/09/26/architect.html
      - lukas099 2y ago
        Probably gonna show a lot of ignorance here, but isn’t that a big part of the difference between our brains and AI? That instead of one system, we are many systems that are kind of sewn together? I secretly think AGI will just be a bunch of different specialized AIs working together.
        Terretta 2y ago
        You're in good company in that secret thought. Have a look at this: https://en.wikipedia.org/wiki/Society_of_Mind https://en.wikipedia.org/wiki/Society_of_Mind
        dhc02 2y ago
        Efficient and effective organizations work this way, too: a CEO to plan in broad strokes, employees to implement that vision in specific ways, and managers to make sure their results match expectations.
  - aledalgrande 2y ago
    Same with Cline
zackproser 2y ago
Not trying to be snarky, but the example prompt you provided is about 1/15th the length and detail of prompts I usually send when working with Cursor. I tend to exhaustively detail what I want, including package names and versions because I've been to that movie before...
- jwpapi 2y ago
  Yes this works good for me too rather take your time and do the first prompt right
- inerte 2y ago
  What works nice also is the text to speech. I find it easier and faster to give more context by talking rather than typing, and the extra content helps the AI to do its job. And even though the speech recognition fails a lot on some of the technical terms or weirdly named packages, software, etc, it still does a good job overall (if I don’t feel like correcting the wrong stuff). It’s great and has become somewhat of a party trick at work. Some people don’t even use AI to code that often, and when I show them “hey have you tried this?” And just tell the computer what I want? Most folks are blown away.
  - cadence- 2y ago
    Does the Cursor have text-to-speech functionality?
  - fud101 2y ago
    you mean speech to text right?
    - chefandy 2y ago
      Not for me. I first ask Advanced Voice to read me some code and have Siri listen and email it to an API I wrote which uses Claude to estimate the best cloud provider to run that code based on its requirements and then a n8n script deploys it and send me the results via twilio.
    - 2y ago
torben-friis 2y ago
My experience with cursor and sonnet is that it is relatively good at first tries, but completely misses the plot during corrections. "My attempt at solving the problem contains a test that fails? No problem, let me mock the function I'm testing, so that, rather than actually run, it returns the expected value!" It keeps doing that kind of shenanigans, applying modifications that solve the newly appearing problem while screwing the original attempt's goal. I usually get much better results from regular chatgpt copying and pasting, the trouble being that it is a major pain to handle the context window manually by pasting relevant info and reminding what I think is being forgotten.
- jwpapi 2y ago
  Yes it’s usually worth it to try to write a really good first prompt
  - earleybird 2y ago
    More than once I've found myself going down this 'little maze of twisty passages, all alike'. At some point I stop, collect up the chain of prompts in the conversation, and curate them into a net new prompt that should be a bit better. Usually I make better progress - at least for a while.
    - dr_dshiv 2y ago
      Why is it so hard to share/find prompts or distill my own damn prompts? There must be good solutions for this —
      - whall6 2y ago
        Don’t outsource the only thing left for our brains to do themselves :/
      - garfij 2y ago
        What do you find difficult about distilling your own prompts? After any back and forth session I have reasonably good results asking something like "Given this workflow, how could I have prompted this better from the start to get the same results?"
energy123 2y ago
Context length possibly. Prompt adherence drops off with context, and anything above 20k tokens is pushing it. I get the best results by presenting the smallest amount of context possible, including removing comments and main methods and functions that it doesn't need to see. It's a bit more work (not that much if you have a script that does it for you), but the results are worth it. You could test in the chatgpt app (or lmarena direct chat) where you ask the same question but with minimal hand curated context, and see if it makes the same mistake.
- mvkel 2y ago
  If it's a context issue, it's an issue with how cursor itself sends the context to these reasoning LLMs. Context alone shouldn't be the reason that sonnet succeeds consistently, but others (some which have even bigger context windows) fail.
  - energy123 2y ago
    Yes, that's what I'm suggesting. Cursor is spamming the models with too much context, which harms reasoning models more than it harms non-reasoning models (hypothesis, but one that aligns with my experience). That's why I recommended testing reasoning models outside of Cursor with a hand curated context. The advertised context length being longer doesn't necessarily map 1:1 with the actual ability the models have to perform difficult tasks over that full context. See for example the plots of performance on ARC vs context length for o-series models.
esperent 2y ago
Claude uses Shadcn-ui extensively in the web interface, to the point where I think it's been trained to use it over other UI components. So I think you got lucky and you're asking it to write using a very specific code library that it's good at, because it happens to use it for it's main userbase on the web chat interface. I wonder if you were using a different component library, or using Svelte instead of React, would you still find Claude the best?
- pollinations 2y ago
  I was recently trying to write a relatively simple htmx service with Claude. I was surprised at how much worse it was when it's not React.
MaxLeiter 2y ago
We've been working on solving a lot of these issues with v0.dev (disclaimer: shadcn and I work on it). We do a lot of pre and post-processing to ensure LLMs output valid shadcn code. We're also talking to the cursor/windsurf/zed folks on how we can improve Next.js and shadcn in the editors (maybe something like llms.txt?)
- mvkel 2y ago
  Thanks for all the work you do! v0 is magical. I absolutely love the feature where I can add a chunky component that v0 made to my repo with npx
kristopolous 2y ago
"not" and other function words; usually work fine today but if I'm having trouble, the best thing to do is probably be inclusive, not exclusive.
eagleinparadise 2y ago
Cursor is also very user-unfriendly in providing alternative models to use in composer (agent). There's a heavy reliance on Anthrophic for cursor. Try using Gemini thinking with Cursor. It barely works. Cmd-k outputs the thinking into the code. Its unusable in chat because the formatting sucks. Is there some relationship between Cursor and Anthropic, i wonder. Plenty of other platforms seem very eager to give users model flexibility, but Cursor seems to be lacking. I could be wrong, just an observation.
- ttroyr 2y ago
  Originally, actually there was a relationship between Cursor & OpenAI. Something like Cursor was supported by the OpenAI startup fund. So Cursor seems to have branched out. I think they are just emphasizing the models they find most effective. I'm surprised they haven't (apparently) incorporated Claude prompt caching yet for Sonnet.
OkGoDoIt 2y ago
I also have been less impressed by o1 in cursor compared to sonnet 3.5. Usually what I will do for a very complicated change is ask o1 to architect it, specifically asking it to give me a detailed plan for how it would be implemented, but not to actually implement anything. I then change the model to Sonnet 3.5 to have it actually do the implementation. And on the side of not being able to get models to understand something specific, there’s a place in a current project where I use a special Unicode apostrophe during some string parsing because a third-party API needs it. But any code modifications by the AI to that file always replace it with a standard ascii apostrophe. I even added a comment on that line to the effect of “never replaced this apostrophe, it’s important to leave it exactly as it is!” And also put that in my cursor rules, and sometimes directly in the prompt as well, but it always replaces it even for completely unrelated changes. I’ve had to manually fix it like 10 times in the last day, it’s infuriating.
deleted 2y ago
[deleted]
foobiekr 2y ago
Have you tried any of the specialty services like Augment? I am curious if they are any better or just snake oil.
chrismsimpson 2y ago
I’ve coded in many languages over the years but reasonably new to the TS/JS/Next world. I’ve found if you give your prompts a kind long form “stream of consciousness”, where you outline snippets of code in markdown along with contextual notes and then summarise/outline at the end what you actually wish to achieve, you can get great results. Think a long form, single page “documentation” type prompts that alternate between written copy/contextual intent/description and code blocks. Annotating code blocks with file names above the blocks I’m sure helps too. Don’t waste your context window on redundant/irrelevant information or code, stating a code sample is abridged or adding commented ellipses seems to do the job.
- twilightfringe 2y ago
  ha! good to confirm! I tend to do this, just kind of as a double-check thing, but never sure if it actually worked or if it was a placebo, lol. Or end with "from the user's perspective: all the "B" elements should light up in excitement when you click "C""
- mvkel 2y ago
  Going to try this! Thanks for the tip
- d357r0y3r 2y ago
  By the time I've fully documented and explained what I want to be done, and then review the result, usually finding that it's worse than what I would have written myself, I end up questioning my instinct to even reach for this tool. I like it for general refactoring and day to day small tasks, but anything that's relatively domain-specific, I just can't seem to get anything that's worth using.
  - noahbp 2y ago
    Like most AI tools, great for beginners, time-savers for intermediate users, and frequently a waste of time in domains where you're an expert. I've used Cursor for shipping better frontend slop, and it's great. I skip a lot of trial and error, but not all of it.
Abishek_Muthian 2y ago
Just curious, did you try a code model like Codestral instead of a MoE?
bugglebeetle 2y ago
o3 mini’s date cut-off is 2023, so it’s unfortunately not gonna be useful for anything that requires knowledge of recent framework updates, which includes probably all big frontend stuff.
hombre_fatal 2y ago
I have the same experience. Just today I was integrating a new logging system with my kubernetes cluster. I tried out the OP model to make changes to my yaml files. It would give short snippets and I’d have to keep trial and erroring its suggestions. Eventually I pasted the original prompt to Claude and it one-shot the dang thing with perfect config. Made me wonder why I even try new models.
pknerd 2y ago
OT: How many tokens are being consumed? How much are you paying for Claude APIs?
harshitaneja 2y ago
So I think I finally understood recently why we have these divergent groups with one thinking Claude 3.5 Sonnet is the best model for coding and another that follow the OpenAI SOTA at that moment. I have been a heavy user of ChatGPT, jumping on to pro without even thinking for more than a second once released. Recently though I took a pause from my usual work on statistical modelling, heuristics work and other things in certain deep domains to focus on building client APIs and frontends and decided to again give Claude a try and it is just so great to work with for this usecase. My hypothesis is its a difference of what you are doing. OpenAI O models are much better than others at mathematical modelling and such tasks and Claude for more general purpose programming.
- mycall 2y ago
  Have you used multi-agent chat sessions with each fielding their own specialities and seeing if that improves your use cases aka MoE?
  - harshitaneja 2y ago
    I have not. Any suggestions on which one(s) to explore to get started.
digitcatphd 2y ago
The reality is I suspect one will use different models for different things. Think of it like having different modes of transportation. You might use your scooter, bike, car, jet - depending on the circumstances. A bike was invented 100 years ago? But it may be the best in the right use case. Would still be using DaVinci for some things because we haven't bothered swapping it and it works fine. For me - the value of R1/o3 is visible logic that provides an analysis that can be critiqued by Sonnet 3.5
- ido 2y ago
  I have an even more topical analogy! Using different languages for different tasks. When I need some one off script do automate some drudgery (take all files with certain pattern in their name, for each do some search and replace in the text inside, zip them, upload zip to URL, etc) I use python. When Im working on a multi-platform game I use c# (and unity). When I need to make something very lean that works in mobile browsers I use JS with some light-weight libraries.
adonese 2y ago
My general workflow with ai so far has been this: - I use copilot mostly for writing unit tests. It mostly works well since the unit tests follow a standard template. - I use the chat one for alternating between different approaches and (in)validating certain approaches My day job is a big monorepo, I have not investigated that yet but I believe the models context sizes fall short there and as such the above use cases only works for me.
dhc02 2y ago
I'm going to give you a video to watch. It's not mine, and I don't know much about this particular youtuber, but it really transformed how I think about writing and structuring the prompts I use, which solved problems similar to what you're describing here. https://youtu.be/y_ywOVQyafE?si=IvKjy7QUYgxGPNgD https://youtu.be/y_ywOVQyafE?si=IvKjy7QUYgxGPNgD PS (I have not bought the guy's course and have no idea whether it's any good)

by mvkel 2y ago

jvanderbot 2y ago

I've found cursor to be too thin a wrapper. Aider is somehow significantly more functional. Try that.

dhc02 2y ago
Aider, with o1 or R1 as the architect and Claude 3.5 as the implementer, is so much better than anything you can accomplish with a single model. It's pretty amazing. Aider is at least one order of magnitude more effective for me than using the chat interface in Cursor. (I still use Cursor for quick edits and tab completions, to be clear).
- ChadNauseam 2y ago
  I normally use aider by just typing in what I want and it magically does it. How do I use o1 or R1 to play the role of the "architect"?
  - macNchz 2y ago
    You can start it with something like: aider --architect --model o1 --editor-model sonnet Then you'll be in "architect" mode, which first prompts o1 to design the solution, then you can accept it and allow sonnet to actually create the diffs. Most of the time your way works well—I use sonnet alone 90% of the time, but the architect mode is really great at getting it unstuck when it can't seem to implement what I want correctly, or keeps fixing its mistakes by making things worse.
    - cruffle_duffle 2y ago
      I really want to see how apps created this way scale to large codebases. I’m very skeptical they don’t turn into spaghetti messes. Coding is basically just about the most precise way to encapsulate a problem as a solution possible. Taking a loose English description and expanding it into piles of code is always going to be pretty leaky no matter how much these models spit out working code. In my experience you have to pay a lot of attention to every single line these things write because they’ll often change stuff or more often make wrong assumptions that you didn’t articulate. And in my experience they never ask you questions unless you specifically prompt them to (and keep reminding them to), which means they are doing a hell of a lot of design and implementation that unless carefully looked over will ultimately be wrong. It really reminds me a bit of when Ruby on Rails came out and the blogosphere was full of gushing “I’ve never been more productive in my life” posts. And then you find out they were basically writing a TODO app and their previous development experience was doing enterprise Java for some massive non-tech company. Of course RoR will be a breath of fresh air for those people. Don’t get me wrong I use cursor as my daily driver but I am starting to find the limits for what these things can do. And the idea of having two of these LLM’s taking some paragraph long feature description and somehow chatting with each other to create a scalable bit of code that fits into a large or growing codebase… well I find that kind of impossible. Sure the code compiles and conforms to whatever best practices are out there but there will be absolutely no constancy across the app—especially at the UX level. These things simply cannot hold that kind of complexity in their head and even if they could part of a developers job is to translate loose English into code. And there is much, much, much, much more to that than simply writing code.
- dwaltrip 2y ago
  I haven't tried aider in quite a while, what does it mean to use one model as an architect and another as the implementer?
  - Terretta 2y ago
    Aider now has experimental support for using two models to complete each coding task: - An Architect model is asked to describe how to solve the coding problem. - An Editor model is given the Architect’s solution and asked to produce specific code editing instructions to apply those changes to existing source files. Splitting up “code reasoning” and “code editing” in this manner has produced SOTA results on aider’s code editing benchmark. Using o1-preview as the Architect with either DeepSeek or o1-mini as the Editor produced the SOTA score of 85%. Using the Architect/Editor approach also significantly improved the benchmark scores of many models, compared to their previous “solo” baseline scores (striped bars). https://aider.chat/2024/09/26/architect.html https://aider.chat/2024/09/26/architect.html
    - lukas099 2y ago
      Probably gonna show a lot of ignorance here, but isn’t that a big part of the difference between our brains and AI? That instead of one system, we are many systems that are kind of sewn together? I secretly think AGI will just be a bunch of different specialized AIs working together.
      - Terretta 2y ago
        You're in good company in that secret thought. Have a look at this: https://en.wikipedia.org/wiki/Society_of_Mind https://en.wikipedia.org/wiki/Society_of_Mind
      - dhc02 2y ago
        Efficient and effective organizations work this way, too: a CEO to plan in broad strokes, employees to implement that vision in specific ways, and managers to make sure their results match expectations.
- aledalgrande 2y ago
  Same with Cline

zackproser 2y ago

Not trying to be snarky, but the example prompt you provided is about 1/15th the length and detail of prompts I usually send when working with Cursor. I tend to exhaustively detail what I want, including package names and versions because I've been to that movie before...

jwpapi 2y ago
Yes this works good for me too rather take your time and do the first prompt right
inerte 2y ago
What works nice also is the text to speech. I find it easier and faster to give more context by talking rather than typing, and the extra content helps the AI to do its job. And even though the speech recognition fails a lot on some of the technical terms or weirdly named packages, software, etc, it still does a good job overall (if I don’t feel like correcting the wrong stuff). It’s great and has become somewhat of a party trick at work. Some people don’t even use AI to code that often, and when I show them “hey have you tried this?” And just tell the computer what I want? Most folks are blown away.
- cadence- 2y ago
  Does the Cursor have text-to-speech functionality?
- fud101 2y ago
  you mean speech to text right?
  - chefandy 2y ago
    Not for me. I first ask Advanced Voice to read me some code and have Siri listen and email it to an API I wrote which uses Claude to estimate the best cloud provider to run that code based on its requirements and then a n8n script deploys it and send me the results via twilio.
  - 2y ago

torben-friis 2y ago

My experience with cursor and sonnet is that it is relatively good at first tries, but completely misses the plot during corrections. "My attempt at solving the problem contains a test that fails? No problem, let me mock the function I'm testing, so that, rather than actually run, it returns the expected value!" It keeps doing that kind of shenanigans, applying modifications that solve the newly appearing problem while screwing the original attempt's goal. I usually get much better results from regular chatgpt copying and pasting, the trouble being that it is a major pain to handle the context window manually by pasting relevant info and reminding what I think is being forgotten.

jwpapi 2y ago
Yes it’s usually worth it to try to write a really good first prompt
- earleybird 2y ago
  More than once I've found myself going down this 'little maze of twisty passages, all alike'. At some point I stop, collect up the chain of prompts in the conversation, and curate them into a net new prompt that should be a bit better. Usually I make better progress - at least for a while.
  - dr_dshiv 2y ago
    Why is it so hard to share/find prompts or distill my own damn prompts? There must be good solutions for this —
    - whall6 2y ago
      Don’t outsource the only thing left for our brains to do themselves :/
    - garfij 2y ago
      What do you find difficult about distilling your own prompts? After any back and forth session I have reasonably good results asking something like "Given this workflow, how could I have prompted this better from the start to get the same results?"

energy123 2y ago

Context length possibly. Prompt adherence drops off with context, and anything above 20k tokens is pushing it. I get the best results by presenting the smallest amount of context possible, including removing comments and main methods and functions that it doesn't need to see. It's a bit more work (not that much if you have a script that does it for you), but the results are worth it. You could test in the chatgpt app (or lmarena direct chat) where you ask the same question but with minimal hand curated context, and see if it makes the same mistake.

mvkel 2y ago
If it's a context issue, it's an issue with how cursor itself sends the context to these reasoning LLMs. Context alone shouldn't be the reason that sonnet succeeds consistently, but others (some which have even bigger context windows) fail.
- energy123 2y ago
  Yes, that's what I'm suggesting. Cursor is spamming the models with too much context, which harms reasoning models more than it harms non-reasoning models (hypothesis, but one that aligns with my experience). That's why I recommended testing reasoning models outside of Cursor with a hand curated context. The advertised context length being longer doesn't necessarily map 1:1 with the actual ability the models have to perform difficult tasks over that full context. See for example the plots of performance on ARC vs context length for o-series models.

esperent 2y ago

Claude uses Shadcn-ui extensively in the web interface, to the point where I think it's been trained to use it over other UI components. So I think you got lucky and you're asking it to write using a very specific code library that it's good at, because it happens to use it for it's main userbase on the web chat interface. I wonder if you were using a different component library, or using Svelte instead of React, would you still find Claude the best?

pollinations 2y ago
I was recently trying to write a relatively simple htmx service with Claude. I was surprised at how much worse it was when it's not React.

MaxLeiter 2y ago

We've been working on solving a lot of these issues with v0.dev (disclaimer: shadcn and I work on it). We do a lot of pre and post-processing to ensure LLMs output valid shadcn code. We're also talking to the cursor/windsurf/zed folks on how we can improve Next.js and shadcn in the editors (maybe something like llms.txt?)

mvkel 2y ago
Thanks for all the work you do! v0 is magical. I absolutely love the feature where I can add a chunky component that v0 made to my repo with npx

kristopolous 2y ago

"not" and other function words; usually work fine today but if I'm having trouble, the best thing to do is probably be inclusive, not exclusive.

eagleinparadise 2y ago

Cursor is also very user-unfriendly in providing alternative models to use in composer (agent). There's a heavy reliance on Anthrophic for cursor. Try using Gemini thinking with Cursor. It barely works. Cmd-k outputs the thinking into the code. Its unusable in chat because the formatting sucks. Is there some relationship between Cursor and Anthropic, i wonder. Plenty of other platforms seem very eager to give users model flexibility, but Cursor seems to be lacking. I could be wrong, just an observation.

ttroyr 2y ago
Originally, actually there was a relationship between Cursor & OpenAI. Something like Cursor was supported by the OpenAI startup fund. So Cursor seems to have branched out. I think they are just emphasizing the models they find most effective. I'm surprised they haven't (apparently) incorporated Claude prompt caching yet for Sonnet.

OkGoDoIt 2y ago

I also have been less impressed by o1 in cursor compared to sonnet 3.5. Usually what I will do for a very complicated change is ask o1 to architect it, specifically asking it to give me a detailed plan for how it would be implemented, but not to actually implement anything. I then change the model to Sonnet 3.5 to have it actually do the implementation. And on the side of not being able to get models to understand something specific, there’s a place in a current project where I use a special Unicode apostrophe during some string parsing because a third-party API needs it. But any code modifications by the AI to that file always replace it with a standard ascii apostrophe. I even added a comment on that line to the effect of “never replaced this apostrophe, it’s important to leave it exactly as it is!” And also put that in my cursor rules, and sometimes directly in the prompt as well, but it always replaces it even for completely unrelated changes. I’ve had to manually fix it like 10 times in the last day, it’s infuriating.

deleted 2y ago

[deleted]

foobiekr 2y ago

Have you tried any of the specialty services like Augment? I am curious if they are any better or just snake oil.

chrismsimpson 2y ago

I’ve coded in many languages over the years but reasonably new to the TS/JS/Next world. I’ve found if you give your prompts a kind long form “stream of consciousness”, where you outline snippets of code in markdown along with contextual notes and then summarise/outline at the end what you actually wish to achieve, you can get great results. Think a long form, single page “documentation” type prompts that alternate between written copy/contextual intent/description and code blocks. Annotating code blocks with file names above the blocks I’m sure helps too. Don’t waste your context window on redundant/irrelevant information or code, stating a code sample is abridged or adding commented ellipses seems to do the job.

twilightfringe 2y ago
ha! good to confirm! I tend to do this, just kind of as a double-check thing, but never sure if it actually worked or if it was a placebo, lol. Or end with "from the user's perspective: all the "B" elements should light up in excitement when you click "C""
mvkel 2y ago
Going to try this! Thanks for the tip
d357r0y3r 2y ago
By the time I've fully documented and explained what I want to be done, and then review the result, usually finding that it's worse than what I would have written myself, I end up questioning my instinct to even reach for this tool. I like it for general refactoring and day to day small tasks, but anything that's relatively domain-specific, I just can't seem to get anything that's worth using.
- noahbp 2y ago
  Like most AI tools, great for beginners, time-savers for intermediate users, and frequently a waste of time in domains where you're an expert. I've used Cursor for shipping better frontend slop, and it's great. I skip a lot of trial and error, but not all of it.

Abishek_Muthian 2y ago

Just curious, did you try a code model like Codestral instead of a MoE?

bugglebeetle 2y ago

o3 mini’s date cut-off is 2023, so it’s unfortunately not gonna be useful for anything that requires knowledge of recent framework updates, which includes probably all big frontend stuff.

hombre_fatal 2y ago

I have the same experience. Just today I was integrating a new logging system with my kubernetes cluster. I tried out the OP model to make changes to my yaml files. It would give short snippets and I’d have to keep trial and erroring its suggestions. Eventually I pasted the original prompt to Claude and it one-shot the dang thing with perfect config. Made me wonder why I even try new models.

pknerd 2y ago

OT: How many tokens are being consumed? How much are you paying for Claude APIs?

harshitaneja 2y ago

So I think I finally understood recently why we have these divergent groups with one thinking Claude 3.5 Sonnet is the best model for coding and another that follow the OpenAI SOTA at that moment. I have been a heavy user of ChatGPT, jumping on to pro without even thinking for more than a second once released. Recently though I took a pause from my usual work on statistical modelling, heuristics work and other things in certain deep domains to focus on building client APIs and frontends and decided to again give Claude a try and it is just so great to work with for this usecase. My hypothesis is its a difference of what you are doing. OpenAI O models are much better than others at mathematical modelling and such tasks and Claude for more general purpose programming.

mycall 2y ago
Have you used multi-agent chat sessions with each fielding their own specialities and seeing if that improves your use cases aka MoE?
- harshitaneja 2y ago
  I have not. Any suggestions on which one(s) to explore to get started.

digitcatphd 2y ago

The reality is I suspect one will use different models for different things. Think of it like having different modes of transportation. You might use your scooter, bike, car, jet - depending on the circumstances. A bike was invented 100 years ago? But it may be the best in the right use case. Would still be using DaVinci for some things because we haven't bothered swapping it and it works fine. For me - the value of R1/o3 is visible logic that provides an analysis that can be critiqued by Sonnet 3.5

ido 2y ago
I have an even more topical analogy! Using different languages for different tasks. When I need some one off script do automate some drudgery (take all files with certain pattern in their name, for each do some search and replace in the text inside, zip them, upload zip to URL, etc) I use python. When Im working on a multi-platform game I use c# (and unity). When I need to make something very lean that works in mobile browsers I use JS with some light-weight libraries.

adonese 2y ago

My general workflow with ai so far has been this: - I use copilot mostly for writing unit tests. It mostly works well since the unit tests follow a standard template. - I use the chat one for alternating between different approaches and (in)validating certain approaches My day job is a big monorepo, I have not investigated that yet but I believe the models context sizes fall short there and as such the above use cases only works for me.

dhc02 2y ago

I'm going to give you a video to watch. It's not mine, and I don't know much about this particular youtuber, but it really transformed how I think about writing and structuring the prompts I use, which solved problems similar to what you're describing here. https://youtu.be/y_ywOVQyafE?si=IvKjy7QUYgxGPNgD https://youtu.be/y_ywOVQyafE?si=IvKjy7QUYgxGPNgD PS (I have not bought the guy's course and have no idea whether it's any good)