6 ms·

I started like this. Then I came around and can’t imagine going back. It’s kinda like having a really smart new grad, who works instantly, and has memorized al

I started like this. Then I came around and can’t imagine going back. It’s kinda like having a really smart new grad, who works instantly, and has memorized all the docs. Yes I have to code review and guide it. That’s an easy trade off to make for typing 1000 tokens/s, never losing focus, and double checking every detail in realtime. First: it really does save a ton of time for tedious tasks. My best example is test cases. I can write a method in 3 minutes, but Sonnet will write the 8 best test cases in 4 seconds, which would have taken me 10 mins of switching back and forth, looking at branches/errors, and mocking. I can code review and run these in 30s. Often it finds a bug. It’s definitely more patient than me in writing detailed tests. Instant and pretty great code review: it can understand what you are trying to do, find issues, and fix them quickly. Just ask it to review and fix issues. Writing new code: it’s actually pretty great at this. I needed a util class for config that had fallbacks to config files, env vars and defaults. And I wanted type checking to work on the accessors. Nothing hard, but it would have taken time to look at docs for yaml parsing, how to find the home directory, which env vars api returns null vs error on blank, typing, etc. All easy, but takes time. Instead I described it in about 20 seconds and it wrote it (with tests) in a few seconds. It’s moved well past the stage “it can answer questions on stack overflow”. If it has been a while (a while=6 months in ML), try again with new sonnet 3.5.

scosman 2y ago
Another fun example from yesterday: pasted a blog post in markdown into a HTML comment. Selected it and told sonnet to convert it to HTML using another blog post as a style reference. Done in 5 seconds.
- kccqzy 2y ago
  And how do you trust that it didn't just alter or omit some sentences from your blog post? I just use Pandoc for that purpose and it takes 30 seconds, including the time to install pandoc. For code generation where you'll review everything, AI makes sense; but for such conversion tasks, it doesn't because you won't review the generated HTML.
  - falconertc 2y ago
    You didn't also factor in the time to learn Pandoc (and to relearn it if you haven't used it lately). This is also just one of many daily use cases for these tools. The time it takes to know how to use a dozen tools like this adds up when an LLM can just do them all.
    - kccqzy 2y ago
      This is actually how I would use AI: if I forgot how to do a conversion task, I would ask AI to tell me the command so that I can run it without rejiggering my memory first. The pandoc command is literally one line with a few flags; it's easily reviewable. Then I run pandoc myself. Same thing with the multitude of other rarely used but extremely useful tools such as jq. In other words, I want AI to help me with invoking other tools to do a job rather than doing the job itself. This nicely sidesteps all the trust issues I have.
      - flir 2y ago
        I do that constantly. jq's syntax is especially opaque to me. "I've got some JSON formatted like <this>. Give me a jq command that does <that>. Google, but better.
        scosman 2y ago
        This
  - TeMPOraL 2y ago
    > it takes 30 seconds, including the time to install pandoc On some speedrunning competition maybe? Just tested on my work machine, `sudo apt-get pandoc` took 11 seconds to complete, and it was this fast only because I already had all the depndencies installed. Also I don't think you'll be able to fulfill the "using another blog post as a style reference" part of GP's requirements - unless, again, you're some grand-master Pandoc speedrunner. Sure, AI will make mistakes with such conversion tasks. It's not worth it if you're going to review everything carefully anyway. In code, fortunately, you don't have to - the compiler is doing 90% of the grunt work for you. In writing, depends on context. Some text you can eyeball quickly. Sometimes you can get help from your tool. Literally yesterday I back-ported a CV from English to Polish via Word's Translation feature. I could've done it by hand, but Word did 90% of it correctly, and fixing the remaining issues was a breeze. Ultimately, what makes LLMs a good tool for random conversions like these is that it's just one tool. Sure, Pandoc can do GP's case better (if inputs are well-defined), but it can't do any of the 10 other ad-hoc conversions they may have needed that day.
tyre 2y ago
I’ve found it better at writing tests because it tests the code you’ve written vs what you intended. I’ve caught logic bugs because it wrote tests with an assertion for a conditional that was backwards. The readable name of the test clearly pointed out that I was doing the wrong thing (the test passed?.)
- scosman 2y ago
  Interesting. I’ve had the opposite experience (I invert or miss a condition, it catches it). It probably comes down to model, naming and context. Until Sonnet 3.5 my experience was similar to yours. After it mostly “just works”.
- williamdclt 2y ago
  That sounds more like a footgun than a desirable thing to be honest!
DeathArrow 2y ago
>My best example is test cases. I can write a method in 3 minutes, but Sonnet will write the 8 best test cases in 4 seconds For me it doesn't work. Generated tests fail to run or they fail. I work in large C# codebases and in each file I have lots of injected dependencies. I have one public method which can call lots of private methods in the same class. AI either doesn't properly mock the dependencies, either ignores what happens in the private methods. If I take a lot of time guiding it where to look, it can generate unit tests that pass. But it takes longer than if I write the unit tests myself.
- deleted 2y ago
  [deleted]
- dep_b 2y ago
  For me it's the same. It's usually just some hallucinated garbage. All of these LLM's don't have the full picture of my project. When I can give them isolated tasks like convert X to Y, create a foo that does bar it's excellent, but for unit testing? Not even going to try anymore. I write 5 unit tests manually that work in the time I write 5 prompts that give me useless stuff that I need to add manually. Why can't we have a LLM cache for a project just like I have a build cache? Analyze one particular commit on the main branch very expensively, then only calculate the differences from that point. Pretty much like git works, just for your model.
  - scosman 2y ago
    With Cursor you can specify which files it reads before starting. Usually have to attached one or two to get an ideal one-shot result. But yeah, I use it for unit testing, not integration testing.
  - ssijak 2y ago
    "It's usually just some hallucinated garbage. All of these LLM's don't have the full picture of my project." Cursor can have whole project in the context, or you can specify specific files that you want.
sebastiansm 2y ago
What is the best workflow to code with an AI? Copy and paste the code to the Claude website? Or use an extension? o something else?
- paulluuk 2y ago
  I personally use copilot, which is integrated into my IDE, almost identical to this Cursor example.
  - ukuina 2y ago
    Does Copilot do multi-file edits now?
    - ukuina 2y ago
      Copilot Editor is a beta feature that can perform multi-file edits.
  - dham 2y ago
    Copilot is about as far away from Cursor with Claude as the Wright Brothers' glider is to the Saturn V.
    - paulluuk 2y ago
      Not based on the link, I didn't see anything in that text that I can't do with copilot or which looked better to me than what copilot outputs.
- scosman 2y ago
  Cursor. Mostly chat mode. Usually adding 1-2 extra files to the context before invoking, and selecting the relevant section for extra focus.
scosman 2y ago
Maybe a TLDR from all the issues I'm reading in this thread: - It's gotten way better in the last 6 months. Both models (Sonnet 3.5 and new October Sonnet 3.5), and tooling (Cursor). If you last tried Co-pilot, you should probably give it another look. It's also going to keep getting better. [1] - It can make errors, and expect to do some code review and guiding. However the error rates are going way way down [1]. I'd say it's already below humans for a lot of tasks. I'm often doing 2/3 iterations before applying a diff, but a quick comment like "close, keep the test cases, but use the test fixture at the top of the file to reduce repeated code" and 5 seconds is all it takes to get a full refactor. Compared to code-review turn around with a team, it's magic. - You need to learn how to use it. Setting the right prompts, adding files to the context, etc. I'd say it's already worth learning. - I just knows the docs, and that's pretty invaluable. I know 10ish languages, which also means I don't remember the system call to get an env var in any of them. It does, and can insert it a lot faster than I can google it. Again, you'll need to code review, but more and more it's nailing idiomatic error checking in each language. - You don't need libraries for boilerplate tasks. zero_pad is the extreme/joke example, but a lot more of my code is just using system libraries. - It can do things other tools can't. Tell it to take the visual style of one blog post and port to another. Take it to use a test file I wrote for style reference, and update 12 other files to follow that style. Read the README and tests, then write pydocs for a library. Write a GitHub action to build docs and deploy to GitHub pages (including suggesting libraries, deploy actions, and offering alternatives). Again: you don't blindly trust anything, you code review, and tests are critical. [1] https://www.anthropic.com/news/3-5-models-and-computer-use https://www.anthropic.com/news/3-5-models-and-computer-use
- DeathArrow 2y ago
  Yes, it works for new code and simple cases. If you have large code bases, it doesn't have the context and you have to baby it, telling which files and functions it should look into before attempting to write something. That takes a lot of time. Yes, it can do simple tasks, like you said, writing a call to get the environment variables. But imagine you work on a basket calculation service, where you have base item prices, where you have to apply some discounts based on some complicated rules, you have to add various kinds of taxes for various countries in the world and you have to use a different number of decimals for each country. Each of your classes calls 5 to 6 other classes, all with a lot of business logic behind. Besides that, you also make lots of API calls to other services. What will the AI do for you? Nothing, it will just help you write one liners to parse or split strings. For everything else it lacks context.
throwup238 2y ago
> Instant and pretty great code review: it can understand what you are trying to do, find issues, and fix them quickly. Just ask it to review and fix issues. Cursor’s code review is surprisingly good. It’s caught many bugs for me that would have taken a while to debug, like off by one errors or improperly refactored code (like changing is_alive to is_dead and forgetting to negate conditionals)
- kurko 2y ago
  > changing is_alive to is_dead and forgetting to negate conditionals No test broke?
  - flappyeagle 2y ago
    Tests don’t care what you name the variable
globnomulous 2y ago
> it can understand It can't understand. That's not what LLMs do.
- golol 2y ago
  This is a prompt I gave to o1-mini a while ago: My instructions follow now. The scripts which I provided you work perfectly fine. I want you to perform a change though. The image_data.pkl and faiss_index.bin are two databases consisting of rows, one for each image, in the end, right? My problem is that there are many duplicates: images with different names but the same content. I want you to write a script which for each row, i.e. each image, opens the image in python and computes the average expected color and the average variation of color, for each of the colors red, green and blue, and over "random" over all the pixels. Make sure that this procedure is normalized with respect to the resolution. Then once this list of "defining features" is obtained, we can compute the pairwise difference. If two images have less than 1% variation in both expectation and variation, then we consider them to be identical. in this case, delete those rows/images, except for one of course, from the .pkl and the .bin I mentioned in the beginning. Write a log file at the end which lists the filenames of identical images. It wrote the script, I ran it and it worked. I had it write another script which displays the found duplicate groups so I could see at a glance that the script had indeed worked. And for you this does not constitute any understanding? Yes it is assembling pieces of code or algorithmic procedures which it has memorized. But in this way it creates a script tailored to my wishes. The key is that it has to understand my intent.
  - marco_craveiro 2y ago
    I think the problem is our traditional notions of "understanding" and "intelligence" fail us. I don't think we understand what we mean by "understanding". Whatever the LLM is doing inside, it's far removed from what a human would do. But on the face of it, from an external perspective, it has many of the same useful properties as if done by a human. And the LLM's outputs seem to be converging closer and closer to what a human would do, even though there is still a large gap. I suggest the focus here shouldn't be so much on what the LLM can't do but the speed at which it is becoming better at doing things.
froobrad 2y ago
This “really smart new grad” take is completely insane to me, especially if you know how LLMs work. Look at this SQL snippet Claude (the new Sonnet) generated recently. -- Get recipient's push token and sender's username SELECT expo_push_token, p.username INTO recipient_push_token, sender_username FROM profiles p WHERE p.id = NEW.recipient_id; Seems like the world has truly gone insane and engineers are tuned into some alternate reality a la Fox News. Well…it’ll be a sobering day when the other shoe falls.

scosman 2y ago

Another fun example from yesterday: pasted a blog post in markdown into a HTML comment. Selected it and told sonnet to convert it to HTML using another blog post as a style reference. Done in 5 seconds.

kccqzy 2y ago
And how do you trust that it didn't just alter or omit some sentences from your blog post? I just use Pandoc for that purpose and it takes 30 seconds, including the time to install pandoc. For code generation where you'll review everything, AI makes sense; but for such conversion tasks, it doesn't because you won't review the generated HTML.
- falconertc 2y ago
  You didn't also factor in the time to learn Pandoc (and to relearn it if you haven't used it lately). This is also just one of many daily use cases for these tools. The time it takes to know how to use a dozen tools like this adds up when an LLM can just do them all.
  - kccqzy 2y ago
    This is actually how I would use AI: if I forgot how to do a conversion task, I would ask AI to tell me the command so that I can run it without rejiggering my memory first. The pandoc command is literally one line with a few flags; it's easily reviewable. Then I run pandoc myself. Same thing with the multitude of other rarely used but extremely useful tools such as jq. In other words, I want AI to help me with invoking other tools to do a job rather than doing the job itself. This nicely sidesteps all the trust issues I have.
    - flir 2y ago
      I do that constantly. jq's syntax is especially opaque to me. "I've got some JSON formatted like <this>. Give me a jq command that does <that>. Google, but better.
      - scosman 2y ago
        This
- TeMPOraL 2y ago
  > it takes 30 seconds, including the time to install pandoc On some speedrunning competition maybe? Just tested on my work machine, `sudo apt-get pandoc` took 11 seconds to complete, and it was this fast only because I already had all the depndencies installed. Also I don't think you'll be able to fulfill the "using another blog post as a style reference" part of GP's requirements - unless, again, you're some grand-master Pandoc speedrunner. Sure, AI will make mistakes with such conversion tasks. It's not worth it if you're going to review everything carefully anyway. In code, fortunately, you don't have to - the compiler is doing 90% of the grunt work for you. In writing, depends on context. Some text you can eyeball quickly. Sometimes you can get help from your tool. Literally yesterday I back-ported a CV from English to Polish via Word's Translation feature. I could've done it by hand, but Word did 90% of it correctly, and fixing the remaining issues was a breeze. Ultimately, what makes LLMs a good tool for random conversions like these is that it's just one tool. Sure, Pandoc can do GP's case better (if inputs are well-defined), but it can't do any of the 10 other ad-hoc conversions they may have needed that day.

tyre 2y ago

I’ve found it better at writing tests because it tests the code you’ve written vs what you intended. I’ve caught logic bugs because it wrote tests with an assertion for a conditional that was backwards. The readable name of the test clearly pointed out that I was doing the wrong thing (the test passed?.)

scosman 2y ago
Interesting. I’ve had the opposite experience (I invert or miss a condition, it catches it). It probably comes down to model, naming and context. Until Sonnet 3.5 my experience was similar to yours. After it mostly “just works”.
williamdclt 2y ago
That sounds more like a footgun than a desirable thing to be honest!

DeathArrow 2y ago

>My best example is test cases. I can write a method in 3 minutes, but Sonnet will write the 8 best test cases in 4 seconds For me it doesn't work. Generated tests fail to run or they fail. I work in large C# codebases and in each file I have lots of injected dependencies. I have one public method which can call lots of private methods in the same class. AI either doesn't properly mock the dependencies, either ignores what happens in the private methods. If I take a lot of time guiding it where to look, it can generate unit tests that pass. But it takes longer than if I write the unit tests myself.

deleted 2y ago
[deleted]
dep_b 2y ago
For me it's the same. It's usually just some hallucinated garbage. All of these LLM's don't have the full picture of my project. When I can give them isolated tasks like convert X to Y, create a foo that does bar it's excellent, but for unit testing? Not even going to try anymore. I write 5 unit tests manually that work in the time I write 5 prompts that give me useless stuff that I need to add manually. Why can't we have a LLM cache for a project just like I have a build cache? Analyze one particular commit on the main branch very expensively, then only calculate the differences from that point. Pretty much like git works, just for your model.
- scosman 2y ago
  With Cursor you can specify which files it reads before starting. Usually have to attached one or two to get an ideal one-shot result. But yeah, I use it for unit testing, not integration testing.
- ssijak 2y ago
  "It's usually just some hallucinated garbage. All of these LLM's don't have the full picture of my project." Cursor can have whole project in the context, or you can specify specific files that you want.

sebastiansm 2y ago

What is the best workflow to code with an AI? Copy and paste the code to the Claude website? Or use an extension? o something else?

paulluuk 2y ago
I personally use copilot, which is integrated into my IDE, almost identical to this Cursor example.
- ukuina 2y ago
  Does Copilot do multi-file edits now?
  - ukuina 2y ago
    Copilot Editor is a beta feature that can perform multi-file edits.
- dham 2y ago
  Copilot is about as far away from Cursor with Claude as the Wright Brothers' glider is to the Saturn V.
  - paulluuk 2y ago
    Not based on the link, I didn't see anything in that text that I can't do with copilot or which looked better to me than what copilot outputs.
scosman 2y ago
Cursor. Mostly chat mode. Usually adding 1-2 extra files to the context before invoking, and selecting the relevant section for extra focus.

scosman 2y ago

Maybe a TLDR from all the issues I'm reading in this thread: - It's gotten way better in the last 6 months. Both models (Sonnet 3.5 and new October Sonnet 3.5), and tooling (Cursor). If you last tried Co-pilot, you should probably give it another look. It's also going to keep getting better. [1] - It can make errors, and expect to do some code review and guiding. However the error rates are going way way down [1]. I'd say it's already below humans for a lot of tasks. I'm often doing 2/3 iterations before applying a diff, but a quick comment like "close, keep the test cases, but use the test fixture at the top of the file to reduce repeated code" and 5 seconds is all it takes to get a full refactor. Compared to code-review turn around with a team, it's magic. - You need to learn how to use it. Setting the right prompts, adding files to the context, etc. I'd say it's already worth learning. - I just knows the docs, and that's pretty invaluable. I know 10ish languages, which also means I don't remember the system call to get an env var in any of them. It does, and can insert it a lot faster than I can google it. Again, you'll need to code review, but more and more it's nailing idiomatic error checking in each language. - You don't need libraries for boilerplate tasks. zero_pad is the extreme/joke example, but a lot more of my code is just using system libraries. - It can do things other tools can't. Tell it to take the visual style of one blog post and port to another. Take it to use a test file I wrote for style reference, and update 12 other files to follow that style. Read the README and tests, then write pydocs for a library. Write a GitHub action to build docs and deploy to GitHub pages (including suggesting libraries, deploy actions, and offering alternatives). Again: you don't blindly trust anything, you code review, and tests are critical. [1] https://www.anthropic.com/news/3-5-models-and-computer-use https://www.anthropic.com/news/3-5-models-and-computer-use

DeathArrow 2y ago
Yes, it works for new code and simple cases. If you have large code bases, it doesn't have the context and you have to baby it, telling which files and functions it should look into before attempting to write something. That takes a lot of time. Yes, it can do simple tasks, like you said, writing a call to get the environment variables. But imagine you work on a basket calculation service, where you have base item prices, where you have to apply some discounts based on some complicated rules, you have to add various kinds of taxes for various countries in the world and you have to use a different number of decimals for each country. Each of your classes calls 5 to 6 other classes, all with a lot of business logic behind. Besides that, you also make lots of API calls to other services. What will the AI do for you? Nothing, it will just help you write one liners to parse or split strings. For everything else it lacks context.

throwup238 2y ago

> Instant and pretty great code review: it can understand what you are trying to do, find issues, and fix them quickly. Just ask it to review and fix issues. Cursor’s code review is surprisingly good. It’s caught many bugs for me that would have taken a while to debug, like off by one errors or improperly refactored code (like changing is_alive to is_dead and forgetting to negate conditionals)

kurko 2y ago
> changing is_alive to is_dead and forgetting to negate conditionals No test broke?
- flappyeagle 2y ago
  Tests don’t care what you name the variable

globnomulous 2y ago

> it can understand It can't understand. That's not what LLMs do.

golol 2y ago
This is a prompt I gave to o1-mini a while ago: My instructions follow now. The scripts which I provided you work perfectly fine. I want you to perform a change though. The image_data.pkl and faiss_index.bin are two databases consisting of rows, one for each image, in the end, right? My problem is that there are many duplicates: images with different names but the same content. I want you to write a script which for each row, i.e. each image, opens the image in python and computes the average expected color and the average variation of color, for each of the colors red, green and blue, and over "random" over all the pixels. Make sure that this procedure is normalized with respect to the resolution. Then once this list of "defining features" is obtained, we can compute the pairwise difference. If two images have less than 1% variation in both expectation and variation, then we consider them to be identical. in this case, delete those rows/images, except for one of course, from the .pkl and the .bin I mentioned in the beginning. Write a log file at the end which lists the filenames of identical images. It wrote the script, I ran it and it worked. I had it write another script which displays the found duplicate groups so I could see at a glance that the script had indeed worked. And for you this does not constitute any understanding? Yes it is assembling pieces of code or algorithmic procedures which it has memorized. But in this way it creates a script tailored to my wishes. The key is that it has to understand my intent.
- marco_craveiro 2y ago
  I think the problem is our traditional notions of "understanding" and "intelligence" fail us. I don't think we understand what we mean by "understanding". Whatever the LLM is doing inside, it's far removed from what a human would do. But on the face of it, from an external perspective, it has many of the same useful properties as if done by a human. And the LLM's outputs seem to be converging closer and closer to what a human would do, even though there is still a large gap. I suggest the focus here shouldn't be so much on what the LLM can't do but the speed at which it is becoming better at doing things.

froobrad 2y ago

This “really smart new grad” take is completely insane to me, especially if you know how LLMs work. Look at this SQL snippet Claude (the new Sonnet) generated recently. -- Get recipient's push token and sender's username SELECT expo_push_token, p.username INTO recipient_push_token, sender_username FROM profiles p WHERE p.id = NEW.recipient_id; Seems like the world has truly gone insane and engineers are tuned into some alternate reality a la Fox News. Well…it’ll be a sobering day when the other shoe falls.