4 ms·

The code produced in the "getInitials" function handles absolutely no corner cases whatsoever. It also didn't add any tests to the PR. All this does is making

The code produced in the "getInitials" function handles absolutely no corner cases whatsoever. It also didn't add any tests to the PR. All this does is making sure your website will crap all over itself 2 weeks into using this tool (death by a thousand cuts style) and you'll need to hire more people to fix whatever this thing fucks up. Just about the opposite of what automation is supposed to help with. Good luck!

williamzeng0 3y ago
That's completely right, the testimonials will look really strange if the names have 3+ words in them. That's why we're targeting really strong developers to review Sweep's PRs. An experienced dev(like you) will be able to read the code, think "hey this needs tests and edge cases" and then request changes instead of merging it.
- marktani 3y ago
  Thanks for staying constructive and on topic. Super interesting tool and amazing video! Is Sweep also taking in suggestions and then incorporates them with follow-up commits to the PR?
  - williamzeng0 3y ago
    Yes Sweep does! It's through file comments and PR comments. We also handle failing GitHub actions.
eldavido 3y ago
I think we're going to see a lot of this. I worked in self-driving and the stuff always 95% worked. Never 100. This is useful in some ways. Thinking about situations like pre-release software testing, there are exploratory test cases that are simply too numerous to ever have a human perform economically. A lot of AI is going to do this kind of very low-valued grunt work where it doesn't matter if it's 90% or 99% correct, it's the fact that it can get done at all. A lot of this work is "additive" in the sense that, it's just too expensive to do today (with a human). The work product of these systems is best seen as a "rough draft" or "suggestion". It's a first cut, not the last word. On the other hand, a lot (most?) of the meat-and-potatoes coding done today, is situations where things have to WORK. Stuff where correctness absolutely matters--billing/money/settlement (calculating tax, handling returns, moving money between accounts), a lot of OS code for things like memory management / locking / resource management, drug dosing, reservation management, etc. Granted, this stuff is a lot more complex and nuanced than the code of an average CRUD app, but then, I also don't spend my days implementing bcrypt, quicksort, or self-synchronizing Unicode parsing. We have libraries for that. The question is whether we're better off relying on agents to write a bunch of grunty code, or come up with better top-level organization / code structures, that doing it "by hand" is the better approach. I'm actually optimistic that we can do better code-wise. But I'd love to see how things develop. Maybe we wouldn't need AI if we just had better programming languages.
- kevinlu1248 3y ago
  I think for teams that want to move fast in a non-critical environment (health, finance etc.) something that works 90% of the time is fine. Getting to 95% takes twice the amount of time but does not provide twice the value. When the 5% difference becomes the difference maker we can fix it later. Further, we're adding better test systems to Sweep. For now, you can just comment to get Sweep to cover the edge cases and write tests. Happy to take any other feedback.
  - andrejcasey 3y ago
    Sorry but unless the core business is making statistical predictions, then you're wrong. Other industries (like health and finance) still need robust applications with like 99.99% uptime.
twelve40 3y ago
Ok i get the skepticism but what i liked about their description is that it's not the overblown hysterical "AI superhuman programmer" pitch, but a more modest "junior" angle. If they keep looking for something that clicks, there are lots of "junior" niches that could be filled - for example, I can see that thing automatically working to beef up the test coverage. It's kind of difficult to screw up tests, the potential fallout is low and there is a unambiguous number (coverage) as the success criteria. If we look around our daily developer lives, there might be more cases that could be automated with this, even if it doesn't ever become good enough for any general programming.
- williamzeng0 3y ago
  Something I'm a big fan of is making a small successful ticket (for example migrate the functions in one file) and then applying it map-reduce style across the entire codebase. This could help a lot, and by definition addresses repetitive work. We have this (to some degree) with issue templates, where you can pre-populate some text and fill in the rest. We're also thinking about good ways to offload that work to Sweep.
gcanyon 3y ago
Hence why: 1. They refer to as an "AI junior developer" 2. It creates pull requests, not commits From (2), your problem is with the person who commits this code without modification, not with the AI.
- williamzeng0 3y ago
  That would be me, completely happy to take the blame here :) We manually update the content here so the code works just fine for now.
robertlagrant 3y ago
Sounds like a junior developer to me :)
- williamzeng0 3y ago
  Haha, we need good senior devs to review Sweep’s PRs :)

williamzeng0 3y ago

That's completely right, the testimonials will look really strange if the names have 3+ words in them. That's why we're targeting really strong developers to review Sweep's PRs. An experienced dev(like you) will be able to read the code, think "hey this needs tests and edge cases" and then request changes instead of merging it.

marktani 3y ago
Thanks for staying constructive and on topic. Super interesting tool and amazing video! Is Sweep also taking in suggestions and then incorporates them with follow-up commits to the PR?
- williamzeng0 3y ago
  Yes Sweep does! It's through file comments and PR comments. We also handle failing GitHub actions.

eldavido 3y ago

I think we're going to see a lot of this. I worked in self-driving and the stuff always 95% worked. Never 100. This is useful in some ways. Thinking about situations like pre-release software testing, there are exploratory test cases that are simply too numerous to ever have a human perform economically. A lot of AI is going to do this kind of very low-valued grunt work where it doesn't matter if it's 90% or 99% correct, it's the fact that it can get done at all. A lot of this work is "additive" in the sense that, it's just too expensive to do today (with a human). The work product of these systems is best seen as a "rough draft" or "suggestion". It's a first cut, not the last word. On the other hand, a lot (most?) of the meat-and-potatoes coding done today, is situations where things have to WORK. Stuff where correctness absolutely matters--billing/money/settlement (calculating tax, handling returns, moving money between accounts), a lot of OS code for things like memory management / locking / resource management, drug dosing, reservation management, etc. Granted, this stuff is a lot more complex and nuanced than the code of an average CRUD app, but then, I also don't spend my days implementing bcrypt, quicksort, or self-synchronizing Unicode parsing. We have libraries for that. The question is whether we're better off relying on agents to write a bunch of grunty code, or come up with better top-level organization / code structures, that doing it "by hand" is the better approach. I'm actually optimistic that we can do better code-wise. But I'd love to see how things develop. Maybe we wouldn't need AI if we just had better programming languages.

kevinlu1248 3y ago
I think for teams that want to move fast in a non-critical environment (health, finance etc.) something that works 90% of the time is fine. Getting to 95% takes twice the amount of time but does not provide twice the value. When the 5% difference becomes the difference maker we can fix it later. Further, we're adding better test systems to Sweep. For now, you can just comment to get Sweep to cover the edge cases and write tests. Happy to take any other feedback.
- andrejcasey 3y ago
  Sorry but unless the core business is making statistical predictions, then you're wrong. Other industries (like health and finance) still need robust applications with like 99.99% uptime.

twelve40 3y ago

Ok i get the skepticism but what i liked about their description is that it's not the overblown hysterical "AI superhuman programmer" pitch, but a more modest "junior" angle. If they keep looking for something that clicks, there are lots of "junior" niches that could be filled - for example, I can see that thing automatically working to beef up the test coverage. It's kind of difficult to screw up tests, the potential fallout is low and there is a unambiguous number (coverage) as the success criteria. If we look around our daily developer lives, there might be more cases that could be automated with this, even if it doesn't ever become good enough for any general programming.

williamzeng0 3y ago
Something I'm a big fan of is making a small successful ticket (for example migrate the functions in one file) and then applying it map-reduce style across the entire codebase. This could help a lot, and by definition addresses repetitive work. We have this (to some degree) with issue templates, where you can pre-populate some text and fill in the rest. We're also thinking about good ways to offload that work to Sweep.

gcanyon 3y ago

Hence why: 1. They refer to as an "AI junior developer" 2. It creates pull requests, not commits From (2), your problem is with the person who commits this code without modification, not with the AI.

williamzeng0 3y ago
That would be me, completely happy to take the blame here :) We manually update the content here so the code works just fine for now.

robertlagrant 3y ago

Sounds like a junior developer to me :)

williamzeng0 3y ago
Haha, we need good senior devs to review Sweep’s PRs :)