6 ms·

The critical element of these systems is that it takes you less time to get to production worthy code using the tool than it would without it. You can't just st

The critical element of these systems is that it takes you less time to get to production worthy code using the tool than it would without it. You can't just stop a demo at the point where you don't have working code. Anyone with a ChatGPT plus subscription can ask it to generate react components from a mockup, but it doesn't do a great job. You have to show your tool doing a good job, not just flipping through component files.

weeksie 3y ago
Yeah, it really didn't seem like there was a lot of "there" there. Maybe I'm missing something in the codebase but they're just piping specs to the gpt-4 api?
ushakov 3y ago
What is “a good job” according to your definition? If it can save even 10% time, then the tool is worth the money. And by the way, they never promised anything > ReactAgent is an experimental autonomous agent
- mdhb 3y ago
  I don’t know how it is even going to achieve that… It’s going to give you a code base that is filled with subtle bugs, no tests and no documentation. You won’t have a good understanding of your own product and you are going to almost certainly end up wasting more time over even a modest time window compared to the feeling of productivity you had at the start.
  - giancarlostoro 3y ago
    I feel like you could build reliable scaffolding tools with zero AI. Code generation tools are underused a lot of the time.
    - mdhb 3y ago
      Correct and from there you can still build the tests and documentation to go with it and actually understand how everything works. This is literally hoping that some linear algebra process is going to magically put all of the right things together in all the right ways while maintaining all of the correct syntax and the underlying logic will make sense. Sometimes I think people forget that it’s just a glorified guessing game of what letter most likely comes next.
  - lgas 3y ago
    It will only give you that codebase if that's what you ask it for. If you ask it for tests it will give you tests. If you ask it for documentation it will give you documentation.
MuffinFlavored 3y ago
> doing a good job Doing a good job is the subjective part, no? Especially with opinionated things like programming in general/React
- givemeethekeys 3y ago
  There are: - Style guidelines - Unit tests - Functional requirements - System requirements If the tool creates commits to pass all of them, then it is objectively doing a good job.
kodapoda 3y ago
For me, a critical part of "doing a good job" is how the implementation fits into a larger system. ChatGPT is not very good at that. I tried asking it to build a basic state management component for a TypeScript/React app. It offered a class-based component. I asked to use a closure instead. It offered a closure but skipped some of my other requirements that it has previously added (like types). I asked to add in-memory caching. It added caching but removed something else. I asked to create a context provider based on this component, it created a context provider but skipped some parts from the state management implementation. Basically, it sort of works if you can hand-hold it and pay attention to every detail. But that barely saves me any time. And the code it generates is definitely not production ready and requires refactoring in order to integrate it into an existing code base.
- lgas 3y ago
  A good trick here is to use the API rather than the web interface and give it access to functions that let it edit files directly. Then you don't have to do the iterations where you ask it for something new and it forgets something old, it just updates the code in place, leaving all the old stuff and changing/adding the new stuff.
  - kodapoda 3y ago
    Interesting. Is there an IDE (ideally, VSCode) extension you could recommend for that? What do you mean by giving access? Is it possible to limit its access only to certain functions/files?
    - vishnumenon 3y ago
      Not the GP, but I've been working on an open platform [0] for integrating OpenAI APIs with other tools (VSCode, JupyterLab, bash, chrome, etc) that you might find interesting, the VSCode integration supports editing specific files / sections etc. Also worth taking a look at Github Copilot Chat[1], it's a bit limited but in certain cases it works well for editing specific parts of files. [0] https://github.com/lightrail-ai/lightrail https://github.com/lightrail-ai/lightrail [1] https://marketplace.visualstudio.com/items?itemName=GitHub.copilot-chat https://marketplace.visualstudio.com/items?itemName=GitHub.c...

weeksie 3y ago

Yeah, it really didn't seem like there was a lot of "there" there. Maybe I'm missing something in the codebase but they're just piping specs to the gpt-4 api?

ushakov 3y ago

What is “a good job” according to your definition? If it can save even 10% time, then the tool is worth the money. And by the way, they never promised anything > ReactAgent is an experimental autonomous agent

mdhb 3y ago
I don’t know how it is even going to achieve that… It’s going to give you a code base that is filled with subtle bugs, no tests and no documentation. You won’t have a good understanding of your own product and you are going to almost certainly end up wasting more time over even a modest time window compared to the feeling of productivity you had at the start.
- giancarlostoro 3y ago
  I feel like you could build reliable scaffolding tools with zero AI. Code generation tools are underused a lot of the time.
  - mdhb 3y ago
    Correct and from there you can still build the tests and documentation to go with it and actually understand how everything works. This is literally hoping that some linear algebra process is going to magically put all of the right things together in all the right ways while maintaining all of the correct syntax and the underlying logic will make sense. Sometimes I think people forget that it’s just a glorified guessing game of what letter most likely comes next.
- lgas 3y ago
  It will only give you that codebase if that's what you ask it for. If you ask it for tests it will give you tests. If you ask it for documentation it will give you documentation.

MuffinFlavored 3y ago

> doing a good job Doing a good job is the subjective part, no? Especially with opinionated things like programming in general/React

givemeethekeys 3y ago
There are: - Style guidelines - Unit tests - Functional requirements - System requirements If the tool creates commits to pass all of them, then it is objectively doing a good job.

kodapoda 3y ago

For me, a critical part of "doing a good job" is how the implementation fits into a larger system. ChatGPT is not very good at that. I tried asking it to build a basic state management component for a TypeScript/React app. It offered a class-based component. I asked to use a closure instead. It offered a closure but skipped some of my other requirements that it has previously added (like types). I asked to add in-memory caching. It added caching but removed something else. I asked to create a context provider based on this component, it created a context provider but skipped some parts from the state management implementation. Basically, it sort of works if you can hand-hold it and pay attention to every detail. But that barely saves me any time. And the code it generates is definitely not production ready and requires refactoring in order to integrate it into an existing code base.

lgas 3y ago
A good trick here is to use the API rather than the web interface and give it access to functions that let it edit files directly. Then you don't have to do the iterations where you ask it for something new and it forgets something old, it just updates the code in place, leaving all the old stuff and changing/adding the new stuff.
- kodapoda 3y ago
  Interesting. Is there an IDE (ideally, VSCode) extension you could recommend for that? What do you mean by giving access? Is it possible to limit its access only to certain functions/files?
  - vishnumenon 3y ago
    Not the GP, but I've been working on an open platform [0] for integrating OpenAI APIs with other tools (VSCode, JupyterLab, bash, chrome, etc) that you might find interesting, the VSCode integration supports editing specific files / sections etc. Also worth taking a look at Github Copilot Chat[1], it's a bit limited but in certain cases it works well for editing specific parts of files. [0] https://github.com/lightrail-ai/lightrail https://github.com/lightrail-ai/lightrail [1] https://marketplace.visualstudio.com/items?itemName=GitHub.copilot-chat https://marketplace.visualstudio.com/items?itemName=GitHub.c...