5 ms·

But how do you know those were the only errors?

HaZeust 2y ago
What's this question even mean? Because they're the only ones that came up in the debugger portion of the IDE, the output serves the intended purposes, the logging and error handling that I wanted to include were in the initial write-up prompt, and I could read the code it wrote because I partially knew the outputted language - and when I wasn't sure of a line, I asked it for clarification and a source from a reputable knowledgebase of the language, and GPT provided it?
- majormajor 2y ago
  I would've expected an answer involving "an exhaustive suite of test cases still passed" - "it looks right" is a low bar for any complex software project these days. It's the long, long, long tail of edge cases - not just porting them, but even identifying them to test - that slow or doom most real-world human rewrites, after all.
  - josephg 2y ago
    True - but you can ask the chatbot to write a test suite too.
    - what 2y ago
      This doesn’t really make sense? If I can’t trust the code it writes, why should I trust that it can write a comprehensive test suite?
      - josephg 2y ago
        Because the bugs in its code and the bugs in its test suites usually don't line up and cancel each other out.
      - simonw 2y ago
        Because you can read the test suite to check what it's testing, then break the implementation and run the tests and check they fail, then break a test and run them and check that fails too. You have to review the code these thing write for you, just like code from any other collaborator.
- VirusNewbie 2y ago
  >nd I could read the code it wrote because I partially knew the outputted language oh ok. this is quite different than what I was picturing. So far this is my favorite use case of LLMs, they seem very good at this. I mistakenly thought you were using it almost as a black box compiler. "look it ported it to Rust, I can't make sense of it, but it seems to work and no segfaults!". What you say sounds pretty sensible, and it is a very nice practical example of the power of LLMs.
- albedoa 2y ago
  That you don't know what the question means should have all of us reevaluating our confidence in every one of your claims in this thread.
  - HaZeust 2y ago
    Only sharing my experiences and observation in the upcoming trajectory of these tools; you're free to have your own. I will tell you this, the second most-used language in my day-to-day (TypeScript) is one that I've seldom sat down and learned, rely on AI for me to create and streamline, and has not given me any issues for 16 months running (since the project has started). AI won't replace jobs; but someone who knows how to use it better will.
    - tharant 2y ago
      > AI won't replace jobs; but someone who knows how to use it better will. So where do I go to learn how to use ‘em better? Or, at least, examples of what works so I can understand what I’m doing wrong?
      - HaZeust 2y ago
        Learn how to think ontologically and break down your requests first by what you're TRULY looking for, and then understand what parts would need to be defined in order to build that system -- that "whole". Here's some guides: 1.) https://platform.openai.com/docs/guides/prompt-engineering https://platform.openai.com/docs/guides/prompt-engineering 2.) https://www.promptingguide.ai/ https://www.promptingguide.ai/
        tharant 2y ago
        I’ve received this /exact/ same unhelpful response multiple times in other threads (from different users even; am I talking to deterministic bots here) so I’ll do the same by offering the response I gave others: “Since I’m dealing with models rather than other engineers should I expect the process of breaking down the problem to be dramatically different from that of writing design documents or API specs? I rarely have difficulty prompting (or creating useful system prompts for) models when chatting or doing RAG work with plain English docs but once I try to get coherent code from a model things fall apart pretty quickly.” Said another way, I long ago learned as an engineer how to do the things you’re suggesting (they are skills I’ve used and evolved over more than twenty years as a professional software engineer) but, in my experience, those same skills do not seem to apply when trying to do non-trivial code-generation tasks for Java/Scala/Python projects with an LLM. I’ve tried prompting ’em with my design documentation and API specs. I’ve tried prompting ’em with a pared-down version of my docs/specs in order to be more succinct. I’ve tried expanding my docs/specs to be more concrete and detailed. I’ve tried very short prompts. I’ve tried very detailed and lengthy prompts. I’ve tried tweaking system prompts. I’ve tried starting with prompts that limit the scope of the project then expanding from there. I’ve tried uploading the docs/specs so that the models can reference them later. I’ve tried giving ‘em access to entire repositories. I’ve tried so many things all to no avail. The best solution I’ve thus far found in these threads is to just try to fit the entirety of a project within the limits of the context window and/or to just keep my whole project in a few short files; that may be sufficient for small projects but it’s not possible nor even reasonable given the size and complexity of projects with which I work. As I’ve said elsewhere, I dearly /want/ these things to work in my environment and for my use-cases as capably as they do in/for yours—this stuff is really interesting and I enjoy learning how to do new things—but after reading all the comments in this thread and others I don’t think the needs of my environment are supported by these models. Maybe they will be someday. I’ll keep playing with things but as of right now I see a significant impedance mismatch between the confidence others have in the models’ ability to do complex coding tasks compared to the kinds of tasks I’ve seen demonstrated here and elsewhere.

What's this question even mean? Because they're the only ones that came up in the debugger portion of the IDE, the output serves the intended purposes, the logging and error handling that I wanted to include were in the initial write-up prompt, and I could read the code it wrote because I partially knew the outputted language - and when I wasn't sure of a line, I asked it for clarification and a source from a reputable knowledgebase of the language, and GPT provided it?

majormajor 2y ago
I would've expected an answer involving "an exhaustive suite of test cases still passed" - "it looks right" is a low bar for any complex software project these days. It's the long, long, long tail of edge cases - not just porting them, but even identifying them to test - that slow or doom most real-world human rewrites, after all.
- josephg 2y ago
  True - but you can ask the chatbot to write a test suite too.
  - what 2y ago
    This doesn’t really make sense? If I can’t trust the code it writes, why should I trust that it can write a comprehensive test suite?
    - josephg 2y ago
      Because the bugs in its code and the bugs in its test suites usually don't line up and cancel each other out.
    - simonw 2y ago
      Because you can read the test suite to check what it's testing, then break the implementation and run the tests and check they fail, then break a test and run them and check that fails too. You have to review the code these thing write for you, just like code from any other collaborator.
VirusNewbie 2y ago
>nd I could read the code it wrote because I partially knew the outputted language oh ok. this is quite different than what I was picturing. So far this is my favorite use case of LLMs, they seem very good at this. I mistakenly thought you were using it almost as a black box compiler. "look it ported it to Rust, I can't make sense of it, but it seems to work and no segfaults!". What you say sounds pretty sensible, and it is a very nice practical example of the power of LLMs.
albedoa 2y ago
That you don't know what the question means should have all of us reevaluating our confidence in every one of your claims in this thread.
- HaZeust 2y ago
  Only sharing my experiences and observation in the upcoming trajectory of these tools; you're free to have your own. I will tell you this, the second most-used language in my day-to-day (TypeScript) is one that I've seldom sat down and learned, rely on AI for me to create and streamline, and has not given me any issues for 16 months running (since the project has started). AI won't replace jobs; but someone who knows how to use it better will.
  - tharant 2y ago
    > AI won't replace jobs; but someone who knows how to use it better will. So where do I go to learn how to use ‘em better? Or, at least, examples of what works so I can understand what I’m doing wrong?
    - HaZeust 2y ago
      Learn how to think ontologically and break down your requests first by what you're TRULY looking for, and then understand what parts would need to be defined in order to build that system -- that "whole". Here's some guides: 1.) https://platform.openai.com/docs/guides/prompt-engineering https://platform.openai.com/docs/guides/prompt-engineering 2.) https://www.promptingguide.ai/ https://www.promptingguide.ai/
      - tharant 2y ago
        I’ve received this /exact/ same unhelpful response multiple times in other threads (from different users even; am I talking to deterministic bots here) so I’ll do the same by offering the response I gave others: “Since I’m dealing with models rather than other engineers should I expect the process of breaking down the problem to be dramatically different from that of writing design documents or API specs? I rarely have difficulty prompting (or creating useful system prompts for) models when chatting or doing RAG work with plain English docs but once I try to get coherent code from a model things fall apart pretty quickly.” Said another way, I long ago learned as an engineer how to do the things you’re suggesting (they are skills I’ve used and evolved over more than twenty years as a professional software engineer) but, in my experience, those same skills do not seem to apply when trying to do non-trivial code-generation tasks for Java/Scala/Python projects with an LLM. I’ve tried prompting ’em with my design documentation and API specs. I’ve tried prompting ’em with a pared-down version of my docs/specs in order to be more succinct. I’ve tried expanding my docs/specs to be more concrete and detailed. I’ve tried very short prompts. I’ve tried very detailed and lengthy prompts. I’ve tried tweaking system prompts. I’ve tried starting with prompts that limit the scope of the project then expanding from there. I’ve tried uploading the docs/specs so that the models can reference them later. I’ve tried giving ‘em access to entire repositories. I’ve tried so many things all to no avail. The best solution I’ve thus far found in these threads is to just try to fit the entirety of a project within the limits of the context window and/or to just keep my whole project in a few short files; that may be sufficient for small projects but it’s not possible nor even reasonable given the size and complexity of projects with which I work. As I’ve said elsewhere, I dearly /want/ these things to work in my environment and for my use-cases as capably as they do in/for yours—this stuff is really interesting and I enjoy learning how to do new things—but after reading all the comments in this thread and others I don’t think the needs of my environment are supported by these models. Maybe they will be someday. I’ll keep playing with things but as of right now I see a significant impedance mismatch between the confidence others have in the models’ ability to do complex coding tasks compared to the kinds of tasks I’ve seen demonstrated here and elsewhere.