3 ms·

I said program synthesis good enough to replace programmers requires AGI. Program synthesis based off of informal specifications in natural language. Not talkin

I said program synthesis good enough to replace programmers requires AGI. Program synthesis based off of informal specifications in natural language. Not talking about highly constrained environments with formal specs. I am not belittling the work going in this space, and I’m sure for highly constrained and narrow use cases a lot can be done even now. But I believe solving the general problem of program synthesis based on informal spec requires AGI. I am hardly the only one who thinks this.

YeGoblynQueenne 5y ago
>> I am not belittling the work going in this space, and I’m sure for highly constrained and narrow use cases a lot can be done even now. No. Program synthesis approaches work very well for a broad array of problems, not for "highly constrained and narrow use cases"- that is a misconception of the kind that results from lack of familiarity with modern program synthesis. Here's a good recent review of the field: https://www.microsoft.com/en-us/research/wp-content/uploads/2017/10/program_synthesis_now.pdf https://www.microsoft.com/en-us/research/wp-content/uploads/... Sumit Gulwani, that I mentioned in my previous comment, is an author. To clarify, I'm not in any way affiliated with him or his collaborators. I'm actually from a rival camp, if you will, but the paper I link to is a very good summary of the state of the art. It should help you if you wish to understand where program synthesis is at. >> I said program synthesis good enough to replace programmers requires AGI. Program synthesis based off of informal specifications in natural language. Program synthesis from natural language is hard to make work because it's difficult to translate natural language specifications to specifications that a program synthesiser can use. But that is a limitation of current natural language analysis, specifically natural language understanding, approaches - not a limitation of program syhtesis approaches. I think you equate formal specifications, or specification by example, with "narrow use cases". There's no connection between the two.
- computerex 5y ago
  If program synthesis is as far advanced as you say it is, how come I make six figures doing something that you seem to be arguing can be totally automated? The reality seems to disagree with your statements. Program synthesis is as of right now limited to academic research and highly narrow use cases. If the opposite was true, I’d be out of a job. I think copilot is probably the first product of its type that might make its way into the hands of users en masse. Edit: Btw I was referring to program synthesis based off informal natural language spec. Spec inference is part of the synthesis pipeline, I think it’s not fair to just ignore that problem.
  - YeGoblynQueenne 5y ago
    The purpose of program synthesis is not to get programmers out of a job. Rather, it's a tool to help programmers better do their job. I think it's easy to see why you're not using it. With few exceptions, advances in research take many years to percolate down to the industry. And of course the industry is famous for following trends without real understanding of anything. Anyway the review I linked to has some examples of real-world applications of program synthesis. Don't be afraid to read it- it's light on formal notation and you don't need special skills to understand it. I appreciate that it's a long document but there's a Table of Contents at the start and you should be able to skim through in a short time just to get a general idea of the subject. Anyway I can see you're trying to "wing it" and reason from first principles about something you know nothing about, in true SWE style. Yet, you don't know what you don't know, so you start from the wrong assumptions ("fully automated" etc) and arrive at the wrong conclusions. That's no way to understand anything. It's certainly not going to give you any good idea about what's going on in an entire field of research you know nothing about. Of course you're not obliged to know anything about program synthesis, but in that case, maybe consider sitting back and listening rather than expressing strong opinions with absolute conviction that is not supported by your knowledge? I think that will make a better conversation, and a better internet, for everyone.
abeppu 5y ago
I think you're holding text-based approaches and synthesis based approaches to radically different expectations. Copilot isn't approaching replacing programmers; presumably a programmer is invoking it, deciding what to keep or change, etc, i.e. generating parts of programs under the guidance of a human programmer. Synthesis can work at the level of providing an expression or a helper function, as a useful tool under the guidance of a programmer. Copilot suggests some code snippets, and not necessarily good ones. To be dismissive of another approach to generate parts of programs because they cannot replace programmers is like saying that belt-drive bikes aren't worth considering over chains because a belt-drive bike isn't a replacement for a Learjet.
- computerex 5y ago
  To be clear I wasn’t dismissing anything, that was not my intent. I think as a programmer assist text based approaches work and I really like copilot for what it is. I was merely saying that for the holy grail, program synthesis from informal spec generalised to any domain, the approach will have to be different.
  - YeGoblynQueenne 5y ago
    Who said that's the holy grail? Again I think you're trying to imagine what program synthesis must be like, rather than trying to find out what it actually is like.
    - computerex 5y ago
      Intuition tells me it’s the holy grail, confirmed by tradition and literature. You are being a little condescending btw, it’s not in the spirit of hacker news. > The ability to automatically discover a program consistent with a given user intent (specification) is the holy grail of Computer Science. https://drops.dagstuhl.de/opus/volltexte/2017/7128/pdf/LIPIcs-SNAPL-2017-16.pdf https://drops.dagstuhl.de/opus/volltexte/2017/7128/pdf/LIPIc... > Since the inception of AI in the 1950s, this problem has been considered the holy grail of Computer Science. https://www.microsoft.com/en-us/research/publication/program-synthesis/ https://www.microsoft.com/en-us/research/publication/program... https://dl.acm.org/doi/fullHtml/10.1145/242224.242304 https://dl.acm.org/doi/fullHtml/10.1145/242224.242304 > Automatic synthesis of programs has long been one of the holy grails of software engineering. https://people.eecs.berkeley.edu/~sseshia/pubdir/icse10-TR.pdf https://people.eecs.berkeley.edu/~sseshia/pubdir/icse10-TR.p...

YeGoblynQueenne 5y ago

>> I am not belittling the work going in this space, and I’m sure for highly constrained and narrow use cases a lot can be done even now. No. Program synthesis approaches work very well for a broad array of problems, not for "highly constrained and narrow use cases"- that is a misconception of the kind that results from lack of familiarity with modern program synthesis. Here's a good recent review of the field: https://www.microsoft.com/en-us/research/wp-content/uploads/2017/10/program_synthesis_now.pdf https://www.microsoft.com/en-us/research/wp-content/uploads/... Sumit Gulwani, that I mentioned in my previous comment, is an author. To clarify, I'm not in any way affiliated with him or his collaborators. I'm actually from a rival camp, if you will, but the paper I link to is a very good summary of the state of the art. It should help you if you wish to understand where program synthesis is at. >> I said program synthesis good enough to replace programmers requires AGI. Program synthesis based off of informal specifications in natural language. Program synthesis from natural language is hard to make work because it's difficult to translate natural language specifications to specifications that a program synthesiser can use. But that is a limitation of current natural language analysis, specifically natural language understanding, approaches - not a limitation of program syhtesis approaches. I think you equate formal specifications, or specification by example, with "narrow use cases". There's no connection between the two.

computerex 5y ago
If program synthesis is as far advanced as you say it is, how come I make six figures doing something that you seem to be arguing can be totally automated? The reality seems to disagree with your statements. Program synthesis is as of right now limited to academic research and highly narrow use cases. If the opposite was true, I’d be out of a job. I think copilot is probably the first product of its type that might make its way into the hands of users en masse. Edit: Btw I was referring to program synthesis based off informal natural language spec. Spec inference is part of the synthesis pipeline, I think it’s not fair to just ignore that problem.
- YeGoblynQueenne 5y ago
  The purpose of program synthesis is not to get programmers out of a job. Rather, it's a tool to help programmers better do their job. I think it's easy to see why you're not using it. With few exceptions, advances in research take many years to percolate down to the industry. And of course the industry is famous for following trends without real understanding of anything. Anyway the review I linked to has some examples of real-world applications of program synthesis. Don't be afraid to read it- it's light on formal notation and you don't need special skills to understand it. I appreciate that it's a long document but there's a Table of Contents at the start and you should be able to skim through in a short time just to get a general idea of the subject. Anyway I can see you're trying to "wing it" and reason from first principles about something you know nothing about, in true SWE style. Yet, you don't know what you don't know, so you start from the wrong assumptions ("fully automated" etc) and arrive at the wrong conclusions. That's no way to understand anything. It's certainly not going to give you any good idea about what's going on in an entire field of research you know nothing about. Of course you're not obliged to know anything about program synthesis, but in that case, maybe consider sitting back and listening rather than expressing strong opinions with absolute conviction that is not supported by your knowledge? I think that will make a better conversation, and a better internet, for everyone.

abeppu 5y ago

I think you're holding text-based approaches and synthesis based approaches to radically different expectations. Copilot isn't approaching replacing programmers; presumably a programmer is invoking it, deciding what to keep or change, etc, i.e. generating parts of programs under the guidance of a human programmer. Synthesis can work at the level of providing an expression or a helper function, as a useful tool under the guidance of a programmer. Copilot suggests some code snippets, and not necessarily good ones. To be dismissive of another approach to generate parts of programs because they cannot replace programmers is like saying that belt-drive bikes aren't worth considering over chains because a belt-drive bike isn't a replacement for a Learjet.

computerex 5y ago
To be clear I wasn’t dismissing anything, that was not my intent. I think as a programmer assist text based approaches work and I really like copilot for what it is. I was merely saying that for the holy grail, program synthesis from informal spec generalised to any domain, the approach will have to be different.
- YeGoblynQueenne 5y ago
  Who said that's the holy grail? Again I think you're trying to imagine what program synthesis must be like, rather than trying to find out what it actually is like.
  - computerex 5y ago
    Intuition tells me it’s the holy grail, confirmed by tradition and literature. You are being a little condescending btw, it’s not in the spirit of hacker news. > The ability to automatically discover a program consistent with a given user intent (specification) is the holy grail of Computer Science. https://drops.dagstuhl.de/opus/volltexte/2017/7128/pdf/LIPIcs-SNAPL-2017-16.pdf https://drops.dagstuhl.de/opus/volltexte/2017/7128/pdf/LIPIc... > Since the inception of AI in the 1950s, this problem has been considered the holy grail of Computer Science. https://www.microsoft.com/en-us/research/publication/program-synthesis/ https://www.microsoft.com/en-us/research/publication/program... https://dl.acm.org/doi/fullHtml/10.1145/242224.242304 https://dl.acm.org/doi/fullHtml/10.1145/242224.242304 > Automatic synthesis of programs has long been one of the holy grails of software engineering. https://people.eecs.berkeley.edu/~sseshia/pubdir/icse10-TR.pdf https://people.eecs.berkeley.edu/~sseshia/pubdir/icse10-TR.p...