4 ms·

CodePhage - Automatic bug repair, without access to source code

62 points by tvvocold 11y ago

Piskvorrr 11y ago
The old adage "fixing a bug introduces at least two more" comes to mind. Now fully automated! ;) But seriously: autogenerating fixes as observed by fuzzing does sound cool.
- collyw 11y ago
  So do "schemaless databases".
  - roflc0ptic 11y ago
    Okay are schemaless databases really such a bad idea? Where does the hate come from? I just implemented an EAV based system and it was and remains terrible, but the requirements were to allow users to create arbitrary fields, and to use MySQL. It seems like mongodb/(xyz norel system) would have been such a better choice. I don't know what other options exist. In our case I think it was blind fear that drove the decision not to use mongo, but I see enough people talking them down...
  - robmccoll 11y ago
    "Schemaless databases" are cool. I would rather maintain as little schema as possible in the database and instead keep it in my code. Schema must exist somewhere because my data model exists. Since the data model and my program are tightly coupled (since the entire point of the program is to manipulate internal data state due to external and internal stimulus and produce output), keeping them together makes the overall system manageable versus the headache of effectively having code in two places. This is similar for complicated routing configurations pushed to proxies and other external config. They are really part of your code.
    - MrBuddyCasino 11y ago
      If you can specify those invariants in your code in a concise and declarative way, minimizing the chance for bugs, then great. Thats not how it usually works though. Also, how do you want to perform foreign key checks outside of the db in a way thats fast?
      - carapace 11y ago
        "If you can specify those invariants in your code in a concise and declarative way" ...then you've got a schema. ;-)
    - collyw 11y ago
      Try programming that way for a few more years. It will wear you down.
tluyben2 11y ago
Related: http://dijkstra.cs.virginia.edu/genprog/ http://dijkstra.cs.virginia.edu/genprog/ Edit; this one is open source; is the MIT one ? Couldn't find references on the page?
higherpurpose 11y ago
Related or just a coincidence? > An Obama Administration official tells Re/code that recent advances in using automated methods to analyze software code for vulnerabilities have spurred interest in government circles to see if there’s a way to standardize how software is tested for security and safety. https://recode.net/2015/06/29/famed-security-researcher-mudge-leaves-google-for-white-house-gig/ https://recode.net/2015/06/29/famed-security-researcher-mudg... I just wonder what will happen to Google's Project Vault [1] now that Mudge is gone. Hopefully it will still be on track. [1] https://www.youtube.com/watch?v=V6qrQzn8uBo https://www.youtube.com/watch?v=V6qrQzn8uBo
- miander 11y ago
  Regarding the quote from the article, I'm guessing it's referring to code fuzzing and static analysis tools which have now become relatively widespread.
tempodox 11y ago
If I exaggerate just a little bit, that means the end of us programmers (though not right away). Just imagine, any-one can make a rough sketch of a “computer program” and have a system like CodePhage fill in the blanks. There would still be CS people for fundamental research and new discoveries, but the rest of the software industry would collapse into one automated know-it-all software replicator. Someone wake me up please!
- TeMPOraL 11y ago
  Part of me is scared, but another thinks "it's about time". Web industry is mostly millions of people coding up the same CRUDs all over again, it cries for being automated.
  - tempodox 11y ago
    As for web apps, I would already be happy if you only needed two languages (client / server, roughly) instead of five-ish. And yes, lots of Java programs accessing a DB, slapping a front end on it that lets people enter / request stuff in the browser are prime auto-generation suspects.
    - anentropic 11y ago
      ...what are the other three?
      - TeMPOraL 11y ago
        HTML, JS, CSS, SQL and something to write your backend in, at minimum.
      - arithma 11y ago
        I guess: Backend, Frontend, Markup, Layout/Presentation, DB Querying
  - tluyben2 11y ago
    The thing is that we keep inventing different ways of doing the same thing; different languages, databases, frontend tech, backend tech, patterns etc. The outcome is the same but the way of doing it is vastly different. Any hope to really automate this (and get a lot more reuse going) can only come from people using at least the same tech and same patterns. Doing it 'a bit faster' in the 'not very stable but very rapid web dev framework used by a group of at least 10 people, maybe' is killing most chances of doing this kind of thing but also of sensible reuse across the web. There is a reason why people are still paying for 'employee list' crud project in framework X while it is easy to find on some other frameworks; we keep reinventing it for every framework on (and off) the web and then forget that ever happened a month later.
vmorgulis 11y ago
https://news.ycombinator.com/item?id=9800883 https://news.ycombinator.com/item?id=9800883
TeeWEE 11y ago
FYI The full paper can be found here: http://delivery.acm.org/10.1145/2740000/2737988/p43-sidirogloudouskos.pdf?ip=80.113.211.165&id=2737988&acc=OPEN&key=4D4702B0C3E38B35%2E4D4702B0C3E38B35%2E4D4702B0C3E38B35%2E6D218144511F3437&CFID=524572299&CFTOKEN=51965653&__acm__=1435665069_096eebca6644a0f5e695dea4e5bdc5c8 http://delivery.acm.org/10.1145/2740000/2737988/p43-sidirogl... In summary, for an app with an bug. You must know a) input that causes to bug to show up, b) input that doesnt cause an error. Then it will look for similar code in github, and try inserting the checks done from that code into the new code. And then it reruns the program, hoping that the bug is resolved. This approach is very cool, and harnesses the power of lots of developer. But its also very limited. However thats what research is for. Small steps together are a big leap for mankind :p I also like this conlusion in the article: """ In recent years the increasing scope and volume of software development efforts has produced a broad range of systems with similar or overlapping goals. Together, these systems capture the knowledge and labor of many developers. But each individual system largely reflects the effort of a single team and, like essentially all software systems, still contains errors. We present a new and, to the best of our knowledge, the first, technique for automatically transferring code between systems to eliminate errors. The system that implements this technique, CP, makes it possible to automatically harness the combined efforts of multiple potentially independent development efforts to improve them all regardless of the relationships that may or may not exist across development organizations. In the long run we hope this research will inspire other techniques that identify and combine the best aspects of multiple systems. The ideal result will be significantly more reliable and functional software systems that better serve the needs of our society. """
- vezzy-fnord 11y ago
  Well, so much for everyone panicking over strong AI taking over their jobs and writing bug-free software. Not that it isn't impressive, but there's been all sorts of academic projects throughout the years that require lots of manual intervention to work their outwardly powerful techniques, and many of them never even seeing the light of a public release.
omouse 11y ago
I wonder if this works mainly for C, C++, ObjC, Java and C#, Python, Ruby or would it also work for Lisp/Scheme and other more powerful languages? Does it only work for crashes? In any case, CS is awesome. Love it when research that a few years ago would have been theoretical is applied.

62 points by tvvocold 11y ago

The old adage "fixing a bug introduces at least two more" comes to mind. Now fully automated! ;) But seriously: autogenerating fixes as observed by fuzzing does sound cool.

collyw 11y ago
So do "schemaless databases".
- roflc0ptic 11y ago
  Okay are schemaless databases really such a bad idea? Where does the hate come from? I just implemented an EAV based system and it was and remains terrible, but the requirements were to allow users to create arbitrary fields, and to use MySQL. It seems like mongodb/(xyz norel system) would have been such a better choice. I don't know what other options exist. In our case I think it was blind fear that drove the decision not to use mongo, but I see enough people talking them down...
- robmccoll 11y ago
  "Schemaless databases" are cool. I would rather maintain as little schema as possible in the database and instead keep it in my code. Schema must exist somewhere because my data model exists. Since the data model and my program are tightly coupled (since the entire point of the program is to manipulate internal data state due to external and internal stimulus and produce output), keeping them together makes the overall system manageable versus the headache of effectively having code in two places. This is similar for complicated routing configurations pushed to proxies and other external config. They are really part of your code.
  - MrBuddyCasino 11y ago
    If you can specify those invariants in your code in a concise and declarative way, minimizing the chance for bugs, then great. Thats not how it usually works though. Also, how do you want to perform foreign key checks outside of the db in a way thats fast?
    - carapace 11y ago
      "If you can specify those invariants in your code in a concise and declarative way" ...then you've got a schema. ;-)
  - collyw 11y ago
    Try programming that way for a few more years. It will wear you down.

tluyben2 11y ago

Related: http://dijkstra.cs.virginia.edu/genprog/ http://dijkstra.cs.virginia.edu/genprog/ Edit; this one is open source; is the MIT one ? Couldn't find references on the page?

higherpurpose 11y ago

Related or just a coincidence? > An Obama Administration official tells Re/code that recent advances in using automated methods to analyze software code for vulnerabilities have spurred interest in government circles to see if there’s a way to standardize how software is tested for security and safety. https://recode.net/2015/06/29/famed-security-researcher-mudge-leaves-google-for-white-house-gig/ https://recode.net/2015/06/29/famed-security-researcher-mudg... I just wonder what will happen to Google's Project Vault [1] now that Mudge is gone. Hopefully it will still be on track. [1] https://www.youtube.com/watch?v=V6qrQzn8uBo https://www.youtube.com/watch?v=V6qrQzn8uBo

miander 11y ago
Regarding the quote from the article, I'm guessing it's referring to code fuzzing and static analysis tools which have now become relatively widespread.

tempodox 11y ago

If I exaggerate just a little bit, that means the end of us programmers (though not right away). Just imagine, any-one can make a rough sketch of a “computer program” and have a system like CodePhage fill in the blanks. There would still be CS people for fundamental research and new discoveries, but the rest of the software industry would collapse into one automated know-it-all software replicator. Someone wake me up please!

TeMPOraL 11y ago
Part of me is scared, but another thinks "it's about time". Web industry is mostly millions of people coding up the same CRUDs all over again, it cries for being automated.
- tempodox 11y ago
  As for web apps, I would already be happy if you only needed two languages (client / server, roughly) instead of five-ish. And yes, lots of Java programs accessing a DB, slapping a front end on it that lets people enter / request stuff in the browser are prime auto-generation suspects.
  - anentropic 11y ago
    ...what are the other three?
    - TeMPOraL 11y ago
      HTML, JS, CSS, SQL and something to write your backend in, at minimum.
    - arithma 11y ago
      I guess: Backend, Frontend, Markup, Layout/Presentation, DB Querying
- tluyben2 11y ago
  The thing is that we keep inventing different ways of doing the same thing; different languages, databases, frontend tech, backend tech, patterns etc. The outcome is the same but the way of doing it is vastly different. Any hope to really automate this (and get a lot more reuse going) can only come from people using at least the same tech and same patterns. Doing it 'a bit faster' in the 'not very stable but very rapid web dev framework used by a group of at least 10 people, maybe' is killing most chances of doing this kind of thing but also of sensible reuse across the web. There is a reason why people are still paying for 'employee list' crud project in framework X while it is easy to find on some other frameworks; we keep reinventing it for every framework on (and off) the web and then forget that ever happened a month later.

vmorgulis 11y ago

https://news.ycombinator.com/item?id=9800883 https://news.ycombinator.com/item?id=9800883

TeeWEE 11y ago

FYI The full paper can be found here: http://delivery.acm.org/10.1145/2740000/2737988/p43-sidirogloudouskos.pdf?ip=80.113.211.165&id=2737988&acc=OPEN&key=4D4702B0C3E38B35%2E4D4702B0C3E38B35%2E4D4702B0C3E38B35%2E6D218144511F3437&CFID=524572299&CFTOKEN=51965653&__acm__=1435665069_096eebca6644a0f5e695dea4e5bdc5c8 http://delivery.acm.org/10.1145/2740000/2737988/p43-sidirogl... In summary, for an app with an bug. You must know a) input that causes to bug to show up, b) input that doesnt cause an error. Then it will look for similar code in github, and try inserting the checks done from that code into the new code. And then it reruns the program, hoping that the bug is resolved. This approach is very cool, and harnesses the power of lots of developer. But its also very limited. However thats what research is for. Small steps together are a big leap for mankind :p I also like this conlusion in the article: """ In recent years the increasing scope and volume of software development efforts has produced a broad range of systems with similar or overlapping goals. Together, these systems capture the knowledge and labor of many developers. But each individual system largely reflects the effort of a single team and, like essentially all software systems, still contains errors. We present a new and, to the best of our knowledge, the first, technique for automatically transferring code between systems to eliminate errors. The system that implements this technique, CP, makes it possible to automatically harness the combined efforts of multiple potentially independent development efforts to improve them all regardless of the relationships that may or may not exist across development organizations. In the long run we hope this research will inspire other techniques that identify and combine the best aspects of multiple systems. The ideal result will be significantly more reliable and functional software systems that better serve the needs of our society. """

vezzy-fnord 11y ago
Well, so much for everyone panicking over strong AI taking over their jobs and writing bug-free software. Not that it isn't impressive, but there's been all sorts of academic projects throughout the years that require lots of manual intervention to work their outwardly powerful techniques, and many of them never even seeing the light of a public release.

omouse 11y ago

I wonder if this works mainly for C, C++, ObjC, Java and C#, Python, Ruby or would it also work for Lisp/Scheme and other more powerful languages? Does it only work for crashes? In any case, CS is awesome. Love it when research that a few years ago would have been theoretical is applied.