4 ms·

> Security starts with deep understanding. I wonder if the way we are approaching it is wrong. We are basically putting text though a deep learning black box.

> Security starts with deep understanding. I wonder if the way we are approaching it is wrong. We are basically putting text though a deep learning black box. The model might have learned some abstractions, but all in all it is just playing word games and trying to guess the most likely continuation of a string. Maybe we should go into the other direction and base such an AI on a really massive ontology. Instead of unstructured strings, put highly structured facts into the model. For example, just like in Copilot you'd start with: def login_user(username, password): But the ontology would also know things like: - This is a web application and this function is going to be called after submitting a form - Security specialist Bob says you should always hash your passwords - Specialist Anne says you should use bcrypt - Tom says Anne is 95% trustworthy ... and thousands of facts more. And then it would take them all into consideration, build a represenation of the problem you are trying to solve, find a strategy, and only in the end generate code. I have a feeling that there was a qualitiative leap going from simple neural networks and multivariate methods to "deep learning" and modern machine learning, and that this is mainly driven by scale and available computing power. Now what if we try the same thing for ontologies, expert systems, and triple store databases? I think the difference will be between some AI parroting what it read on Wikipedia (direct speach), and a smarter AI being able to reason about what it read on Wikipedia (indirect speach).

lmilcin 5y ago
I think one way this could be improved is, instead of giving an exact answer (which is provably impossible to do correctly) maybe it could be possible to point the developer to other repositories where other people were solving similar problem. There are already services that do this for you and I actually find them useful. For example, I might be trying to use a function from some library and it fails. If I get pointed to some public repositories that use the same library in function for similar purpose, I may learn that I am missing some critical setup. I can also browse different uses of this function/library and get informed on how it is at the very least used successfully by others.
DonHopkins 5y ago
You're touching on the "Neat -vs- Scruffy" dichotomy in AI. (But it's not necessarily a dichotomy -- they can be combined!) https://en.wikipedia.org/wiki/Neats_and_scruffies https://en.wikipedia.org/wiki/Neats_and_scruffies From the "Scruffy" side, there's Charles Rich's classic work on "Programmer's Apprentice". https://dspace.mit.edu/handle/1721.1/6054 https://dspace.mit.edu/handle/1721.1/6054 https://dspace.mit.edu/bitstream/handle/1721.1/6054/AIM-1004.pdf?sequence=2&isAllowed=y https://dspace.mit.edu/bitstream/handle/1721.1/6054/AIM-1004... >The Programmer's Apprentice Project: A Research Overview >MIT AI Lab Memo No. 1004, November 1987. >Rich, Charles; Waters, Richard C. >Abstract: The goal of the Programmer's Apprentice project is to develop a theory of how expert programmers analyze, synthesize, modify, explain, specify, verify, and document programs. This research goal overlaps both artificial intelligence and software engineering. From the viewpoint of artificial intelligence, we have chosen programming as a domain in which to study fundamental issues of knowledge representation and reasoning. From the viewpoint of software engineering, we seek to automate the programming process by applying techniques from artificial intelligence. https://dspace.mit.edu/handle/1721.1/41967 https://dspace.mit.edu/handle/1721.1/41967 https://dspace.mit.edu/bitstream/handle/1721.1/41967/AI_WP_147.pdf?sequence=1&isAllowed=y https://dspace.mit.edu/bitstream/handle/1721.1/41967/AI_WP_1... >Plan Recognition in a Programmer's Apprentice. Ph.D. Thesis proposal. >MIT AI Lab Working Paper 147, May 1977. >Rich, Charles >Abstract: Brief Statement of the Problem: Stated most generally, the proposed research is concerned with understanding and representing the teleological structure of engineered devices. More specifically, I propose to study the teleological structure of computer programs written in LISP which perform a wide range of non-numerical computations. The major theoretical goal of the research is to further develop a formal representation for teleological structure, called plans, which will facilitate both the abstract description of particular programs, and the compilation of a library of programming expertise in the domain of non-numerical computation. Adequacy of the theory will be demonstrated by implementing a system (to eventually become part of a LISP Programmer's Apprentice) which will be able to recognize various plans in LISP programs written by human programmers and thereby generate cogent explanations of how the programs work, including the detection of some programming errors.
- captainmuon 5y ago
  Thanks for the term and the resources! Sometimes one has a vague idea and it's really nice to see that this is a thing people put thought into. Funny that I would describe a solution based on machine learning as scruffy and a solution based on bayesian logic and knowledge databases as neat, whereas Wikipedia defines it the other way around.
perl4ever 5y ago
There was a project started in 1984 to do that: https://en.wikipedia.org/wiki/Cyc https://en.wikipedia.org/wiki/Cyc Supposedly an attempt to assemble a database of "common sense" facts and reasoning. It has always been controversial and it's not clear what kind of success it's had.

lmilcin 5y ago

I think one way this could be improved is, instead of giving an exact answer (which is provably impossible to do correctly) maybe it could be possible to point the developer to other repositories where other people were solving similar problem. There are already services that do this for you and I actually find them useful. For example, I might be trying to use a function from some library and it fails. If I get pointed to some public repositories that use the same library in function for similar purpose, I may learn that I am missing some critical setup. I can also browse different uses of this function/library and get informed on how it is at the very least used successfully by others.

DonHopkins 5y ago

You're touching on the "Neat -vs- Scruffy" dichotomy in AI. (But it's not necessarily a dichotomy -- they can be combined!) https://en.wikipedia.org/wiki/Neats_and_scruffies https://en.wikipedia.org/wiki/Neats_and_scruffies From the "Scruffy" side, there's Charles Rich's classic work on "Programmer's Apprentice". https://dspace.mit.edu/handle/1721.1/6054 https://dspace.mit.edu/handle/1721.1/6054 https://dspace.mit.edu/bitstream/handle/1721.1/6054/AIM-1004.pdf?sequence=2&isAllowed=y https://dspace.mit.edu/bitstream/handle/1721.1/6054/AIM-1004... >The Programmer's Apprentice Project: A Research Overview >MIT AI Lab Memo No. 1004, November 1987. >Rich, Charles; Waters, Richard C. >Abstract: The goal of the Programmer's Apprentice project is to develop a theory of how expert programmers analyze, synthesize, modify, explain, specify, verify, and document programs. This research goal overlaps both artificial intelligence and software engineering. From the viewpoint of artificial intelligence, we have chosen programming as a domain in which to study fundamental issues of knowledge representation and reasoning. From the viewpoint of software engineering, we seek to automate the programming process by applying techniques from artificial intelligence. https://dspace.mit.edu/handle/1721.1/41967 https://dspace.mit.edu/handle/1721.1/41967 https://dspace.mit.edu/bitstream/handle/1721.1/41967/AI_WP_147.pdf?sequence=1&isAllowed=y https://dspace.mit.edu/bitstream/handle/1721.1/41967/AI_WP_1... >Plan Recognition in a Programmer's Apprentice. Ph.D. Thesis proposal. >MIT AI Lab Working Paper 147, May 1977. >Rich, Charles >Abstract: Brief Statement of the Problem: Stated most generally, the proposed research is concerned with understanding and representing the teleological structure of engineered devices. More specifically, I propose to study the teleological structure of computer programs written in LISP which perform a wide range of non-numerical computations. The major theoretical goal of the research is to further develop a formal representation for teleological structure, called plans, which will facilitate both the abstract description of particular programs, and the compilation of a library of programming expertise in the domain of non-numerical computation. Adequacy of the theory will be demonstrated by implementing a system (to eventually become part of a LISP Programmer's Apprentice) which will be able to recognize various plans in LISP programs written by human programmers and thereby generate cogent explanations of how the programs work, including the detection of some programming errors.

captainmuon 5y ago
Thanks for the term and the resources! Sometimes one has a vague idea and it's really nice to see that this is a thing people put thought into. Funny that I would describe a solution based on machine learning as scruffy and a solution based on bayesian logic and knowledge databases as neat, whereas Wikipedia defines it the other way around.

perl4ever 5y ago

There was a project started in 1984 to do that: https://en.wikipedia.org/wiki/Cyc https://en.wikipedia.org/wiki/Cyc Supposedly an attempt to assemble a database of "common sense" facts and reasoning. It has always been controversial and it's not clear what kind of success it's had.