14 ms·

How AI based programming could work

89 points by bjenik 10y ago

userbinator 10y ago
Suppose you have an AI that generated some code for you, but it doesn't do exactly what you want. Now try to debug it... what if programming wouldn't involve defining exact steps, but instead just roughly defining what we have and what we want and maybe giving a few hints, and having the computer generally do the right thing - wouldn't that be awesome? I experience enough frustration with things like debugging generated code, ostensibly "smart" devices which seem to "have a mind of their own" (maybe that's the point, but it's not doing what I want), and getting Google's search engine to find exactly what I want with its pseudo-AI machine-learning algorithms doing completely inexplicable things with my queries, that I think "generally do the right thing" is not a good idea. Edge cases matter a lot.
- bbctol 10y ago
  Yeah, I think this article is talking about something that isn't programming at all. There will always be programmers (unless we're talking about a real robot uprising situation) and if we enter a world where people can tell AIs to perform programming tasks effectively, that won't just be the provenance of programmers; it will reshape society in general.
- coredog64 10y ago
  Don't we already have a similar situation today with the compiler and application stacks? I'm not writing my business logic in assembler -- I'm writing it in Java, maybe pulling in some third party libraries, using Spring, and then running on the JVM. All kinds of opportunities for things not to work as I expect and then I have to figure out why.
  - userbinator 10y ago
    The difference is that, as the article points out, traditional programming languages are far more 'exact' - large systems can become difficult to debug, but that largely arises from the complexity of layered abstractions, whose behaviour is still exact. AI-based systems not only add the complexity of abstraction, but also with their inexactness.
- reitanqild 10y ago
  > and getting Google's search engine to find exactly what I want with its pseudo-AI machine-learning algorithms doing completely inexplicable things with my queries, that I think "generally do the right thing" is not a good idea. Actually Google works kind of OK now after they overshot seriously some years ago. Today if you use doublequotes around the search terms it actually seems to search for that exact term. Which I think is a good solution: reasonable AI with easy fallback when you need it. I guess an AI VM would need an exact modifier: don't attempt any unspecified cleverness on this variable/method/class. Specifying some kind of runtime assertions on steroids could perhaps also alleviate some problems. Debugging could be simplified by having the VM outputting all its assumptions.
  - zdkl 10y ago
    Aw man, I was hoping we'd be rid of compiler flags by then
wellsjohnston 10y ago
The whole point of programming is to get exact results by solving all edge cases. Machine learning/Neural nets are only good at guessing results, and cannot solve all edge cases without specific direction.
- joe_the_user 10y ago
  Sure, A program that looks at a pattern, emulates it and then enumerates the various edge-cases, looking for further patterns from the programmer's answer. It would still require effort but it could be a lot easier than present approaches and it might even find edge-cases a human would miss.
  - runeks 10y ago
    > A program that looks at a pattern [...] This is what programming already is, the "pattern" being code. A compiler is a program that looks at a code pattern and, if it's a good compiler, it can tell you if you've missed edge-cases through compile time errors.
- bpicolo 10y ago
  That's the traditional focus of programming, but I wouldn't say that makes it the whole point. I very much expect programming to change over time
anthk 10y ago
>what if programming wouldn't involve defining exact steps, but instead just roughly defining what we have and what we want and maybe giving a few hints, and having the computer generally do the right thing - wouldn't that be awesome? So, Prolog?
- giardini 10y ago
  Why Prolog? I have never noticed that Prolog was any better than other computer languages at reading one's mind!
  - acobster 10y ago
    Generally doing "the right thing" based on a rough definition of "what we have and what we want" is not mind-reading, but it is essentially what Prolog does. The author seems to conflate AI with declarative programming here, and I think that was the point.
  - jcranmer 10y ago
    Prolog, or at least logic programming languages, were thought to be the wave of the future of programming in the 80s. They, of course, turned out to be a massive bust.
    - yazr 10y ago
      You are factually correct But, a key problem was the runtime (in)efficiency of the prolog implementation engine. Maybe today with x1M more powerful computers it is worth trying again ? Neural networks were abandoned in the 80s and made a massive comeback with a. more compute power b. convolutional networks In fact, maybe a ML-optimized VM will make higher-level languages useful again ;)
      - ScottBurson 10y ago
        No, the problem was not that Prolog was too slow. The problem was that it was too stupid. Prolog's strict left-to-right, depth-first search strategy allowed the implementations to be, in fact, quite fast. But that same strategy also kept it from being more than a small step toward true declarative programming, because Prolog programmers have to understand the order in which subgoals are tried. For example, consider a goal that can be satisfied by either of two clauses, which we'll call A and B, and where every problem instance can be solved by exactly one of these; however, clause A always completes (either succeeding or failing) in constant time, but clause B, if invoked on an instance that would cause it to fail, goes into infinite recursion (an easy thing to do by accident in Prolog). It's imperative (ha!) for the Prolog programmer to make sure that clause A is tried before B. This kind of thing comes up all the time in Prolog programming. On the one hand, you have to be aware of the order of operations so you don't cause infinite recursions (or, short of that, cause your program to take exponential time); on the other, Prolog programmers frequently exploit the left-to-right, depth-first strategy by using the operator known as the "cut". The cut is a non-logical operator in the sense that its semantics can be expressed only by reference to the Prolog search order. Programs written in a true declarative language might superficially look a lot like Prolog programs, but the implementation would need to be much smarter -- doing things like figuring out the correct order of operations on its own. (I think machine learning might actually turn out to help with this.) Anyway the problem with your suggestion that we just need to throw more hardware at Prolog should be evident by this point. It doesn't matter how fast your machine is if your algorithm is exponential in the typical case (never mind if it contains an infinite recursion!).
nemaar 10y ago
In a sense we already do this. For example, every time you create a website in HTML/CSS and javascript, you use high level instructions and a lot of declarative stuff. It is already very far away from the actual hardware. If you use some framework with builtin templates, all you need is the actual content and some plumbing. Everything else is handled by the lower layers and it is much more declarative than you realize. It also means that you do not specify thousands of things/little details and you trust the lower layers to do the right thing. It is already happening and it will only get better. People freak out every time when this idea appears and the recurring argument is that "the AI will not do the right thing, it is too stupid". They forget that our frameworks/libraries and complicated software stacks already work as dumb AIs. They follow hardcoded rules and try to please us and most of the time it actually works otherwise we would not use libraries. Using a neural network may not be the right choice for this problem but the general idea is correct. A good library should hide the "how" as much as it can and only require the user to specify the "end goal".
chris_va 10y ago
I suspect we'll see supervised seq2seq generated code first. Like: --- Programmer inputs on left: compute std dev of x please AI on right proposes edit to code: + import numpy as np ... + stddev_x = np.std(x) --- Not super complicated to start with, but you can see where it will go from there.
- fizx 10y ago
  How in the world do you train that? Where's the corpus of english <-> code mappings come from?
  - sapphireblue 10y ago
    The corpus can be acquired from companies that host coding competitions. In this paper http://arxiv.org/pdf/1510.07211v1.pdf http://arxiv.org/pdf/1510.07211v1.pdf (I already mentioned it many times, but it is very relevant) researchers took the data from a similar source >To accomplish this goal, we leverage a dataset from a pedagogical programming online judge (OJ) system,2 intended for the undergraduate course, Introduction to Computing. The OJ system comprises different programming problems. Students submit their source code to a specific problem, and the OJ system judges its validity automatically (via running). And trained seq2seq model on it on character level and it almost works, barring a few typos. IMHO this is an underappreciated breakthrough. With more data and a better model generating competition-grade programs seems possible.
sapphireblue 10y ago
The author proposes a lot of vague ideas in this article (for example "I believe one of the biggest problems is the use of Error Propagation and Gradient Descent") without references or any solid proofs why they are necessary to solve the proposed program (Automate programming using ML?). In fact there is already a lot of solid work just on this subject: * Learning algorithms from examples http://arxiv.org/abs/1511.07275 http://arxiv.org/abs/1511.07275 https://arxiv.org/abs/1410.5401 https://arxiv.org/abs/1410.5401 * Generating source code from natural language description http://arxiv.org/abs/1510.07211 http://arxiv.org/abs/1510.07211 * And, the most closest work to what author probably wants, a way to write a program in forth while leaving some functions as neural blackboxes to be learned from examples: http://arxiv.org/abs/1605.06640 http://arxiv.org/abs/1605.06640 * Also there is a whole research program by nothing less than Facebook AI Research that explicitly aims at creating a conversational AI agent that is able to translate user's natural language orders into programs (asking to the user additional questions if necessary): http://arxiv.org/abs/1511.08130 http://arxiv.org/abs/1511.08130 (there is also a summary here http://colinraffel.com/wiki/a_roadmap_towards_machine_intelligence http://colinraffel.com/wiki/a_roadmap_towards_machine_intell... ) And deepmind is also working on conversational agents: https://youtu.be/vQXAsdMa_8A?t=1265 https://youtu.be/vQXAsdMa_8A?t=1265 Given current success of such models, automating simple programming tasks maybe not as much research as engineering and scaling up problem. There is a lot of exciting machine learning research out there nowadays. Almost all of this research is available for free from papers posted on arxiv. It is a really good idea to read more about state of the art before coming with new ideas.
- bjenik 10y ago
  The forth black box paper seems to be quite close actually. The big difference is that I would like to go for higher level tasks. The low level variant - having code missing a few lines/functions and filling those is indeed quite well researched. More interesting would be the opposite: Having blocks of code (called components in my text) and letting the net find a good way to use them. Normally the way a programmer would build something like let's say the amazon website is sketching up a few ideas how it could look - cutting it into small parts and then writing (or reusing) the code for the small parts. A lot of the "using AI for programming" papers (AI in this case being anything from nets to logic stuff) focus on building these small parts - I'm more interested in the levels above that. The "generating code from natural language/dialogs" also kind of misses my idea as it still assumes a "programmer" to tell it what to do (in the amazon example "display the products on a grid" or "make the buy button bigger"). The programmer would in this case either guess the right thing ("everybody has their products in a grid") or run something like A/B testing against some metric ("items sold" or "$ revenue") and then go back to the conversational programming tool and tell it "change the button". Why not skip the programmer and give the metric directly to the net? The thing I would like to get working is telling the net "here is a ton of different buttons, grids, lists and a lot of other UI stuff, and there are all our products - I want to maximize revenue - do whatever you like to get there". Regarding the vague ideas: I believe (also totally unsubstantiated - though research in this area would be quite interesting) that writing down ideas first and then looking into them (either by reading other papers or doing own research) is far better than the other way round, because it has a higher probability of leading to something new, because you don't follow assumptions and errors others potentially made which could lead to a dead end. Of course there is a certain trade off, because if you try to reinvent everything you will not get to the point of something new.
cardigan 10y ago
Starts off with an interesting idea (programming is perfectly specified versus being underspecified but more efficient due to shared context) but then devolves significantly: neural networks are not the be all end all of smart computers. Interesting part is that a smart computer could resolve ambiguities in a human description of program desired in a way similar to humans. Missing part to this idea is that unless it's perfect, it would need a way to explain choices and results and probably some dialog system to iteratively improve upon how it resolved ambiguities. E.g, make me an app like Uber but for cats -> here is a version you can play with -> oh I don't really like the cat icon, can we change it -> sure, which of these would you like ; etc etc Computer requires ever changing human context, and maybe could have individualized context. So it has to learn over time. The point is to maximize efficiency of getting things made: how little needs to be specified before computer gives you something you accept / how fast can you go from some vague idea in your head to working acceptable software. You should at least be able to get as good as an arbitrarily large number of people who lived a very similar life to yours and were put in a time bubble where you could talk with them to have them write software for you, assuming they're organized very well and highly motivated and great programmers and all that.
brett40324 10y ago
I feel dumber after reading this article. So much, that my own human net of neurons can't figure out or explain why. A partial explanation is that I dont like the assumption that once work is being performed at higher and higher abstracted layers, that machine hardware, OS kernels, compilers and linkers, package and dependency managers, and all other systems software supporting high level programming paradigms (including AI based methods) are going to fall in line at the application layer.
- runeks 10y ago
  I think the quality on Hacker News would benefit from more people being able to downvote. At least take away people's right to upvote if they have no right to downvote. It seems to me there's missing a balance if we can't downvote a story that turns out to be lower quality than what it appeared to be.
jcoffland 10y ago
neural networks != AI
- yazr 10y ago
  "How ML based programming will allow us to program using a higher-abstraction programming language" Better ?! There is also the famous quote about how any working (and understood) AI is no longer considered AI...
  - jcoffland 10y ago
    That quote will apply up until we encounter real AI.
iammyIP 10y ago
Since the c language is already a human readable abstraction away from the actual hardware instructions, which then again are abstractions on what actually happens on the chip with the transistors and electrons, you could argue that programming in c is already pretty much 'talking to the computer human-style'. As others said the use of frameworks and even higher level scripting languages with an ide is already a dumb-ai kind of human-friendly conversation with the machine. A problem with driving this further down the road like the article suggests could be, that if i wanted to talk to the machine like i talk with other persons, the natural language used in this case migh not be very well suited to express what i actually want. I would need special definitions and terminology, and in some cases a general description would not suffice, so the need for precision in expression might stay. The actual dialogue with the machine then might last as along as a programmer would have needed to program it. I could see how this might work for very simple programs (like for some uber-app which is not much more than a wrapper for a telephone call to the taxi central), but inherent complexity that other programs need cannot be simplified away by switching the language to a conversational chit chat level.
ww520 10y ago
It's kind of vague. It would be better if he used a concrete example to illustrate how AI help programming.
- bjenik 10y ago
  Actually the examples in the text are quite concrete - the few lines would be everything you'd have to write to get such a site working (modulo a bit of http plumbing). Obviously we are nowhere near such a thing. The rest of the text acts as a bit of a pointer what could help to get there. From my other comment (in this case the example was an amazon like shopping site): > The thing I would like to get working is telling the net "here is a ton of different buttons, grids, lists and a lot of other UI stuff, and there are all our products - I want to maximize revenue - do whatever you like to get there". The way of telling a neural network these things would be these few lines of code which would hook up different parts to get it working.
  - brett40324 10y ago
    You're gonna have to make a clear distinction between web dev project configuration, and AI for making good choices that optimize meeting sales and marketing goals. How is your 'project codebase markup' any different from filling in some radio buttons on a template generator or a CMS?
tiagoespinha 10y ago
Interesting idea, but to be honest I think it will never take off in a large scale. If you want to write a vague and generic piece of code that can figure out by itself what it is that its outputs ought to be with regards to its inputs, you are, with effect, creating artificial intelligence. This, in turn, requires teaching. What if suddenly you have new pages on your website? Your AI program thingy wouldn't be able to serve them until you taught it what these pages are and in what situations the end user might be interested in viewing them. Imagine having a 5 year old kid handling your shop's cashier. And now a new product arrived and you have to explain to him/her that there's this new product, what it's called, the price and which types of people are allowed to buy it (e.g., alcohol couldn't be sold to people younger than X years old). If you're really teaching your application with natural language, like you would with a five year old, then the effort it'd take to get that info into your system would defeat the purpose of actually using a computer to do it (computers are good at storing and looking up stuff in large data sets, better than humans for really large data sets). This whole hype of machine learning is suitable for situations where you have large amounts of data and you want them crunched according to some basic pre-established and non-changing logic without relying on actual human labor. If the logic evolves, you'll always require humans to sit at the helm and steer the right and wrong. My 2 cents.
- orasis 10y ago
  I disagree with your conclusion. In my app, 7 Second Meditation, I am constantly authoring new content. It uses improve.ai to show that new content to some small number of users and learn how they react to that content. The learning happens from the users themselves. In my particular case, within about 300-500 views the system can tell if a new piece of content is total crap and will basically stop showing it. If the content is amazing, within 100-200 views it has a strong idea and will immediately prioritize this content. The results speak for themselves, before I started using machine learning I was getting 4 star ratings and usage was flat. Now I'm getting 5 star ratings and the app is growing. Will it ever take off at large scale? I have no clue, but I've seen such strong benefits from the machine learning that I'm now working on developing improve.ai full time.
  - 10y ago
orasis 10y ago
My project, improve.ai, is a first concrete step forward in this direction. We start by replacing if/then statements with a decide() function and some goal events. I've used this approach in my app, 7 Second Meditation, to achieve a solid 5 star rating across 248 reviews and 40% first month retention. https://itunes.apple.com/us/app/7-second-meditation-daily/id667197203?mt=8 https://itunes.apple.com/us/app/7-second-meditation-daily/id... I'll be writing some articles deep diving into 7 Second Meditation and breaking down how I'm achieving this results. For now, here is a taste: https://blog.improve.ai/hi-world-how-to-write-code-that-learns-efdb8b5af940#.1upmk4vxo https://blog.improve.ai/hi-world-how-to-write-code-that-lear...
teddyknox 10y ago
Why do we need to jump to ML-based programming again? I'm confident that as we build simpler interfaces and workflows for replacing the most modular components in our programs with AI, we'll begin to see which components are the next lowest-hanging fruit and what concrete ML problems need to be solved to model them. We imagine a future where AI becomes a dominant paradigm for "writing" software -- I think that will be the case, but not in the way everyone suspects. I think 80% of the new value of software in the future will be derived from AI components, but that 80% of the production costs will still go into the structural glue code that supports the value-powerhouse models. Thus, most of the software written in the future will look similar to most of the code written today. I also suspect that as the complexity of AI models increases, the structural code required to support these models will keep pace in complexity. For this reason, I see a future where the unit cost of high-quality software does not see any big drop, but where the value of this software continues to increase exponentially. A corollary to this is that valuable software will be no easier to program in the future than it is today.
ThePhysicist 10y ago
Very intersting ideas! I also think that in the future we will see a lot more declarative programming languages that model high-level concepts of a given program, while low-level "plumbing" code will be generated automatically. I'm not yet sure about the role that AI will play in this process though, and I think there are many things that we have to solve before we'll see something that can write generic programs. Program synthesis is a very active field of research and many AI-based methods have been proposed in the last decades, IMHO what most systems lack is the applicability to real-world programming languages and use cases though. Having worked on static program analysis, I know that even our current ability to understand and reason about existing programs is still very limited. A main reason for this is that most real-world systems are composed of many parts that are not easily specifiable under a single paradigm (e.g. templates, database code, configuration files). To build a usable AI-based programming system, we will need: * A description languge that is able to model ALL aspects of a given real-world system under a single paradigm * A system to analyze and understand the artefacts produced by the above system * A way to generate real-world code from the specification above, including a way to "fill in the blanks" that the user did not specify (as leaving out the details the whole point of such a system) * A way to test the generated code against the specifications provided by the user and further "reasonable" assumptions, which will be needed as the specifications by the user will not be complete, see above * A way to guide this process towards a reasonable program through user feedback within a reasonably small number of steps While none of these things are impossible, implementing them is a significant challenge with a lot of unknowns for which we don't have good solutions yet. I therefore think the first AI-based systems which we'll see in the coming years will be limited to specific problem domains (e.g. data analysis, logic programming) for which we can more easily build a system as the above.
jcranmer 10y ago
The keyword you want to search for is "program synthesis." It already exists, and no neural nets need apply (neural nets tend to be useful only if you can't get anything else to work). As a bonus, it's not probabilistic like AI techniques tend to be but exact, based on SMT solvers and verified proof correctness. An example of work showcased at this year's PLDI that's capable of doing this sort of stuff: Fast synthesis of fast collections (https://homes.cs.washington.edu/~mernst/pubs/collection-synthesis-pldi2016-abstract.html https://homes.cs.washington.edu/~mernst/pubs/collection-synt...): specify a database-like data structure for a collection of objects and the list of queries performed on that data structure, and get out code that's as fast as hand-tuned data structures with fewer bugs. Programmatic and Direct Manipulation, Together at Last (http://dl.acm.org/citation.cfm?id=2908103&dl=ACM&coll=DL&CFID=609587099&CFTOKEN=40441624 http://dl.acm.org/citation.cfm?id=2908103&dl=ACM&coll=DL&CFI...): Take an image generated by code (e.g., periodic stripes), and be able to manipulate that image by drag-and-drop, e.g., changing stripe size or period. Stratified synthesis: automatically learning the x86-64 instruction set (https://stefanheule.com/publications/pldi16-strata/ https://stefanheule.com/publications/pldi16-strata/): 60% of the x86-64 instruction set can be formally specified starting from about 60 base or pseudo-instructions (basically, describing each instruction as an assembly program of simpler instructions).
- YeGoblynQueenne 10y ago
  Nice, thanks- particularly the x86-64 instruction set learning. I would kind of put that down as grammar induction rather than program synthesis though.
viach 10y ago
Interesting, how AI based project management could work? It is much simpler to implement and could really save lots of money for organizations, imo
YeGoblynQueenne 10y ago
>> I believe a lot of research today is limited by first looking at, and becoming an expert in, the status quo and then building small iterative improvements. It would be better to first find a goal and then looking for a way of getting there - at least that's the way we got to the moon. Nope.
brett40324 10y ago
#3 on HN for the moment is very relevant to some of the discussion in this thread pertaining to websites. https://news.ycombinator.com/item?id=12198572 https://news.ycombinator.com/item?id=12198572
BucketSort 10y ago
I've been toying around with the idea of making user agents that simulate the functions a user must be able to accomplish through a software and using that as a cost function for some evolutionary algorithm.
pennyhill 10y ago
The workout then could to start with get started with a single neuron relating to the feedback and also the production and dynamically removing and adding them right up until being nearly some tolerance of effectiveness. A completely new part - once again with a single neuron - could then be generated choosing yet another metric. Maybe something like if one layer is a lot bigger than both of its neighbors. During the long-term I hope it can be much better to depart from the part construction. http://www.crazysales.com.au/ http://www.crazysales.com.au/ Particularly when moving in direction of fully commited hardware as an alternative to float-crunching on GPUs. Aside from that, a number of the more advanced themes might not exactly be also conceivable with a tidy coating system.

89 points by bjenik 10y ago

Suppose you have an AI that generated some code for you, but it doesn't do exactly what you want. Now try to debug it... what if programming wouldn't involve defining exact steps, but instead just roughly defining what we have and what we want and maybe giving a few hints, and having the computer generally do the right thing - wouldn't that be awesome? I experience enough frustration with things like debugging generated code, ostensibly "smart" devices which seem to "have a mind of their own" (maybe that's the point, but it's not doing what I want), and getting Google's search engine to find exactly what I want with its pseudo-AI machine-learning algorithms doing completely inexplicable things with my queries, that I think "generally do the right thing" is not a good idea. Edge cases matter a lot.

bbctol 10y ago
Yeah, I think this article is talking about something that isn't programming at all. There will always be programmers (unless we're talking about a real robot uprising situation) and if we enter a world where people can tell AIs to perform programming tasks effectively, that won't just be the provenance of programmers; it will reshape society in general.
coredog64 10y ago
Don't we already have a similar situation today with the compiler and application stacks? I'm not writing my business logic in assembler -- I'm writing it in Java, maybe pulling in some third party libraries, using Spring, and then running on the JVM. All kinds of opportunities for things not to work as I expect and then I have to figure out why.
- userbinator 10y ago
  The difference is that, as the article points out, traditional programming languages are far more 'exact' - large systems can become difficult to debug, but that largely arises from the complexity of layered abstractions, whose behaviour is still exact. AI-based systems not only add the complexity of abstraction, but also with their inexactness.
reitanqild 10y ago
> and getting Google's search engine to find exactly what I want with its pseudo-AI machine-learning algorithms doing completely inexplicable things with my queries, that I think "generally do the right thing" is not a good idea. Actually Google works kind of OK now after they overshot seriously some years ago. Today if you use doublequotes around the search terms it actually seems to search for that exact term. Which I think is a good solution: reasonable AI with easy fallback when you need it. I guess an AI VM would need an exact modifier: don't attempt any unspecified cleverness on this variable/method/class. Specifying some kind of runtime assertions on steroids could perhaps also alleviate some problems. Debugging could be simplified by having the VM outputting all its assumptions.
- zdkl 10y ago
  Aw man, I was hoping we'd be rid of compiler flags by then

wellsjohnston 10y ago

The whole point of programming is to get exact results by solving all edge cases. Machine learning/Neural nets are only good at guessing results, and cannot solve all edge cases without specific direction.

joe_the_user 10y ago
Sure, A program that looks at a pattern, emulates it and then enumerates the various edge-cases, looking for further patterns from the programmer's answer. It would still require effort but it could be a lot easier than present approaches and it might even find edge-cases a human would miss.
- runeks 10y ago
  > A program that looks at a pattern [...] This is what programming already is, the "pattern" being code. A compiler is a program that looks at a code pattern and, if it's a good compiler, it can tell you if you've missed edge-cases through compile time errors.
bpicolo 10y ago
That's the traditional focus of programming, but I wouldn't say that makes it the whole point. I very much expect programming to change over time

anthk 10y ago

>what if programming wouldn't involve defining exact steps, but instead just roughly defining what we have and what we want and maybe giving a few hints, and having the computer generally do the right thing - wouldn't that be awesome? So, Prolog?

giardini 10y ago
Why Prolog? I have never noticed that Prolog was any better than other computer languages at reading one's mind!
- acobster 10y ago
  Generally doing "the right thing" based on a rough definition of "what we have and what we want" is not mind-reading, but it is essentially what Prolog does. The author seems to conflate AI with declarative programming here, and I think that was the point.
- jcranmer 10y ago
  Prolog, or at least logic programming languages, were thought to be the wave of the future of programming in the 80s. They, of course, turned out to be a massive bust.
  - yazr 10y ago
    You are factually correct But, a key problem was the runtime (in)efficiency of the prolog implementation engine. Maybe today with x1M more powerful computers it is worth trying again ? Neural networks were abandoned in the 80s and made a massive comeback with a. more compute power b. convolutional networks In fact, maybe a ML-optimized VM will make higher-level languages useful again ;)
    - ScottBurson 10y ago
      No, the problem was not that Prolog was too slow. The problem was that it was too stupid. Prolog's strict left-to-right, depth-first search strategy allowed the implementations to be, in fact, quite fast. But that same strategy also kept it from being more than a small step toward true declarative programming, because Prolog programmers have to understand the order in which subgoals are tried. For example, consider a goal that can be satisfied by either of two clauses, which we'll call A and B, and where every problem instance can be solved by exactly one of these; however, clause A always completes (either succeeding or failing) in constant time, but clause B, if invoked on an instance that would cause it to fail, goes into infinite recursion (an easy thing to do by accident in Prolog). It's imperative (ha!) for the Prolog programmer to make sure that clause A is tried before B. This kind of thing comes up all the time in Prolog programming. On the one hand, you have to be aware of the order of operations so you don't cause infinite recursions (or, short of that, cause your program to take exponential time); on the other, Prolog programmers frequently exploit the left-to-right, depth-first strategy by using the operator known as the "cut". The cut is a non-logical operator in the sense that its semantics can be expressed only by reference to the Prolog search order. Programs written in a true declarative language might superficially look a lot like Prolog programs, but the implementation would need to be much smarter -- doing things like figuring out the correct order of operations on its own. (I think machine learning might actually turn out to help with this.) Anyway the problem with your suggestion that we just need to throw more hardware at Prolog should be evident by this point. It doesn't matter how fast your machine is if your algorithm is exponential in the typical case (never mind if it contains an infinite recursion!).

nemaar 10y ago

In a sense we already do this. For example, every time you create a website in HTML/CSS and javascript, you use high level instructions and a lot of declarative stuff. It is already very far away from the actual hardware. If you use some framework with builtin templates, all you need is the actual content and some plumbing. Everything else is handled by the lower layers and it is much more declarative than you realize. It also means that you do not specify thousands of things/little details and you trust the lower layers to do the right thing. It is already happening and it will only get better. People freak out every time when this idea appears and the recurring argument is that "the AI will not do the right thing, it is too stupid". They forget that our frameworks/libraries and complicated software stacks already work as dumb AIs. They follow hardcoded rules and try to please us and most of the time it actually works otherwise we would not use libraries. Using a neural network may not be the right choice for this problem but the general idea is correct. A good library should hide the "how" as much as it can and only require the user to specify the "end goal".

chris_va 10y ago

I suspect we'll see supervised seq2seq generated code first. Like: --- Programmer inputs on left: compute std dev of x please AI on right proposes edit to code: + import numpy as np ... + stddev_x = np.std(x) --- Not super complicated to start with, but you can see where it will go from there.

fizx 10y ago
How in the world do you train that? Where's the corpus of english <-> code mappings come from?
- sapphireblue 10y ago
  The corpus can be acquired from companies that host coding competitions. In this paper http://arxiv.org/pdf/1510.07211v1.pdf http://arxiv.org/pdf/1510.07211v1.pdf (I already mentioned it many times, but it is very relevant) researchers took the data from a similar source >To accomplish this goal, we leverage a dataset from a pedagogical programming online judge (OJ) system,2 intended for the undergraduate course, Introduction to Computing. The OJ system comprises different programming problems. Students submit their source code to a specific problem, and the OJ system judges its validity automatically (via running). And trained seq2seq model on it on character level and it almost works, barring a few typos. IMHO this is an underappreciated breakthrough. With more data and a better model generating competition-grade programs seems possible.

sapphireblue 10y ago

The author proposes a lot of vague ideas in this article (for example "I believe one of the biggest problems is the use of Error Propagation and Gradient Descent") without references or any solid proofs why they are necessary to solve the proposed program (Automate programming using ML?). In fact there is already a lot of solid work just on this subject: * Learning algorithms from examples http://arxiv.org/abs/1511.07275 http://arxiv.org/abs/1511.07275 https://arxiv.org/abs/1410.5401 https://arxiv.org/abs/1410.5401 * Generating source code from natural language description http://arxiv.org/abs/1510.07211 http://arxiv.org/abs/1510.07211 * And, the most closest work to what author probably wants, a way to write a program in forth while leaving some functions as neural blackboxes to be learned from examples: http://arxiv.org/abs/1605.06640 http://arxiv.org/abs/1605.06640 * Also there is a whole research program by nothing less than Facebook AI Research that explicitly aims at creating a conversational AI agent that is able to translate user's natural language orders into programs (asking to the user additional questions if necessary): http://arxiv.org/abs/1511.08130 http://arxiv.org/abs/1511.08130 (there is also a summary here http://colinraffel.com/wiki/a_roadmap_towards_machine_intelligence http://colinraffel.com/wiki/a_roadmap_towards_machine_intell... ) And deepmind is also working on conversational agents: https://youtu.be/vQXAsdMa_8A?t=1265 https://youtu.be/vQXAsdMa_8A?t=1265 Given current success of such models, automating simple programming tasks maybe not as much research as engineering and scaling up problem. There is a lot of exciting machine learning research out there nowadays. Almost all of this research is available for free from papers posted on arxiv. It is a really good idea to read more about state of the art before coming with new ideas.

bjenik 10y ago
The forth black box paper seems to be quite close actually. The big difference is that I would like to go for higher level tasks. The low level variant - having code missing a few lines/functions and filling those is indeed quite well researched. More interesting would be the opposite: Having blocks of code (called components in my text) and letting the net find a good way to use them. Normally the way a programmer would build something like let's say the amazon website is sketching up a few ideas how it could look - cutting it into small parts and then writing (or reusing) the code for the small parts. A lot of the "using AI for programming" papers (AI in this case being anything from nets to logic stuff) focus on building these small parts - I'm more interested in the levels above that. The "generating code from natural language/dialogs" also kind of misses my idea as it still assumes a "programmer" to tell it what to do (in the amazon example "display the products on a grid" or "make the buy button bigger"). The programmer would in this case either guess the right thing ("everybody has their products in a grid") or run something like A/B testing against some metric ("items sold" or "$ revenue") and then go back to the conversational programming tool and tell it "change the button". Why not skip the programmer and give the metric directly to the net? The thing I would like to get working is telling the net "here is a ton of different buttons, grids, lists and a lot of other UI stuff, and there are all our products - I want to maximize revenue - do whatever you like to get there". Regarding the vague ideas: I believe (also totally unsubstantiated - though research in this area would be quite interesting) that writing down ideas first and then looking into them (either by reading other papers or doing own research) is far better than the other way round, because it has a higher probability of leading to something new, because you don't follow assumptions and errors others potentially made which could lead to a dead end. Of course there is a certain trade off, because if you try to reinvent everything you will not get to the point of something new.

cardigan 10y ago

Starts off with an interesting idea (programming is perfectly specified versus being underspecified but more efficient due to shared context) but then devolves significantly: neural networks are not the be all end all of smart computers. Interesting part is that a smart computer could resolve ambiguities in a human description of program desired in a way similar to humans. Missing part to this idea is that unless it's perfect, it would need a way to explain choices and results and probably some dialog system to iteratively improve upon how it resolved ambiguities. E.g, make me an app like Uber but for cats -> here is a version you can play with -> oh I don't really like the cat icon, can we change it -> sure, which of these would you like ; etc etc Computer requires ever changing human context, and maybe could have individualized context. So it has to learn over time. The point is to maximize efficiency of getting things made: how little needs to be specified before computer gives you something you accept / how fast can you go from some vague idea in your head to working acceptable software. You should at least be able to get as good as an arbitrarily large number of people who lived a very similar life to yours and were put in a time bubble where you could talk with them to have them write software for you, assuming they're organized very well and highly motivated and great programmers and all that.

brett40324 10y ago

I feel dumber after reading this article. So much, that my own human net of neurons can't figure out or explain why. A partial explanation is that I dont like the assumption that once work is being performed at higher and higher abstracted layers, that machine hardware, OS kernels, compilers and linkers, package and dependency managers, and all other systems software supporting high level programming paradigms (including AI based methods) are going to fall in line at the application layer.

runeks 10y ago
I think the quality on Hacker News would benefit from more people being able to downvote. At least take away people's right to upvote if they have no right to downvote. It seems to me there's missing a balance if we can't downvote a story that turns out to be lower quality than what it appeared to be.

jcoffland 10y ago

neural networks != AI

yazr 10y ago
"How ML based programming will allow us to program using a higher-abstraction programming language" Better ?! There is also the famous quote about how any working (and understood) AI is no longer considered AI...
- jcoffland 10y ago
  That quote will apply up until we encounter real AI.

iammyIP 10y ago

Since the c language is already a human readable abstraction away from the actual hardware instructions, which then again are abstractions on what actually happens on the chip with the transistors and electrons, you could argue that programming in c is already pretty much 'talking to the computer human-style'. As others said the use of frameworks and even higher level scripting languages with an ide is already a dumb-ai kind of human-friendly conversation with the machine. A problem with driving this further down the road like the article suggests could be, that if i wanted to talk to the machine like i talk with other persons, the natural language used in this case migh not be very well suited to express what i actually want. I would need special definitions and terminology, and in some cases a general description would not suffice, so the need for precision in expression might stay. The actual dialogue with the machine then might last as along as a programmer would have needed to program it. I could see how this might work for very simple programs (like for some uber-app which is not much more than a wrapper for a telephone call to the taxi central), but inherent complexity that other programs need cannot be simplified away by switching the language to a conversational chit chat level.

ww520 10y ago

It's kind of vague. It would be better if he used a concrete example to illustrate how AI help programming.

bjenik 10y ago
Actually the examples in the text are quite concrete - the few lines would be everything you'd have to write to get such a site working (modulo a bit of http plumbing). Obviously we are nowhere near such a thing. The rest of the text acts as a bit of a pointer what could help to get there. From my other comment (in this case the example was an amazon like shopping site): > The thing I would like to get working is telling the net "here is a ton of different buttons, grids, lists and a lot of other UI stuff, and there are all our products - I want to maximize revenue - do whatever you like to get there". The way of telling a neural network these things would be these few lines of code which would hook up different parts to get it working.
- brett40324 10y ago
  You're gonna have to make a clear distinction between web dev project configuration, and AI for making good choices that optimize meeting sales and marketing goals. How is your 'project codebase markup' any different from filling in some radio buttons on a template generator or a CMS?

tiagoespinha 10y ago

Interesting idea, but to be honest I think it will never take off in a large scale. If you want to write a vague and generic piece of code that can figure out by itself what it is that its outputs ought to be with regards to its inputs, you are, with effect, creating artificial intelligence. This, in turn, requires teaching. What if suddenly you have new pages on your website? Your AI program thingy wouldn't be able to serve them until you taught it what these pages are and in what situations the end user might be interested in viewing them. Imagine having a 5 year old kid handling your shop's cashier. And now a new product arrived and you have to explain to him/her that there's this new product, what it's called, the price and which types of people are allowed to buy it (e.g., alcohol couldn't be sold to people younger than X years old). If you're really teaching your application with natural language, like you would with a five year old, then the effort it'd take to get that info into your system would defeat the purpose of actually using a computer to do it (computers are good at storing and looking up stuff in large data sets, better than humans for really large data sets). This whole hype of machine learning is suitable for situations where you have large amounts of data and you want them crunched according to some basic pre-established and non-changing logic without relying on actual human labor. If the logic evolves, you'll always require humans to sit at the helm and steer the right and wrong. My 2 cents.

orasis 10y ago
I disagree with your conclusion. In my app, 7 Second Meditation, I am constantly authoring new content. It uses improve.ai to show that new content to some small number of users and learn how they react to that content. The learning happens from the users themselves. In my particular case, within about 300-500 views the system can tell if a new piece of content is total crap and will basically stop showing it. If the content is amazing, within 100-200 views it has a strong idea and will immediately prioritize this content. The results speak for themselves, before I started using machine learning I was getting 4 star ratings and usage was flat. Now I'm getting 5 star ratings and the app is growing. Will it ever take off at large scale? I have no clue, but I've seen such strong benefits from the machine learning that I'm now working on developing improve.ai full time.
- 10y ago

orasis 10y ago

My project, improve.ai, is a first concrete step forward in this direction. We start by replacing if/then statements with a decide() function and some goal events. I've used this approach in my app, 7 Second Meditation, to achieve a solid 5 star rating across 248 reviews and 40% first month retention. https://itunes.apple.com/us/app/7-second-meditation-daily/id667197203?mt=8 https://itunes.apple.com/us/app/7-second-meditation-daily/id... I'll be writing some articles deep diving into 7 Second Meditation and breaking down how I'm achieving this results. For now, here is a taste: https://blog.improve.ai/hi-world-how-to-write-code-that-learns-efdb8b5af940#.1upmk4vxo https://blog.improve.ai/hi-world-how-to-write-code-that-lear...

teddyknox 10y ago

Why do we need to jump to ML-based programming again? I'm confident that as we build simpler interfaces and workflows for replacing the most modular components in our programs with AI, we'll begin to see which components are the next lowest-hanging fruit and what concrete ML problems need to be solved to model them. We imagine a future where AI becomes a dominant paradigm for "writing" software -- I think that will be the case, but not in the way everyone suspects. I think 80% of the new value of software in the future will be derived from AI components, but that 80% of the production costs will still go into the structural glue code that supports the value-powerhouse models. Thus, most of the software written in the future will look similar to most of the code written today. I also suspect that as the complexity of AI models increases, the structural code required to support these models will keep pace in complexity. For this reason, I see a future where the unit cost of high-quality software does not see any big drop, but where the value of this software continues to increase exponentially. A corollary to this is that valuable software will be no easier to program in the future than it is today.

ThePhysicist 10y ago

Very intersting ideas! I also think that in the future we will see a lot more declarative programming languages that model high-level concepts of a given program, while low-level "plumbing" code will be generated automatically. I'm not yet sure about the role that AI will play in this process though, and I think there are many things that we have to solve before we'll see something that can write generic programs. Program synthesis is a very active field of research and many AI-based methods have been proposed in the last decades, IMHO what most systems lack is the applicability to real-world programming languages and use cases though. Having worked on static program analysis, I know that even our current ability to understand and reason about existing programs is still very limited. A main reason for this is that most real-world systems are composed of many parts that are not easily specifiable under a single paradigm (e.g. templates, database code, configuration files). To build a usable AI-based programming system, we will need: * A description languge that is able to model ALL aspects of a given real-world system under a single paradigm * A system to analyze and understand the artefacts produced by the above system * A way to generate real-world code from the specification above, including a way to "fill in the blanks" that the user did not specify (as leaving out the details the whole point of such a system) * A way to test the generated code against the specifications provided by the user and further "reasonable" assumptions, which will be needed as the specifications by the user will not be complete, see above * A way to guide this process towards a reasonable program through user feedback within a reasonably small number of steps While none of these things are impossible, implementing them is a significant challenge with a lot of unknowns for which we don't have good solutions yet. I therefore think the first AI-based systems which we'll see in the coming years will be limited to specific problem domains (e.g. data analysis, logic programming) for which we can more easily build a system as the above.

jcranmer 10y ago

The keyword you want to search for is "program synthesis." It already exists, and no neural nets need apply (neural nets tend to be useful only if you can't get anything else to work). As a bonus, it's not probabilistic like AI techniques tend to be but exact, based on SMT solvers and verified proof correctness. An example of work showcased at this year's PLDI that's capable of doing this sort of stuff: Fast synthesis of fast collections (https://homes.cs.washington.edu/~mernst/pubs/collection-synthesis-pldi2016-abstract.html https://homes.cs.washington.edu/~mernst/pubs/collection-synt...): specify a database-like data structure for a collection of objects and the list of queries performed on that data structure, and get out code that's as fast as hand-tuned data structures with fewer bugs. Programmatic and Direct Manipulation, Together at Last (http://dl.acm.org/citation.cfm?id=2908103&dl=ACM&coll=DL&CFID=609587099&CFTOKEN=40441624 http://dl.acm.org/citation.cfm?id=2908103&dl=ACM&coll=DL&CFI...): Take an image generated by code (e.g., periodic stripes), and be able to manipulate that image by drag-and-drop, e.g., changing stripe size or period. Stratified synthesis: automatically learning the x86-64 instruction set (https://stefanheule.com/publications/pldi16-strata/ https://stefanheule.com/publications/pldi16-strata/): 60% of the x86-64 instruction set can be formally specified starting from about 60 base or pseudo-instructions (basically, describing each instruction as an assembly program of simpler instructions).

YeGoblynQueenne 10y ago
Nice, thanks- particularly the x86-64 instruction set learning. I would kind of put that down as grammar induction rather than program synthesis though.

viach 10y ago

Interesting, how AI based project management could work? It is much simpler to implement and could really save lots of money for organizations, imo

YeGoblynQueenne 10y ago

>> I believe a lot of research today is limited by first looking at, and becoming an expert in, the status quo and then building small iterative improvements. It would be better to first find a goal and then looking for a way of getting there - at least that's the way we got to the moon. Nope.

brett40324 10y ago

#3 on HN for the moment is very relevant to some of the discussion in this thread pertaining to websites. https://news.ycombinator.com/item?id=12198572 https://news.ycombinator.com/item?id=12198572

BucketSort 10y ago

I've been toying around with the idea of making user agents that simulate the functions a user must be able to accomplish through a software and using that as a cost function for some evolutionary algorithm.

pennyhill 10y ago

The workout then could to start with get started with a single neuron relating to the feedback and also the production and dynamically removing and adding them right up until being nearly some tolerance of effectiveness. A completely new part - once again with a single neuron - could then be generated choosing yet another metric. Maybe something like if one layer is a lot bigger than both of its neighbors. During the long-term I hope it can be much better to depart from the part construction. http://www.crazysales.com.au/ http://www.crazysales.com.au/ Particularly when moving in direction of fully commited hardware as an alternative to float-crunching on GPUs. Aside from that, a number of the more advanced themes might not exactly be also conceivable with a tidy coating system.