9 ms·

Concrete AI tasks for forecasting (2014)

88 points by apsec112 10y ago

Buttons840 10y ago
After we've achieved all these things, we can start asking when "general intelligence" will arrive.
- rogaha 10y ago
  Do you mean self consciousness machines?
  - KellhusSmellhus 10y ago
    AGI is a pretty hard to define term, but I never saw it defined as "self-conscious". Most definitions are along the lines of "do what a human does", i.e. walk into a random house and make coffee or get a college degree, or, more mathematically, "Average a good score over all possible problems, weighted by their likelyhood and the compactness of your code"
- nnq 10y ago
  Most humans are pretty bad at at least a few of those things. Also, some are tasks that the human brain is specifically optimized for - human natural language most likely plays well to lots of human-hardware-design-quircks. So using them a test sounds pretty bad / unfair - lots of human-level but independently evolved alien intelligences would probably fail miserably at them. You could have above-human intelligence barely reaching human performance at either of these tasks but vastly surpassing human intelligence at others... Imho GI "will arrive" after the point that we have swarms of similar close-to-human-level ai-agents trying to model each other's behavior while competing and collaborating in artificial and natural environments. All theories of self-consciousness that make some sense imho are about agents trying to model self-similar agents and then somewhow reframing this ability to model past and future self. So you could have self-conscious dog-level agents (yeah, I believe "self conscience" is quantitative and a dog can have some of it and a human a different level of it, not 0 or 1), or non-self conscious above-human-"general"-AIs (though I don't know if you'd want to label "general" any non-self conscious AI, and I think any non-self-conscious above-human AI would get "self conscience" quite fast if you were to push it be be "general purpose" - striving to solve more and more types of problems, you inevitably arrive to some formulation of the "model something like yourself in order to predict its actions" problem).
  - hacker_9 10y ago
    Hardware plays a massive part too; I don't see a static micro processor becoming conscious anytime soon. Our brains though are literally physical neural networks, that grow, move, and change physically all the time.
    - nnq 10y ago
      It does not. Any kind of software can run on any kind of turing complete hardware. Unless you're one of those believing in the quantum mind theory https://en.wikipedia.org/wiki/Quantum_mind https://en.wikipedia.org/wiki/Quantum_mind . Just as a human brain can simulate a microprocessor (although unusably slow and with errors), so can a microprocessor simulate a human brain (also unusable slow). Read a bit more on turing equivalence - a badly explained topic unfortunately, because you know how bad are mathematicians at explaining stuff, even if imho it should be part of junior school curricula in a simplified version. (Oh, and maybe it's brushed off because it can also lead to pretty trippy conclusions: take a cockroach brain which is likely turing complete, mount it as a cpu in some kind of computer, stuck in a maigc time-machine box that makes time run order of magnitude faster, add some really fancy input/output tech, and you could run a human mind on it... think of what the reincarnation weirdoes would milk from this :) ) I'm pretty sure that if you find a way to extract a mathematical model of a human's personality, and a way to transform it to reduce "chemical noise quircks" and other hardware dependent but unnecessary "features"/bugs, you could run it just fine on a big datacenter and have a working immortal human mind. (I'm also pessimistic enough to believe that we'll solve the "general AI" problem before solving the "mind extraction" one unfortunately. Hopefully our creations don't destroy us completely before solving problem (2) and we have time to pass on some direct immortal legacy of what it means to be "human"...) P.S. but yeah, I agree that engineering better suited hardware might be actually easier than getting the current type of hardware fast enough or than inventing the mathematical transforms needed to run a mind's set of equations on current type hardware... we're slowly getting to custom hardware for ai ( https://cloudplatform.googleblog.com/2016/05/Google-supercharges-machine-learning-tasks-with-custom-chip.html https://cloudplatform.googleblog.com/2016/05/Google-supercha... ), but I imagine the future will be more analog/clockless...
- daveguy 10y ago
  I think several of these will be post general intelligence. Especially the one(s) that require a simple explanation of reasoning, the generalized learning from an example or two and contextual understanding of natural language.
blazespin 10y ago
Some are pretty silly unfortunately: "Write a novel or short story good enough to make it to the New York Times best-seller list."
- sanxiyn 10y ago
  What's silly about it? If AI could write (for example) Fifty Shades of Grey, it would be super impressive.
gallerdude 10y ago
Some of these are pretty awesome/crazy. I hope to see many achieved in the near future.
- wonko1 10y ago
  Yes and to be honest I wouldn't go so far as to call a lot of these "Concrete" tasks. But it is an interesting and throught provoking list. And makes me consider more quantitive metrics for qualatitive things like how good a piece of music is.
Kartificial 10y ago
Number 31 ("Play poker well enough to win the World Series of Poker.") is a pretty empty statement, as 1 tournament does not reflect your skill at poker at all. Short term variance is too big of a factor when it comes to one 1 tournament result.
- dmichulke 10y ago
  Well, in 2016 there were 6737 entrants [0], so I suppose the AI needs to have all its stars aligned or some other connection to god to win it without being good. Biggest problems are usually not winning heads-up but meaningfully sampling the state space with more than a handful of players + accounting for the "no limit" part. [0] https://en.wikipedia.org/wiki/World_Series_of_Poker#Main_Event https://en.wikipedia.org/wiki/World_Series_of_Poker#Main_Eve...
amelius 10y ago
> Translate a text written in a newly discovered language into English as well as a team of human experts, using a single other document in both languages (like a Rosetta stone). Huh? We can't even translate texts correctly between mainstream languages yet.
- Houshalter 10y ago
  There is some interesting work applying word2vec on the voynich manuscript. It found all sorts of patterns that humans weren't aware of. It's theoretically possible even without strong AI.
- d_theorist 10y ago
  It's a bit confusing, but I don't think this list is supposed to be a list of tasks we can do now. I think it's a list of well-defined (ish) tasks that are milestones to look out for in the future. Or something like that anyway. It's not well explained.
Chestofdraw 10y ago
I'd really love to see - 'Beat tier one teams at least 50% of the time in Dota 2 across multiple patches' It's cool that people are starting to work on Starcraft 2 AIs but the game is kind of dead so competition at the top isn't going to be as strong as it is in more popular games. Also I think Dota presents a much more interesting and more difficult challenge for AI makers.
- t_fatus 10y ago
  The challenge is different: you've got way more units to control in SC2 than in Dota. And rushes can be punitive, which do not occur as frequently in Dota
  - Chestofdraw 10y ago
    The more units thing is part of the reason I think Dota is a better challenge, I can't help but feel a SC2 AI could use inhuman micro as a crutch whereas in Dota mechanical / micro skill doesn't provide quite as big an advantage. Not too sure what you mean by 'rushes can be punitive' but Dota certainly has a variety of early game aggression strats which tend to be high risk / reward and are seen fairly regularly.
reacweb 10y ago
IMHO the wording of point 12 is a silly way to pretend that Alphago has not completely achieved its goal. Human players learn Go using manuals that compile the experiences of millions of games. The fact that Alphago can use directly huge collections of games is not very relevant for its success.
- reacweb 10y ago
  BTW, the point 32 is very inspiring. Could an AI imagine experiences to create models to explain the physic of our world. An AI to reinvent QM and relativity from scratch ? Maybe reconcile both ?
- afro88 10y ago
  There's a difference between needing 100,000,000 games and 50,000 for training to be the best. That difference is learning efficiency and astuteness - a good way of judging AI progress, no?
  - daveguy 10y ago
    I think "50,000 games played" puts it a bit too simply. How many games has he watched how many equivalents of a game has he visualized as sequences of moves? If "played" is over 50k I'd be willing to bet "considered" is over 1,000,000.
- conjectures 10y ago
  > The fact that Alphago can use directly huge collections of games is not very relevant for its success. On the contrary, it is, and the _need_ for such quantities of data is one of the biggest challenges in neural networks. How about 50,000 games + access to manuals ;)
- Almaviva 10y ago
  We know that AlphaGo uses a Monte Carlo tree search, and presumably contains innovations and refinements to techniques applicable to turn based board games with perfect information and a clear binary win condition. We also know it uses a Deep Learning algorithm to imitate play from the best humans. And presumably, being stronger than any human now, it could also recursively refine and train on its own games. Do we know how important the Deep Learning component is? Would AlphaGo be just as strong or nearly so without that part? I know that over the last few years chess engines have become dramatically more powerful, even on the same hardware, to the tune of hundreds of Elo points. They still stick to the classical techniques though (Alpha-Beta pruning, minimax is still the core). This is not to understate the refinements, but the general ideas are (I think?) only applicable to turn based games with finite moves and a binary mathematical criterion for victory. Is it possible that the improvements AlphaGo has made are mostly of this type, and the Deep Learning parts are not that important to its actual play strength? Or can we totally rule this out? Is there an in depth discussion of this question somewhere?
LeifCarrotson 10y ago
"relatively well specified" If these are "well specified", then the author and my boss are operating on some definition unknown to me. Also, quite a few tasks don't seem to be obviously about AI. Stuff like: > Play poker well enough to win the World Series of Poker. It's one thing to play Starcraft or Go well. But implementing this is either mostly a question of luck, or an incredibly lifelike robot with incredible facial recognition skills. Very different tasks! > Beat the fastest human runners in a 5 kilometer race through city streets using a bipedal robot body. This requires no AI, just a lot of improvement in energy density and actuator technology.
- hacker_9 10y ago
  "It's one thing to play Starcraft or Go well. But implementing this is either mostly a question of luck, or an incredibly lifelike robot with incredible facial recognition skills." In terms of strategy though, it is the next level up I'd say. Videogames come first, then more difficult games such as Poker come later where you need to be able to tell a bluff from the truth. Much more subtle. "This requires no AI, just a lot of improvement in energy density and actuator technology." It did say city streets though, which makes me think it would need to strategise and be able to: 1. Adjust speed and take corners close such as what you see in F1 races. 2. Switch between endurance and sprinting when necessary to maintain energy for the whole race. 3. Run behind others to stay in the tailwind and reduce energy cost. All require human level sensory input and a strategising component.
  - LeifCarrotson 10y ago
    At human 5k speeds, on asphalt, taking corners close doesn't require speed adjustment. Traction is not really a problem. Just take a direct route. Drag is quadratic with respect to speed, so switching between endurance and sprinting is sub-optimal. Just maintain pace. And at these speeds and the required clearances for safety, the benefit from drafting is less than 5%. This is huge for runners pushing the limits of human performance, but if a robot has the same limits where this matters, it will probably be by design.
  - chriswarbo 10y ago
itp 10y ago
Why is this tagged 2014? The article is dated Dec. 14, 2016 and includes a reference to AlphaGo.
- gumby 10y ago
  Because someone didn't understand the ISO-8601 date format. 2016-12-14 => 14 December 2016
  - maverick_iceman 10y ago
    Is dang reading this?
hacker_9 10y ago
"Take a written passage and output a recording that can’t be distinguished from a voice actor, by an expert listener." Role Playing Games would explode if this happened. The ability to generate speech on the fly would create dramatically more immersive worlds.
- mlboss 10y ago
  https://developer.amazon.com/alexa-voice-service https://developer.amazon.com/alexa-voice-service
nl 10y ago
This is a pretty good list. I see some criticism of the poker tasks, but that is unfounded. Poker is an important area of AI research because of bluffing (detection and when to do it). CMU is probably the most active in this area. See https://www.cmu.edu/news/stories/archives/2016/february/poker-bot.html https://www.cmu.edu/news/stories/archives/2016/february/poke... and http://www.cs.cmu.edu/~noamb/poker.html http://www.cs.cmu.edu/~noamb/poker.html. Their bot (Claudico) was demoed at NIPS15. Interestingly, I've seen solid progress on 3, 6, and 8 in the last 2 months of 2016 and on 7 in 2017 already.
maverick_iceman 10y ago
Imagine the implications for the real world if you can solve #32. I wonder will it be able to detect emergent laws (e.g. thermodynamics, Newtonian gravity as an approximation to general relativity etc.)? Also it will be very interesting to see which experiments AI chooses to perform.
saurabh20n 10y ago
#22 - "system should write code that sorts a list, rather than just being able to sort lists." Using symbolic techniques, synthesizing algorithms is tractable. See Sec5.3 on pg11- of http://saurabh-srivastava.com/pubs/popl10-synthesis.pdf http://saurabh-srivastava.com/pubs/popl10-synthesis.pdf - Figure 2(b) is the synthesized selection sort algorithm, and Table 3 has QuickSort synthesis time at 160seconds. Very old work (2010) and used templates to guide synthesis. SAT solvers have improved, and much more CPU cycles are available, so in 2016, exhaustively exploring the template space will be trivial. I.e., rather than reconstructing code from concrete program execution traces, giving the synthesizer (AI?) the ability to manipulate symbolic spaces leads to a tractable search space. Finding specification-matching programs within this symbolic space is doable.

88 points by apsec112 10y ago

Buttons840 10y ago

After we've achieved all these things, we can start asking when "general intelligence" will arrive.

rogaha 10y ago
Do you mean self consciousness machines?
- KellhusSmellhus 10y ago
  AGI is a pretty hard to define term, but I never saw it defined as "self-conscious". Most definitions are along the lines of "do what a human does", i.e. walk into a random house and make coffee or get a college degree, or, more mathematically, "Average a good score over all possible problems, weighted by their likelyhood and the compactness of your code"
nnq 10y ago
Most humans are pretty bad at at least a few of those things. Also, some are tasks that the human brain is specifically optimized for - human natural language most likely plays well to lots of human-hardware-design-quircks. So using them a test sounds pretty bad / unfair - lots of human-level but independently evolved alien intelligences would probably fail miserably at them. You could have above-human intelligence barely reaching human performance at either of these tasks but vastly surpassing human intelligence at others... Imho GI "will arrive" after the point that we have swarms of similar close-to-human-level ai-agents trying to model each other's behavior while competing and collaborating in artificial and natural environments. All theories of self-consciousness that make some sense imho are about agents trying to model self-similar agents and then somewhow reframing this ability to model past and future self. So you could have self-conscious dog-level agents (yeah, I believe "self conscience" is quantitative and a dog can have some of it and a human a different level of it, not 0 or 1), or non-self conscious above-human-"general"-AIs (though I don't know if you'd want to label "general" any non-self conscious AI, and I think any non-self-conscious above-human AI would get "self conscience" quite fast if you were to push it be be "general purpose" - striving to solve more and more types of problems, you inevitably arrive to some formulation of the "model something like yourself in order to predict its actions" problem).
- hacker_9 10y ago
  Hardware plays a massive part too; I don't see a static micro processor becoming conscious anytime soon. Our brains though are literally physical neural networks, that grow, move, and change physically all the time.
  - nnq 10y ago
    It does not. Any kind of software can run on any kind of turing complete hardware. Unless you're one of those believing in the quantum mind theory https://en.wikipedia.org/wiki/Quantum_mind https://en.wikipedia.org/wiki/Quantum_mind . Just as a human brain can simulate a microprocessor (although unusably slow and with errors), so can a microprocessor simulate a human brain (also unusable slow). Read a bit more on turing equivalence - a badly explained topic unfortunately, because you know how bad are mathematicians at explaining stuff, even if imho it should be part of junior school curricula in a simplified version. (Oh, and maybe it's brushed off because it can also lead to pretty trippy conclusions: take a cockroach brain which is likely turing complete, mount it as a cpu in some kind of computer, stuck in a maigc time-machine box that makes time run order of magnitude faster, add some really fancy input/output tech, and you could run a human mind on it... think of what the reincarnation weirdoes would milk from this :) ) I'm pretty sure that if you find a way to extract a mathematical model of a human's personality, and a way to transform it to reduce "chemical noise quircks" and other hardware dependent but unnecessary "features"/bugs, you could run it just fine on a big datacenter and have a working immortal human mind. (I'm also pessimistic enough to believe that we'll solve the "general AI" problem before solving the "mind extraction" one unfortunately. Hopefully our creations don't destroy us completely before solving problem (2) and we have time to pass on some direct immortal legacy of what it means to be "human"...) P.S. but yeah, I agree that engineering better suited hardware might be actually easier than getting the current type of hardware fast enough or than inventing the mathematical transforms needed to run a mind's set of equations on current type hardware... we're slowly getting to custom hardware for ai ( https://cloudplatform.googleblog.com/2016/05/Google-supercharges-machine-learning-tasks-with-custom-chip.html https://cloudplatform.googleblog.com/2016/05/Google-supercha... ), but I imagine the future will be more analog/clockless...
daveguy 10y ago
I think several of these will be post general intelligence. Especially the one(s) that require a simple explanation of reasoning, the generalized learning from an example or two and contextual understanding of natural language.

blazespin 10y ago

Some are pretty silly unfortunately: "Write a novel or short story good enough to make it to the New York Times best-seller list."

sanxiyn 10y ago
What's silly about it? If AI could write (for example) Fifty Shades of Grey, it would be super impressive.

gallerdude 10y ago

Some of these are pretty awesome/crazy. I hope to see many achieved in the near future.

wonko1 10y ago
Yes and to be honest I wouldn't go so far as to call a lot of these "Concrete" tasks. But it is an interesting and throught provoking list. And makes me consider more quantitive metrics for qualatitive things like how good a piece of music is.

Kartificial 10y ago

Number 31 ("Play poker well enough to win the World Series of Poker.") is a pretty empty statement, as 1 tournament does not reflect your skill at poker at all. Short term variance is too big of a factor when it comes to one 1 tournament result.

dmichulke 10y ago
Well, in 2016 there were 6737 entrants [0], so I suppose the AI needs to have all its stars aligned or some other connection to god to win it without being good. Biggest problems are usually not winning heads-up but meaningfully sampling the state space with more than a handful of players + accounting for the "no limit" part. [0] https://en.wikipedia.org/wiki/World_Series_of_Poker#Main_Event https://en.wikipedia.org/wiki/World_Series_of_Poker#Main_Eve...

amelius 10y ago

> Translate a text written in a newly discovered language into English as well as a team of human experts, using a single other document in both languages (like a Rosetta stone). Huh? We can't even translate texts correctly between mainstream languages yet.

Houshalter 10y ago
There is some interesting work applying word2vec on the voynich manuscript. It found all sorts of patterns that humans weren't aware of. It's theoretically possible even without strong AI.
d_theorist 10y ago
It's a bit confusing, but I don't think this list is supposed to be a list of tasks we can do now. I think it's a list of well-defined (ish) tasks that are milestones to look out for in the future. Or something like that anyway. It's not well explained.

Chestofdraw 10y ago

I'd really love to see - 'Beat tier one teams at least 50% of the time in Dota 2 across multiple patches' It's cool that people are starting to work on Starcraft 2 AIs but the game is kind of dead so competition at the top isn't going to be as strong as it is in more popular games. Also I think Dota presents a much more interesting and more difficult challenge for AI makers.

t_fatus 10y ago
The challenge is different: you've got way more units to control in SC2 than in Dota. And rushes can be punitive, which do not occur as frequently in Dota
- Chestofdraw 10y ago
  The more units thing is part of the reason I think Dota is a better challenge, I can't help but feel a SC2 AI could use inhuman micro as a crutch whereas in Dota mechanical / micro skill doesn't provide quite as big an advantage. Not too sure what you mean by 'rushes can be punitive' but Dota certainly has a variety of early game aggression strats which tend to be high risk / reward and are seen fairly regularly.

reacweb 10y ago

IMHO the wording of point 12 is a silly way to pretend that Alphago has not completely achieved its goal. Human players learn Go using manuals that compile the experiences of millions of games. The fact that Alphago can use directly huge collections of games is not very relevant for its success.

reacweb 10y ago
BTW, the point 32 is very inspiring. Could an AI imagine experiences to create models to explain the physic of our world. An AI to reinvent QM and relativity from scratch ? Maybe reconcile both ?
afro88 10y ago
There's a difference between needing 100,000,000 games and 50,000 for training to be the best. That difference is learning efficiency and astuteness - a good way of judging AI progress, no?
- daveguy 10y ago
  I think "50,000 games played" puts it a bit too simply. How many games has he watched how many equivalents of a game has he visualized as sequences of moves? If "played" is over 50k I'd be willing to bet "considered" is over 1,000,000.
conjectures 10y ago
> The fact that Alphago can use directly huge collections of games is not very relevant for its success. On the contrary, it is, and the _need_ for such quantities of data is one of the biggest challenges in neural networks. How about 50,000 games + access to manuals ;)
Almaviva 10y ago
We know that AlphaGo uses a Monte Carlo tree search, and presumably contains innovations and refinements to techniques applicable to turn based board games with perfect information and a clear binary win condition. We also know it uses a Deep Learning algorithm to imitate play from the best humans. And presumably, being stronger than any human now, it could also recursively refine and train on its own games. Do we know how important the Deep Learning component is? Would AlphaGo be just as strong or nearly so without that part? I know that over the last few years chess engines have become dramatically more powerful, even on the same hardware, to the tune of hundreds of Elo points. They still stick to the classical techniques though (Alpha-Beta pruning, minimax is still the core). This is not to understate the refinements, but the general ideas are (I think?) only applicable to turn based games with finite moves and a binary mathematical criterion for victory. Is it possible that the improvements AlphaGo has made are mostly of this type, and the Deep Learning parts are not that important to its actual play strength? Or can we totally rule this out? Is there an in depth discussion of this question somewhere?

LeifCarrotson 10y ago

"relatively well specified" If these are "well specified", then the author and my boss are operating on some definition unknown to me. Also, quite a few tasks don't seem to be obviously about AI. Stuff like: > Play poker well enough to win the World Series of Poker. It's one thing to play Starcraft or Go well. But implementing this is either mostly a question of luck, or an incredibly lifelike robot with incredible facial recognition skills. Very different tasks! > Beat the fastest human runners in a 5 kilometer race through city streets using a bipedal robot body. This requires no AI, just a lot of improvement in energy density and actuator technology.

hacker_9 10y ago
"It's one thing to play Starcraft or Go well. But implementing this is either mostly a question of luck, or an incredibly lifelike robot with incredible facial recognition skills." In terms of strategy though, it is the next level up I'd say. Videogames come first, then more difficult games such as Poker come later where you need to be able to tell a bluff from the truth. Much more subtle. "This requires no AI, just a lot of improvement in energy density and actuator technology." It did say city streets though, which makes me think it would need to strategise and be able to: 1. Adjust speed and take corners close such as what you see in F1 races. 2. Switch between endurance and sprinting when necessary to maintain energy for the whole race. 3. Run behind others to stay in the tailwind and reduce energy cost. All require human level sensory input and a strategising component.
- LeifCarrotson 10y ago
  At human 5k speeds, on asphalt, taking corners close doesn't require speed adjustment. Traction is not really a problem. Just take a direct route. Drag is quadratic with respect to speed, so switching between endurance and sprinting is sub-optimal. Just maintain pace. And at these speeds and the required clearances for safety, the benefit from drafting is less than 5%. This is huge for runners pushing the limits of human performance, but if a robot has the same limits where this matters, it will probably be by design.
- chriswarbo 10y ago

itp 10y ago

Why is this tagged 2014? The article is dated Dec. 14, 2016 and includes a reference to AlphaGo.

gumby 10y ago
Because someone didn't understand the ISO-8601 date format. 2016-12-14 => 14 December 2016
- maverick_iceman 10y ago
  Is dang reading this?

hacker_9 10y ago

"Take a written passage and output a recording that can’t be distinguished from a voice actor, by an expert listener." Role Playing Games would explode if this happened. The ability to generate speech on the fly would create dramatically more immersive worlds.

mlboss 10y ago
https://developer.amazon.com/alexa-voice-service https://developer.amazon.com/alexa-voice-service

nl 10y ago

This is a pretty good list. I see some criticism of the poker tasks, but that is unfounded. Poker is an important area of AI research because of bluffing (detection and when to do it). CMU is probably the most active in this area. See https://www.cmu.edu/news/stories/archives/2016/february/poker-bot.html https://www.cmu.edu/news/stories/archives/2016/february/poke... and http://www.cs.cmu.edu/~noamb/poker.html http://www.cs.cmu.edu/~noamb/poker.html. Their bot (Claudico) was demoed at NIPS15. Interestingly, I've seen solid progress on 3, 6, and 8 in the last 2 months of 2016 and on 7 in 2017 already.

maverick_iceman 10y ago

Imagine the implications for the real world if you can solve #32. I wonder will it be able to detect emergent laws (e.g. thermodynamics, Newtonian gravity as an approximation to general relativity etc.)? Also it will be very interesting to see which experiments AI chooses to perform.

saurabh20n 10y ago

#22 - "system should write code that sorts a list, rather than just being able to sort lists." Using symbolic techniques, synthesizing algorithms is tractable. See Sec5.3 on pg11- of http://saurabh-srivastava.com/pubs/popl10-synthesis.pdf http://saurabh-srivastava.com/pubs/popl10-synthesis.pdf - Figure 2(b) is the synthesized selection sort algorithm, and Table 3 has QuickSort synthesis time at 160seconds. Very old work (2010) and used templates to guide synthesis. SAT solvers have improved, and much more CPU cycles are available, so in 2016, exhaustively exploring the template space will be trivial. I.e., rather than reconstructing code from concrete program execution traces, giving the synthesizer (AI?) the ability to manipulate symbolic spaces leads to a tractable search space. Finding specification-matching programs within this symbolic space is doable.