> Rather, it's Inductive Programming and more specifically Inductive Logic Programming (ILP), which is learning programs from examples, i.e. "incomplete specifi

> Rather, it's Inductive Programming and more specifically Inductive Logic Programming (ILP), which is learning programs from examples, i.e. "incomplete specifications". I'm not familiar with the General Program Synthesis Benchmark Suite, but the problem you list (test three strings are ordered by length) is trivial for ILP approaches. The General Program Synthesis Benchmark Suite works from input-output examples, not “complete, formal specifications”. How would you tackle this with ILP? > However, I have to say that even so, if something is a difficult problem for program synthesis approaches, then it's very unlikely that neural networks will do any better at it. For instance, do you know how well deep neural nets perform on this benchmark? I'm not aware of any serious at-scale attempts. Your option is basically to try few-shot with GPT-3. OTOH, learning these trivial programs from 100 examples is a largely artificial framing used to support a field which hadn't worked its way up to meaningful problems, and in the more general sense, large networks are promising; eg. the GitHub-trained GPT: https://www.youtube.com/watch?v=y5-wzgIySb4 https://www.youtube.com/watch?v=y5-wzgIySb4 or any of the GPT-3 programming demos: https://twitter.com/sharifshameem/status/1284103765218299904 https://twitter.com/sharifshameem/status/1284103765218299904 https://twitter.com/sharifshameem/status/1284815412949991425 https://twitter.com/sharifshameem/status/1284815412949991425 https://www.reddit.com/r/commandline/comments/jl8jyr/the_nlc2cmd_challenge_site_has_a_gpt3powered/gaop9ji/ https://www.reddit.com/r/commandline/comments/jl8jyr/the_nlc... > These approaches are still state of the art for their respective tasks and there is no other approach that has been shown to do any better, including deep neural networks. In what sense are they "no longer more than briefly and tangentially relevant" as you say? “if you're interested in learning AI” These techniques were invented from the field of AI, but that does not mean they remain in the field of AI. > You clearly have a strong opinion on GOFAI and the AI winter of the '80s, but what knowledge does this opinion come from? Can you say? I can argue why ML approaches are good and promising and point at that. I can argue why ML approaches make conceptual sense whereas GOFAI does not, though I don't see us resolving that short-term so I'd rather not. But what I can't so easily do is point to the non-existence of GOFAI AI successes. It's just not there. You do have tools Watson and WolframAlpha which use GOFAI techniques for fact search over a large set of human-built knowledge repositories (trivia q's / math tools), but Watson is mostly considered a stunt, and I'm not aware of anyone calling WolframAlpha AI. > the ability for reasoning (despite big claims to the contrary) The nebulousness of the term ‘reasoning’ is pulling a lot of weight here. It's clearly doing sophisticated computations of some sort, beyond brute memorization. > or arithmetic (ditto) http://gptprompts.wikidot.com/logic:math#toc6 http://gptprompts.wikidot.com/logic:math#toc6 There are more examples too, this is just addressing the one point people get wrong most often. BPEs are an interim performance hack, not an indictment on the approach in general. > or generation of novel programs Is clearly false. > For instance, the append() example you show above is clearly memorised: you haven't given the model any examples of append(), so it can't possibly learn its definition from examples. This is true, but it's mostly just an artifact of me having to prompt it through FitnessAI. Unlike smaller models, few-shot learning works, it just takes more space than I have to prompt with. See the GitHub-trained example for something that integrates with more arbitrary code. There are many other examples, like the database prompt below (all bold is human input), or see some of the examples I linked above. https://www.gwern.net/GPT-3#the-database-prompt https://www.gwern.net/GPT-3#the-database-prompt Or I can ask Q: “If z(str) = str + " " + str + " z" (for example, z("dumbell") = "dumbell dumbell z"), and g(str) = "k " + str + " j" then what is g("run")?” A: “g("run") = "k run j"” (The inverse problem doesn't work so well, giving “g(str) = "k run j"” for one example (valid but vapid) and “g(str) = "k str j"” for two (close but no banana), and confusion for more complex prompts, though I suspect the format is partially to blame. I can list other failure cases. But my point isn't that GPT-3 is reliable here; it's a language model.) > btw, why do you need to give it the list "a"? What happens if this is ommitted from the prompt? That example was from me trying to emulate an example I saw on Twitter I've since lost, which was a similar thing but multi-step, where each step GPT-3 returned all three lists, modified or queried per the given commands. Omitting `a`, I get Q: “b = ["lifting", "curls", "squats"], c = ["running", "jogging"], so what is b after b.append("pushups")?” A: “lifting,curls,squats,pushups” I had to change the prompt a bit because initially the result was truncated (FitnessAI is not made for this), or said “b.append("pushups") will add the string "pushups" to the end of b.”, which is correct but not what I wanted. Few-shot would fix formatting inconsistencies; right now the model is just guessing.