5 ms·
Can programming languages boost each other via instruction tuning?
- netbioserror 3y agoSo the equivalent of transfer learning. Neat, but I'd like to see it forced into a new paradigm or syntax. How would it handle Haskell or Clojure?
- SomewhatLikely 3y agoThey do show training on HTML is still leading to improvements across languages. It looks to me like training on C led to the biggest average improvement among the tested languages. I'm curious if the effect translates to humans as well. I know that for myself learning C taught me a lot about what the higher languages are abstracting away.
- pipo234 3y ago> More surprisingly, we found that CodeM-HTML 7B trained on the HTML corpus can improve Java by an absolute 15.24% pass@1. (Tongue in cheek reaction:) Any sufficiently advanced technology is indistinguishable from magic — Arthur C. Clarke
- xeckr 3y agoCompared to 17% when pre-trained on Python. Looks like the debate on whether or not HTML is a programming language is settled!
- codetrotter 3y agoWas the HTML corpus it trained on only HTML and text, or was there JS code in the HTML?
- sp332 3y agoThere sure was. Here is a snippet from the very first example in the html.json file: <input type="text" id="searchQuery" name="searchQuery" placeholder="Enter name..."> <button onclick="searchEmployees()">Search</button> </div> <script> function searchEmployees() { var input, filter, table, tr, td, i, txtValue; ...etc... There's no code in the repo, but the paper says this about how the inputs were generated: For all the selected languages except HTML, we adopt an in-depth evolution, by asking GPT-3.5 to rewrite the seed instruction (Python) into a more complicated version relevant to the target language (Python, JavaScript, TypeScript, C, C++, Java, or Go). However, for HTML, we adopt in-breadth evolution to produce a brand-new HTML-related instruction...
- catskul2 3y agoSince this is on arxiv one presumes it's not been peer reviewed yet, right? Is there a way to leave comments? There are a couple of confusing language/grammar mistakes including some in the first sentence.
- behnamoh 3y agoIf there's anything I've learnt in the past year about LLMs, it's this: there's so much hype, ungrounded research "findings", and next-big-thing libraries (remember langchain?) LLMs, at least the ones companies use, are APIs all the way down. Stay away from anything or anyone who claims they've discovered something no one else knew about LLMs.
- aalimov_ 3y agoMaybe staying away from someone selling something while making that claim, but staying curious seems appropriate. > LLMs, at least the ones companies use, are APIs all the way down. Really does seem like it. So many interesting little projects that end up being nothing more than a hidden prompt and an api call to OpenAI
- scott_s 3y agoEmail the first author.
- mcphage 3y agoEven the first sentence of the summary is unclear: > When human programmers have mastered a programming language, it would be easier when they learn a new programming language. I think they're saying "When human programmers have mastered a programming language, it makes it easier for them to learn a new programming language", but... I'm not entirely sure.
- toast0 3y agoYeah, that's what they're saying. English is not an easy language. "It would be easier to learn a programming language, when you have mastered any other programming language" is clear, IMHO, but flipping the clauses makes it less clear.
- leetharris 3y agoThis is true with natural language as well in LLMs, ASRs, and more. At Rev we found that adding data from other languages to primarily-English ASR models helped reduce WER in the target language substantially (especially when accents were involved). We found similar benefits when training translation models.
- wcoenen 3y agoIt was already known that training on code improves performance on all reasoning tasks[1], so this is not really surprising. [1] https://arxiv.org/abs/2210.07128 https://arxiv.org/abs/2210.07128
- deleted 3y ago[deleted]
- habitue 3y agoI don't think it can be dismissed that easily. You could imagine that most of the benefit from being trained on programming languages comes from learning any language at all, but a 15% gain from one language to another is pretty surprising I'd say.
- air7 3y agoWow, this is the first contradiction to Betteridge's law that I've encountered... https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headlines https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headline...
- lainga 3y agoSort of. None of the programming languages mentioned in the paper were actually boosted in any way, an LLM was.
- pmontra 3y agoBetteridge's Law can be trivially confuted. For example one could make a blog post titled "Did the Sun Raise Yesterday?" I don't dare trying the law with a headline like that about tomorrow though.
- AtlasBarfed 3y agoI've just realized that the most ultimate "no good deed goes unpunished" is that the mountain of open source code released by programmers will be used by LLMs to put them out of jobs.
- csjh 3y agoFortunately we’ll get revolutionary new products from it, such as calendar app (with AI integration), email app (with AI integration), resume builder app (with AI integration), and more!
- sumtechguy 3y agooh no and a wave of patents with 'using ai' it will be 1999 all over again.
- deleted 3y ago[deleted]
- munificent 3y agoAnd the mountain of concept art created by digital artists and uploaded to the web is putting them out of jobs thanks to DALL·E and friends. And the mountain of writing created by authors and uploaded to the web is putting them out of jobs. And the mountain of music created by musicians is going to start putting them out of jobs. Generative AI is an unprecedented transfer of wealth-generating capability from creative people to giant corporations.
- bcrosby95 3y agoAnd it will occur faster than any transition in history. Meatspace has natural limits to how fast things can do this.
- staunton 3y agoBut, contrary to at any such developments in history, the people who are losing out aren't the "poor working class". They already have cultural and political power that they stand to lose. This might still go a lot of ways.
- cmdrk 3y agois it the programming languages themselves, or the programmer comments within the training sets that improve output?
- sp332 3y agoThere are comments in the data, but 95% of the comments in the HTML one seem to be variations of /*CSS styles go here*/.
- deleted 3y ago[deleted]
- Sparkyte 3y agoNot even, you just need to comprehend the syntax and understand the caveats of a particular language. Once a person has the fundamentals almost all languages are their oyster.