4 ms·
Nature paper: https://www.science.org/doi/10.1126/science.abq1158 https://www.science.org/doi/10.1126/science.abq1158 For those wondering, yes, these are the
by YeGoblynQueenne 4y ago
Nature paper:
https://www.science.org/doi/10.1126/science.abq1158 https://www.science.org/doi/10.1126/science.abq1158
For those wondering, yes, these are the same results reported in the February 2022 pre-print here:
https://arxiv.org/abs/2203.07814 https://arxiv.org/abs/2203.07814
As in the preprint the authors report their system's average ranking as top 54.3% in past competitions but the meat and potatoes of their results are in the reporting of test accuracy, which they don't advertise in the abstract- because it's not that good. From the body of the Nature article:
>> With up to 100,000 samples per problem (10@100K), the AlphaCode 41B model solved 29.6% of problems in the CodeContests test set.
So the best test set performance they got was 29.6% with the 10@100k metric. See also Figure 3 in the Nature paper (graphically presenting results listed in Appendix Table A2 in the preprint).
"10@100k" means that their LLM generated millions of programs for each exercise, of which 100,000 (100k) were selected by filtering and clustering and various other heuristics, and of those 100k, 10 were selected to submit as the system's solution.
So 10@100k means to take a few million guesses, then take another 100k guesses, and finally andother 10. And still only get it right 30% ish percent of the time.
This may be enough to rank in the 54% of CodeForces participants, for a system fine-tuned on CodeForces-like data (the CodeContests dataset developed by DeepMind specifically for this task). But it's not enough to claim that AlphaCode "conquers coding, performing as well as humans", per the title of TFA.
- hajile 4y agoNow imagine what it would do with the actual, poorly specified projects most programmers are subjected to rather than these basic problems with fixed outcomes that have easily quantifiable metrics of success.
- Dopameaner 4y agoI would argue that poorly specified requirements will be solved by an Auto-coder that can produce the end-result in a very short time. Allowing the PM, to see how bad their requirements were and make it more elaborate.
- SoftTalker 4y agoWhen your requiremets are so detailed and comprehensive that the generated program is correct -- haven't you basically written the program?
- philipkglass 4y agoIt really depends on the program. Some problems derive most of their complexity from trying to correctly model complex real world rules, like calculating pension payments for employees that were hired at different times, for different positions, under different union agreements, having worked varying numbers of years. Some problems have short, clear definitions yet remain challenging. An algorithm for solving them may have been discovered decades (or even centuries) ago. But trying to produce more efficient solutions can be a life's work and more. One of the most obvious examples is integer factorization, which had a basic algorithm published 800 years ago yet is still a research problem today: https://en.wikipedia.org/wiki/Trial_division https://en.wikipedia.org/wiki/Trial_division https://en.wikipedia.org/wiki/Integer_factorization#Factoring_algorithms https://en.wikipedia.org/wiki/Integer_factorization#Factorin...
- aerovistae 4y agoThis tangentially reminds me of a very interesting idea I think I encountered in a Borges story - for a map to be useful, it must be inaccurate. That is to say, a perfectly accurate map would just be a replica of the the terrain it is mapping - anything less would technically be inaccurate. Only by leaving things out and "summarizing" an area does a map become a useful representation of the terrain.
- mrexroad 4y ago“The map is not the territory”… until the map is the territory?
- berniedurfee 4y agoThe was (still is?) one of the major issues with outsourcing development work. External contractors tend to need excruciatingly detailed requirements to produce the expected results, because they typically lack the domain knowledge to guess correctly when faced with ambiguity. Very often this means getting into long cycles of iteration. If time zones are distant, those cycles can be very long and painful. Perhaps outsourcing to AI will work much better, as those iterative cycles can be done in near real-time. Maybe there needs to be many different models, each with some industry/domain specific training to increase the odds it’ll make the right guess when it needs to.
- dragonwriter 4y agoI mean, coding contests tend not only to be well-specified, but also extremely micro tasks. A whole lot of the work of software development is taking even reasonably well-specified business requirements and determing the appropriate structure of tasks at that level to meet them. Coding contest problems may be a common hiring filter, but they are very much not representative of software development work (OTOH, AI coding assistance, which this, Copilot, ChatGPT, etc., illustrate) are going to render them less valid as hiring filters, both because they will make them harder to use as skill tests, and because they will further reduce the role of the micro-focus skills they center in real-world software development.
- visarga 4y agochatGPT seems amazing at taking corrections, iterating on the requirements, fixing as it goes. So the badly-specified requests are going to be sorted out by multiple rounds of interaction.
- Widdershin 4y agoTry to get chatgpt to solve a basic math problem using BEDMAS instead of PEMDAS, even with multiple interactions and reminders it still gets incredibly confused.
- wcoenen 4y agoChatGPT cannot correctly perform step by step operations within strict formal systems. On the other hand, it can provide a high level sketch of how tomathematically prove something[1]. I think this is amazing, because this is the opposite of how we normally think about computers. ChatGPT is highly intuitive instead of a formal reasoning system. No doubt some future AIs will combine this intuition with checks against formal verification engines to check correctness. [1] Section 6 of https://yaofu.notion.site/How-does-GPT-Obtain-its-Ability-Tracing-Emergent-Abilities-of-Language-Models-to-their-Sources-b9a57ac0fcf74f30a1ab9e3e36fa1dc1 https://yaofu.notion.site/How-does-GPT-Obtain-its-Ability-Tr...
- sgerenser 4y ago
- smaddox 4y agoAgreed. This is overblown. but they haven't even begun to apply RLHF (or at least they haven't published anything). I think it's safe to say that combining an LLM with RL, all existing techniques, would result in a much stronger AI competitive coder.
- eloff 4y agoI think what this means is AI assistance during interviews may be enough to fool the interviewer. This may finally put an end to remote leet code interviews. Companies with deep pockets and local companies will probably stick to the status quo and just move to in person interviews again.
- visarga 4y agoI mean, why not embrace the future? Allow people to use AI in interviews, schools and exams. Everyone will use it at work, why not test AI management skills together with problem solving skills. Maybe the best skills of 2019 are not the best skills to have in 2023.
- asvitkine 4y agoBecause coding interviews are only a proxy for what someone needs to do on the job. Presumably, it will become a worse proxy if assisted by AI but other tasks they'll do on the job aren't. For example, debugging a subtle bug in a large codebase. Maybe one day AI can do that too, but that doesn't seem to be the focus here.
- canadianfella 4y ago
- ElevenLathe 4y agoExactly. Interviewing a post-LLM programmer without the aid of an LLM is like doing a running test for a bike messenger job.
- qualudeheart 4y agoDon’t need deep pockets. You don’t have to fly someone out to interview on site. Just have him pay for it himself.
- eloff 4y ago
- visarga 4y agoBy the next year they will have generated 100M new problems and solutions tested to pass verification and retrain the model, making it 10x smarter. This approach creates new training data and we can iterate on the coding model multiple times.
- deleted 4y ago[deleted]
- YeGoblynQueenne 4y agoThe problem with that approach is that you end up with a system that is very good at solving programming problems of a particular kind, but does not generalise very well to arbitrary programming tasks. For example, their system will not have very good performance on, I don't know, leetcode (if it had, we'd have heard all about it). So I don't doubt that we'll see better performance on Codecontests/ Codeforces in the future, perhaps some time next year (just to leave some time after the Nature publication). But not anything of substance. Or they'll just come up with a new dataset that their system performs well on. To be honest, I'm scandalised that such a poorly thought-out approach got a publication in Nature. If I had tried to publish something like that in my field ("we made a firehose that spams code willy-nilly and by heuristic this and that got it to hit the right target 30% of the time! And we beat some humans at it!") I would have been shot down in flames, probably also ruined my reputation for good. But, I guess, it's DeepMind publishing in Nature so it's effectively a racket. One just shrugs and moves on.
- gwn7 4y agoTotally agree. The title is misleading and a clickbait. Here's another article which imo is the best evaluation of the technology I've seen so far: https://news.ycombinator.com/item?id=33997603 https://news.ycombinator.com/item?id=33997603 Yet, it has virtually no upvotes and no discussion. Well, evangelism sells better than criticism.