54 ms·
Deepmind’s AlphaCode conquers coding, performing as well as humans
- thinkmcfly 4y agoSo, can it make a traveling salesman solver for current quantum computers?
- tromp 4y agoIt probably can, but it won't run in polynomial time.
- thinkmcfly 4y agoShucks
- kybernetyk 4y agoIn competitive programming that is.
- autotune 4y agoThe sooner we can fully automate competitive programming away from the standard interview process the sooner things will improve for all software devs, imho.
- Jabrov 4y agoWhy is that? Can you explain what is wrong with the current interview process and how that would improve things for devs?
- xwolfi 4y agoMaybe they would focus on asking us about test strategy, devops culture, release deadline commitment, quality vs deadline cutoff, business understanding, team integration ability, all the sort of things we actually do 90% of the time.
- throwawaysleep 4y agoRun of the mill devs are not decision makers for the most part on any of this.
- scottLobster 4y agoUh, yeah they are. If I start a 60 hour (estimated) story the week before the sprint ends, I'm going to have some very pointed questions how thoroughly we want to test this new feature, and perhaps argue that it should be pushed into next sprint. Knowing how to formulate such issues to my management is a very typical part of my job. If you work at a place where devs are just code monkeys who implement the orders from on high with no feedback whatsoever or avenue for pushback... get a better job.
- Deestan 4y agoToo many of them are in effect a long-winded quiz for a memorization task. How would you swap two integers without using a temporary variable? Seen it before? Pass. Not seen it before? Fail.
- kadoban 4y agoYou can absolutely puzzle out _a_ solution to that problem, it's not just a quiz. It's not even hard, given the context that you know that it's possible to do. Steps, mostly driven by just basically knowing the goal and that there's not many operations that could possibly help: "a" "b" "a+b" "b" "a+b" "-a" "b" "-a" "b" "a" Then once you have that, you can enumerate the downsides to that, look for more efficient and less error-prone ways to proceed.
- Deestan 4y agoI'm sorry. That one uses a hidden temp storage for the intermediate calculations. We were looking for the solution based on XOR which works with only two registers. We wish you luck with your job search elsewhere.
- kadoban 4y agoThat's not what someone giving even this poor interview question would answer with. It's also not true. These interview questions are fairly shit, but they're not _that_ shit, and the people giving them are doing their best.
- hamilton987 4y ago[dead]
- winstonprivacy 4y agoWhat irks me about this question is that it is almost utterly unrelated to any type of development task, save one: pipelining a repeated math equation in a long loop. And even then, I would bet the time it takes to swap two registers (or even L1 cache) using a third would far outperform the three operations needed because of the fact that they are sequential in nature (ie: they cannot be performed simultaneously by even an advanced CPU). In nearly 30 years of diverse coding experience, I've never once encountered a situation where this solution would be useful.
- throwawaysleep 4y agoIt has nothing to do with the day to day job. You are just memorizing vast problem sets.
- jayd16 4y agoDoes not reflect on the job work to regurgitate algorithm answers and it's too easily gamed. A novice that has practiced algorithms will outperform an expert that has not studied recently. That said, its probably more used as a filter for people interested enough in working there to study.
- jstx1 4y agoThis seems like a flawed premise. It's an interview, there's nothing to be automated. It's a test, a task that the company wants you to do as a candidate, not a task that they need automated. They don't even need the task done, they aren't using the code you write to solve real world problems, they're using it to assess your suitability for the job. To put it another way - they can already automate solving their leetcode problems by looking up the solution in their database, no need for AI. But that's not the point at all.
- autotune 4y ago>They aren't using the code you write to solve real world problems, they're using it to assess your suitability for the job. If they aren't evaluating how the code you write fits into the context of a real world problem, how can they possibly use it to asses your suitability for the job? Using fake code problems to evaluate candidates is the flawed premise here.
- jstx1 4y agoYou're changing the topic - whether data structures and algorithms problems are suitable for interviews is separate question. My claim is that if you're using those types of interview questions, an AI model being able to solve them well makes no difference to your interview process because the interview isn't something that you want to automate. Like I said, they were always able to automate solving those problems by doing a database lookup.
- autotune 4y agoI am literally responding directly to what you wrote in the way that you wrote it. My counter claim to your initial point is that candidates will be able to use said AI in competitive programming interviews to fool the interviewer by the interviewee, thereby making competitive programming pointless as an interview tool. It automates it for the interviewee, not the interviewer.
- jstx1 4y ago
- kevmo314 4y agoPeter Norvig has written a good analysis with some interesting takeaways: https://github.com/norvig/pytudes/blob/main/ipynb/AlphaCode.ipynb https://github.com/norvig/pytudes/blob/main/ipynb/AlphaCode....
- emptybits 4y agoKevin Wang, competitive coder: "I also try to minimize the amount of code I write: each line of code is just another chance for a typo." Peter Norvig: This is why AlphaCode learned to write code with one-letter variable names, and with no comments or docstrings. !!
- Y_Y 4y agoTo be fair, unless you're doing something absurd the length of variable names shouldn't significantly affect the number of lines. Also comments with typos aren't really a problem. I don't like hard-to-read code either, but the more years I work the more I think that the shortest (within reason) code is the best code. That is to say, golfing is bad unless for fun, shorter code is usually better, the shitty hard to read code is probably a reflection of human laziness rather than misguided tersity.
- psychomugs 4y ago"If I had more time, I would have written shorter code"
- strofcon 4y agoHeh definitely an eyebrow-raising comment! On a second read of it though, I took it more as "because minimizing the code written is a pattern observed in the wild, AlphaCode mimics it as that's what it learned from." I might be terribly wrong though. :-D
- pfdietz 4y agoThis is why neural networks need to have pain buttons that humans can repeatedly press. Of course that's how we get killbots.
- borbulon 4y agoEven if it were bulletproof I don't think it's necessarily a bad thing, it would probably make the average engineer more productive if most of their job was architecture, and the writing of individual bits of code was something automated. It's a lot like the conversation not too long ago that AI was going to replace managers. Sure, some aspect of their work might be gone, but it didn't obviate the need for the human.
- claudiulodro 4y agoSince this sort of coding is exactly what FAANG interviews test for, maybe FAANG engineers will be some of the first coders automated out of a job. That would be an interesting irony.
- madspindel 4y agoYes, I believe this is something Sam Altman is predicting will happen.
- ohwellhere 4y agoI don't know Mr. Altman's rationale, but FAANG companies being first to automate away programmers sounds reasonable to me on the basis of scale alone. They have more code than most companies with which to make up a company- and problem-specific dataset, as well as the resources and expertise to train and deploy complex models.
- michpoch 4y agoOnly as long as you expect that FAANG engineers are solving FAANG interviews as a part of their daily job.
- andromeduck 4y agoYou never need algorithms until you really do.
- WrtCdEvrydy 4y agoThe algorithm tests are there to have an excuse to discriminate on other criteria. If you can mask your racial/age/gender reason for not hiring as a FAANG with "oh, we didn't like their algorithm solution" you always have an out.
- bluedevilzn 4y agoFAANG is the only place where you don’t need to be white to succeed. So, I have no idea what racial bias you’re talking about.
- wendyshu 4y agoIs this a new version of AlphaCode or just a new evaluation of it?
- erikpukinskis 4y agoThe most important thing I need from a programmer I work with is their ability to have a fruitful conversation with me about why the code might be written one way or another, and for them to iterate on the code in response to those conversations. If a coder can write code, but they can’t do that, they’re useless to the org. It will be faster for me to write their code myself than to maintain what they’ve done. So really that’s what I’d need from an AI coder. Writing the code is good, but can we talk about it and can you learn the specific architecture principles we have applie in this specific codebase.
- roflyear 4y ago99% of companies don't care about that. I would say 90% of companies you'll create friction trying to have those conversations "just get it done!"
- jchanimal 4y agoIt's true. For almost all of my career I've worked in the software equivalent of a lab building a diamond making robot. I did a stint at a consulting firm -- that was more like coal mining than manufacturing gemstones. By this I mean never ending surface area, and very little incentive to ever go back and refactor anything.
- londons_explore 4y agoFor most orgs, code isn't the end product. Code is the way to build the product to sell to the user. Therefore, in most orgs, code architecture/style is a distant secondary to 'does it achieve what the user will pay money for' and 'why isn't it finished yesterday?'.
- belter 4y ago"I'm sorry Dave...I can't do that"
- cdelsolar 4y agoChatGPT can actually do that sometimes. I've been using it as a rubber duck to help me debug code and it tells me what is wrong with what I wrote and how to fix it.
- swissdevgirl 4y agoIt seems preposterous, but DeepMind’s new coding AI just trounced roughly 50 percent of human coders in a highly competitive programming competition.
- agloeregrets 4y agoSo you're telling me it had requirements and expected results? My job is safe I see.
- JacobThreeThree 4y agoWait until the AI starts writing requirements and specifications docs.
- SketchySeaBeast 4y agoIt'll still be gibberish, but at least it's more likely it'll be grammatically correct.
- belter 4y agoCreated by ChatGPT just now... Title: Online Website for Chocolate Donuts Objective: The objective of this website is to provide a platform for customers to order and purchase chocolate donuts online. The website will offer a variety of chocolate donut flavors and allow customers to customize their orders by choosing toppings and packaging options. Functional Requirements: Customers should be able to create an account and log in to the website. Customers should be able to browse and view the available chocolate donut flavors, toppings, and packaging options. Customers should be able to add chocolate donuts and toppings to their cart and specify the quantity of each item. Customers should be able to view their cart and checkout to place their order. The website should process payment and provide confirmation of the order. Customers should be able to track the status of their order and receive updates on the expected delivery time. Non-Functional Requirements: The website should be secure and protect customer information. The website should be easy to use and navigate. The website should be accessible on a range of devices, including desktop computers, laptops, and mobile devices. The website should load quickly and be responsive to user actions. The website should have a clean and professional design. User Roles and Permissions: Customers: Customers will be able to create an account, log in, browse and order chocolate donuts, track their order status, and access their account information. Admin: The admin will be responsible for managing the website, including adding and removing products, processing orders, and managing customer information. User Interface Design: The website will have a clean and professional design, with a navigation menu at the top of the page and a search bar for customers to easily find specific products. The home page will feature a carousel of images showcasing the available chocolate donut flavors and toppings. The product pages will display images and descriptions of the chocolate donuts, along with options for toppings and packaging. The cart page will display the items in the customer's cart, along with the total price and a checkout button. The checkout page will allow customers to enter their shipping and payment information, review their order, and place their order. The order confirmation page will display the details of the customer's order and provide an estimated delivery time. The account page will allow customers to view and update their account information, including their shipping and payment details. Database Design: The website will have a database to store customer information, including their name, email address, shipping and payment details, and order history. The database will also store information on the available chocolate donut flavors, toppings, and packaging options. The database will keep track of the current orders and their status, including the expected delivery time. Technical Requirements: The website will be built using HTML, CSS, and JavaScript for the front-end, and a server-side language such as PHP or Ruby for the back-end. The website will use a database management system such as MySQL to store and manage the data. The website will be hosted on a web server and accessed through a domain name. The website will use a secure socket layer (SSL) to encrypt data and protect customer information. Testing: The website will be tested to ensure that it meets the functional and non-functional requirements. User acceptance testing will be performed to ensure that the website is easy to use and navigate. The website will be tested for compatibility on different devices
- commandlinefan 4y agoHeadline: "DeepMind’s AlphaCode Conquers Coding, Performing as Well as Humans" Actual text: "AI just trounced roughly 50 percent of human coders" I think we already all knew that about half of the coders out there weren't all that great.
- hgsgm 4y ago
- altgeek 4y agoThat maps rather well to an old routine from George Carlin "Think of how stupid the average person is, and realize half of them are stupider than that." https://youtu.be/AKN1Q5SjbeI?t=19 https://youtu.be/AKN1Q5SjbeI?t=19
- EGreg 4y agoThat’s what we said when AI beat chessplayers ranked below 1450. “Oh big deal” Now they never lose to any human in any chess game ever.
- bradleykingz 4y agoAnd yet humans still play chess
- Filligree 4y agoBut very few do so for money.
- chakintosh 4y ago> Rather than copying and pasting sections of previous training code, AlphaCode came up with clever snippets without copying large chunks of code or logic in its “reading material.” Gave me a good chuckle lol
- dandare 4y ago> When challenged with the CodeContest—the battle rap torment of competitive programming—the AI solved about 30 percent of the problems, while beating half the human competition. This was essentially a competition in the speed of programming. But if we want to discuss practical application, as in laying off armies of coders, we need to realise that there is a tremendous gap in the productivity of, let's say a solo startup founder and the productivity of a team of coders working for a multinational behemoth barely producing anything of a value over the whole sprint.
- taneq 4y agoJob's done, crew, time to go home. ;)
- andrekandre 4y agoproduct owner here, now that your free i have some more ideas i'd like you to work on...
- taneq 4y agoTell AlphaCodeZeroPrime, I’m on my wine break. slams window shut
- spaceman_2020 4y agoWe’re not far from coding becoming just a hobby or a sport - like chess.
- solumunus 4y agoWe're nowhere even remotely close to that. I have to assume that anyone reacting on this level are in the same group of people who 10 years ago believed we would have fully self driving cars by now.
- wittycardio 4y agoYup our current version of AI is great for cool demos and very bad for real world use cases.
- Workaccount2 4y agoIn 2014 there were AI experts on record saying that an AI beating a top GO champion was at least 15 years out.
- solumunus 4y agoWell, they were wrong. This isn't the same question though. The ML models that were trying to beat Go were on the correct path, it was just a matter of improvement. It was inevitable that they would beat the game. ChatGPT doesn't logically reason about code or business contexts. The developers of ChatGPT are not trying to do that, the developers don't know HOW to do that, not even slightly. Until they're on that pathway it's not even a question: ML that very efficiently copies and pastes from StackOverflow is simply not going to significantly replace programming jobs. The question of how long is left until the AI replaces programmers is completely irrelevant right now. Using Go/Chess analogy: > If the game is to replace effective human programmers then the AI is not even playing the game yet, let alone X years away from beating it.
- Workaccount2 4y agoThey beat the game hardly two years later, in early 2016. I'm sure every programmer has their finger crossed that the long tail is very long. And not that I think programmers will be out of jobs, probably not, but their value as a prompt technician will be substantially less than as a software engineer.
- nathias 4y agoI fear this will eventually shift a large chunk our time from coding to meeting.
- TheFattestNinja 4y agoY'all getting any coding time? Meme image to be insertrd here
- born-jre 4y agocan it do something like ARC challenge [0]? [0]: https://www.kaggle.com/competitions/abstraction-and-reasoning-challenge/overview https://www.kaggle.com/competitions/abstraction-and-reasonin...
- angarg12 4y agoI've conducted hundreds of FAANG-level coding interviews, and recently tried to run my interview questions through ChatGPT. The model was able to spit out a correct, optimal, clean Python solution effortlessly. Interestingly when I asked a follow up, a harder version of the same problem, ChatGPT spits out code that sort-of looks correct, but is actually nonsense. So, would it pass the interview? That's difficult to say. Interviews don't happen in a vacuum, you also consider the candidate's thought process, explanations, alternatives, tradeoffs... Still, I see this is a game changer for this kind of interviews. So long as candidates understand and explain the output code, there is a good chance they would clear the interview. Even if the code is incorrect, it might given them some hints towards the right solution. So where do we go from here? I always loathed this interview format, and these languages model reinforce it even further. Interview cheating has always been there, but it is generally so rare that it isn't a real concern. However these tools are too effective and easy to use. I can see a real divide between people who use them and people who doesn't. This type of interview might become even more useless at telling good coders apart. My take is we have 3 choices: a) Ignore it. b) Try to fight it. c) Embrace it. a) is not an option. b) would make an already pretty dreadful process even more intolerable. My money is on c) I can envision an interview format where we allow, or even encourage people to use ChatGPT and AlphaCode during the interview, much like you would use your IDE or a search engine. In fact seeing how a candidate understands and uses those code snippets can be a very interesting data point. Either that or scrap leetcode-style interviews altogether. P.S.: I was thinking about writing a blog post about this, if people think it'd be interesting.
- red_admiral 4y agoBy the time someone gets an on-premises interview, you'd notice if they're using a bot or not? Or are all the interview rounds remote these days?
- sgerenser 4y agoIn my experience (with FAANGs at least) the interviews have been virtual since the pandemic began. Haven't heard of any plans to return to return to flying out candidates for in-person interviews.
- carlsborg 4y agoWhy not post the blog post from the AlphoCode authors https://www.deepmind.com/blog/competitive-programming-with-alphacode https://www.deepmind.com/blog/competitive-programming-with-a... which walks through an example
- tuckerpo 4y agoWelp, back to school for EE I go.
- weatherlite 4y agoEE? I think plumbing and prostitution is all there will be left soon...and I'm really horrible with tools and not that good looking so unemployment it is...
- pmontra 4y agoHopefully this will kill the algorithmic coding interviews and will focus them on actually understanding the job to be done.
- Dopameaner 4y agoI spent extensive time on chatGPT. It does give an answer, but it isnt the most optimal, sometimes non-compiled version of them. Its suggestions havent reached the level of creativity. Example, if I give it a system design problem of a real-world problem. It suggests to use Apache Kafka, but what if Kafka isnt sufficient for whatever reason? E.g, Lai–Yang algorithm will give you no hits, Fascination observation, I asked for Project Euler #193 (a problem I solved using mobius function). chatGPT solved it using bruteforce, even after I asked for the most efficient way it could solve it with. It just used memorization, which wasnt enough to find an answer for that problem quick enough. I asked whether it could use the mobius function, it couldnt translate it to code, and I had to give it python code to make it work. - If you ask in-depth technical questions on how task queueing works within Elasticsearch, it wont be able to give you answer. - George Hotz in Lex friedmen interview mentions that GPT-3 has 100 most recent messages as a limit. He isnt convinced that is enough to completely build a complex tool like Solve FSD or implement me a kafka.
- alar44 4y ago-Lai Yang, pretty obscure, it doesn't have a Wikipedia page and I can't find much info on it anywhere, so this shouldn't be surprising. -Euler #193 - This problem is not super obvious and I see discussions on it invoking Riemann zeta. ChatGPT was able to create a correct Mobius function on its first try. I have no idea how this might be related to the problem though. -Elasticsearch - Why on earth would you expect it to know this? You've cherry picked some super niche stuff that this brand new AI can't do and name dropped some fancy theorems. We're all very impressed.
- samvher 4y agoI'm confused - how much has changed since February? The DeepMind blog post [0] seems to suggest not much. When I saw this headline on the front page I thought there were some major advances, again, but it seems not? [0] https://www.deepmind.com/blog/competitive-programming-with-alphacode https://www.deepmind.com/blog/competitive-programming-with-a...
- espadrine 4y agoNothing has changed. The publication process to the Science journal took time and was finally published on 8 Dec.[0] [0]: https://www.deepmind.com/blog/competitive-programming-with-alphacode https://www.deepmind.com/blog/competitive-programming-with-a...
- MichaelRazum 4y agoI really love do coding in python and always asked myself when the skill set will become legacy like COBOL. Not sure if you could compare it, but it seems that the next gen coders will say to the AI what to code rather than to code by themselves. A complete different skill set.
- lagrange77 4y agoI wouldn't say it's a completely different skill set. You still want to have people, who actually understand what a piece of code or program does. A magical black box to throw prompts at, might be nice for simple settings, but potentially can cause major fuck ups for complex systems.
- MichaelRazum 4y agoYeah sure, I just mean that it is like the step from COBOL -> JAVA -> Python in terms of productivity and the amount of code you write. So my point is, you would write much much less code. So maybe your future "code" would be just some sets of well placed comments and the AI does the rest. The same way you could test the system to prevent any major fuck ups. Sure you have to undestand it, but your core skill wouldn't be coding any more.
- cobertos 4y ago> the DeepMind team built a custom dataset from CodeContests from two previous datasets, with over 13,500 challenges. Each came with an explanation of the task at hand, and multiple potential solutions across multiple languages. The result is a massive library of training data tailored to the challenge at hand. Isn't this just over-fitting the model?
- SilverBirch 4y agoIsn't that exactly how the vast majority of engineers pass interviews today? They study up on the handful of algorithms that come up in interviews and then (with varying levels of acting) recite the answer to the question they knew they would get. What? Overfittings only bad in if you're a robot huh?
- Workaccount2 4y agoIt tickles me that organic neural nets that learn by viewing information are incredulous that neural nets could learn by viewing information.
- woojoo666 4y agoYeah the article mentions that they outperformed Codex, but was Codex specifically trained on code contests?
- krzat 4y agoI hope we will get some AI for static analysis. It would be nice to hover on a symbol and get high level description of what it does, what pattern it follows, etc. AI can access not only current state of the code, but it's entire history, which no human would have time for.
- dlkf 4y agoAnd the award for dumbest headline of the year goes to
- sys32768 4y agoThese AI advances are thrilling and a bit spooky. RPG games will include options for laws of karma or configuring an omniscient AI "god" who ingests all of your actions and develops consequences or new plots and dialogue. And of course the player can pray to the god for forgiveness, for blessings, etc.
- kulahan 4y agoThis is exactly what's been exciting to me about the future of games - AI will allow for a level of dynamism that simply was not possible before. It's seriously exciting stuff.
- sys32768 4y agoIt doesn't take much for us to believe someone is human, mostly confirming expected behaviors, doing small talk, etc. But AI NPCs will do all that and seem to have will and even conscience. I feel dizzy trying to think through the social and emotional ramifications of becoming seriously attached to NPCs who seem real and can articulate their feelings and share experiences with us.
- kulahan 4y agoWow, that second part is really nuts to think about, and I hadn't considered it before. You'll form connections based on personal conversations privy ONLY to you, then potentially watch that character die forever. If it's permanent, and not deterministic, you've quite literally lost that person forever.
- greenthrow 4y ago> the AI solved about 30 percent of the problems I think "conquers coding" may be overstating it.
- nwoli 4y agoAlgorithm code challenges is kind of a limited domain though (with solutions often really easy to beat average humans if you were allowed a lookup table of existing solutions). I wonder how well it does on open ended everyday challenges
- Havoc 4y ago>> It doesn’t have any built-in knowledge about computer code syntax or structure. It's pretty impressive that this naive approach works on something as complicated as coding. In hindsight it seems "obvious" that it would work in game-like contexts like chess/go. Programming feels like its a significantly higher level challenge in a way (not that go is trivial)
- polotics 4y ago"When pitted against over 5,000 human participants, the AI outperformed about 45 percent...". So 55 percent of humans outperformed the AI. How motivated were the humans in the sample? How many tests were run with these 5000 persons?
- highwaylights 4y agoBut when can it replace my stable of 10X coders and my three 100X react devs? (aka the Clydesdales).
- neilv 4y agoWhat's the analogue for cleaning the Clydesdale stable?
- highwaylights 4y agoFuture hires.
- foota 4y agoDepending on the pool of course, but 50th percentile is pretty poor in a programming challenge.
- amag 4y agoWe software engineers have to be the stupidest "smart" people on the planet. No other occupations work so hard to destroy entire businesses including our own. I get it, I'm a software engineer who loves automation. "But AI will just be a tool in our tool-set, software engineers will still be the system architects." Sure, for a while and then AI will do that too. "But eventually we will live in a fully automated world in abundance, wouldn't that be great?" Doing what? When we get there, anything we can consider doing, an AI can do faster and better. Write a poem? Write a book? Write music? Paint a picture? Life will be like a computer game with cheat-codes, whenever we struggle with something, instead of fighting on and improving we will turn to our universal cheat-engine: AI. Anecdotally, I did an analog mistake in my early twenties when I wrote a cheat-program for save-files. It worked like a typical cheat-engine, search the save-file for a specific value, go back to the game and change that value, go back to the save and search for the new value but only in those locations that had the original value. This is how I ruined "Heroes of Might and Magic II" :(. I used to love that game. I could spend hours playing it. Writing the cheat program was a lot of fun for a couple of hours but when it was done, there was no longer any reason for me to play the game. You might say that I didn't need to use my cheat program, but once the genie was out of the box it was too tempting to resist when I met some obstacle in the game. This is what I fear about AI making our jobs superfluous; it will also make our hobbies or anything we enjoy doing superfluous. Sorry for the bleak comment but this my fear and I feel the genie is already out of the box.
- nnoitra 4y ago
- dangond 4y agoA few months ago I spent three or four days putting together a website where I could store my cooking recipes in a nice, searchable format with some nice UI features that I wanted. Once AI is good enough, I'll be able to do the same without needing to brush up on how MongoDB works, or how to vertically center text in a div, or how to update the page's URL when navigating to a different recipe without triggering a full reload of the page. I couldn't care less about any of those things, I just wanted a cooking recipe website.
- Animats 4y agoSo when do we get a web site designer whee you write what you want and a web site is generated? That sort of thing is stylized and there's lots of training data available.
- YeGoblynQueenne 4y agoNature paper: https://www.science.org/doi/10.1126/science.abq1158 https://www.science.org/doi/10.1126/science.abq1158 For those wondering, yes, these are the same results reported in the February 2022 pre-print here: https://arxiv.org/abs/2203.07814 https://arxiv.org/abs/2203.07814 As in the preprint the authors report their system's average ranking as top 54.3% in past competitions but the meat and potatoes of their results are in the reporting of test accuracy, which they don't advertise in the abstract- because it's not that good. From the body of the Nature article: >> With up to 100,000 samples per problem (10@100K), the AlphaCode 41B model solved 29.6% of problems in the CodeContests test set. So the best test set performance they got was 29.6% with the 10@100k metric. See also Figure 3 in the Nature paper (graphically presenting results listed in Appendix Table A2 in the preprint). "10@100k" means that their LLM generated millions of programs for each exercise, of which 100,000 (100k) were selected by filtering and clustering and various other heuristics, and of those 100k, 10 were selected to submit as the system's solution. So 10@100k means to take a few million guesses, then take another 100k guesses, and finally andother 10. And still only get it right 30% ish percent of the time. This may be enough to rank in the 54% of CodeForces participants, for a system fine-tuned on CodeForces-like data (the CodeContests dataset developed by DeepMind specifically for this task). But it's not enough to claim that AlphaCode "conquers coding, performing as well as humans", per the title of TFA.
- hajile 4y agoNow imagine what it would do with the actual, poorly specified projects most programmers are subjected to rather than these basic problems with fixed outcomes that have easily quantifiable metrics of success.
- Dopameaner 4y agoI would argue that poorly specified requirements will be solved by an Auto-coder that can produce the end-result in a very short time. Allowing the PM, to see how bad their requirements were and make it more elaborate.
- 4y ago
- habibur 4y agoNext, the AI should be able to maintain and improve itself without human intervention. Until then, human are needed to build those AIs.
- henning 4y ago> Performing as Well as Humans > While not yet on the level of humans,
- rcpt 4y agoAlright everyone it's time to unionize
- q-base 4y agoI have made most of my money as a freelancer making weird integrations between legacy systems, migrating data that no one knew anything about, building or designing systems people weren't even able to formulate in any coherent way to me as a human and without having to change all the things they were certain they needed, when they found out that they really did not. Building the thing right vs. building the right thing. I am not saying that AI will not be able to revolutionize a lot of areas, but solving well known coding competitions are far removed from where a large portion of coders and technical people make their money.
- vladcodes 4y agoA Tesla also drives as well as humans on a straight California highway.
- nulld3v 4y ago> https://alphacode.deepmind.com/#layer=18,problem=137,heads=11111111111 https://alphacode.deepmind.com/#layer=18,problem=137,heads=1... > > Here AlphaCode could not come up with an algorithm to solve the problem [snip] > > But then AlphaCode behaves a bit like a desperate human, and hardcodes the answer for the example case to pass it even though its solution is wrong, hoping that it works in all other cases but just not on the example. Humans do this as well, and such hope is almost always wrong - as it is in this case. It behaves just like me...
- keepquestioning 4y agoHorrible news!
- zh3 4y agoI'm old enough to remmber the furore when calculators came on the scene ("Calculators Conquer Arithmetic, Performing better than humans"). Having played with various AIs, it feels similar - impressive when there's a known solution to look up (probabilistically guessed by the AI in this case) but flummoxed as soon as novel analysis is neccessary.
- Konohamaru 4y agoThere's this paradox in AI known since the 80s (I think it's Moraevac's Paradox?) that the abilities we acquired latest in our evolutionary history (e.g. chess, language, algorithmic thinking) are the easiest for AI to automate. But the tasks that are deep in our evolutionary history (e.g. object recognition, navigating the environment, etc...) are extremely difficult. Some AI speculators, using Gödelesque reasoning, posit that computer programming will be the last job to be automated by AI, so it can bootstrap itself. In reality, it will be one of the first jobs automated by AI, while jobs like cook, barista, etc... will be much later automated, if ever.
- Der_Einzige 4y agoI just sat in a Burmese restaurant with an automated barista. Korea has been long ahead of us on automating food service too. Automatic 24/7 drink making machine hands are common in Seoul.
- Konohamaru 4y agoTasks that involve manipulating physical objects to maintain biological homeostasis (e.g. cook) are hyper-optimized by biological evolution. It is impossible for engineers to engineer a unit that performs better with the same energy restrictions humans have.
- Der_Einzige 4y agoGood thing fusion actually is just 20 years away after all this time!
- mannykannot 4y ago"AlphaCode tackles the problem by generating over a million potential solutions for a single problem—multitudes larger than previous AI attempts. "As a sanity check and to narrow the results down, the AI runs candidate solves through simple test cases. It then clusters similar ones so it nails down just one from each cluster to submit to the challenge." The big question for me is, where did the test cases come from? They seem to be contributing much more to the outcome here than just "a sanity check and to narrow the results down." If they were effectively spelled out as part of the challenge, then this is an impressive result extending the trajectory of other recent achievements, but not so far as to justify the title's "conquers coding." If they were not, then this would seem to be taking things to a whole new level.
- btilly 4y agoI am dubious about the stated result. Language models have been very good at spitting out chunks of text verbatim that were in their training data. Buried in the github training data will belots of examples where people post their solutions to fun problems..including past code competitions. How often is it managing to match a description of a repository and spitting out correct code that cribbed heavily from it? That is, instead of figuring out the problem, it is pattern matching to a solution that someone else already figured out? How would we know?
- antipotoad 4y agoI think the much better blog post directly from DeepMind might answer your questions [0], but suffice it to say, the model does seem to be solving novel problems. [0]: https://www.deepmind.com/blog/competitive-programming-with-alphacode https://www.deepmind.com/blog/competitive-programming-with-a...
- paxys 4y agoPeople who think that tools like these will end software engineering as a career are the same ones who buy into the "10x rockstar programmer" myth and think that the job description entails sitting in a room and writing code for 12 hours a day and nothing else. A software engineer's job will be the same as it always was – to translate unclear and always-changing requirements into something that works. Until ChatGPT, Deepmind or whoever else can learn to deal with my client or product manager, my job is safe. In fact AI making programmers more productive is a good thing. It is only going to increase the problem space and we'll need yet more engineers to fill it, like all other productivity advancements that came before.
- gorjusborg 4y agoWe just need to replace the product manager and client with AI too ;)
- paxys 4y agoWell we can replace everyone with AI and not even need to exist at all. Until that happens, we'll still need good old fashioned managers and software.
- spaceman_2020 4y agoAgreed, but the question is: how many other other programmers do you need on your team now? How much of their work will be taken over by our new AI magic boxes? You have to see this from the perspective of non-tech companies, who see tech as a cost-center, not a profit or innovation center. Does your regional grocery store that just needs some tools that can help it track inventory really care whether its code comes from a team of big brained humans or two programmers in a basement copy-pasting chatGPT answers?
- paxys 4y agoYou can say the same thing for any other advancement in software engineering. Did good IDE tooling reduce the number of programmers? Explosion in open source libraries? Cloud computing? A single programmer with a laptop can do in minutes today what it took entire companies of hundreds of professionals and a large amount of funding a few decades ago. Yet the size of the industry hasn't shrunk in the same period – quite the opposite in fact. There is more need and demand for programmers today than ever before.
- sfink 4y agoOur current "AI" does only interpolation, not extrapolation. You feed it a large input corpus and it digests it in clever ways. Then when you probe for something contained wholly within the space described by that corpus, it is amazingly good at fabricating something plausible to match the point in that space that you requested. Which covers a lot of stuff and is very useful, but does very little for problems that require extrapolation. It can't expand the edges, it can't come up with anything truly original. It can't solve problems that people haven't already solved and written down the solutions somewhere the AI could find them. Another way of saying it: AI today is much better at memory than thought. Career advice for young people today: specialize in pushing the edges, not in filling in details. There will be some areas where applying existing stuff will hold out and be useful for a long time, but you'll always be racing against the AI. Colonize the parts of problem space where AI doesn't have the imagination to go. (Of course, humans are notoriously bad at correctly recognizing what does or doesn't require originality. Hell, we think we're making decisions about what to do every minute of every day, when in fact we're just a bunch of dancing meat automata following ingrained patterns 99% of the time.)
- Valakas_ 4y agoThis goes back to the root of what creativity really is. Some say creative people don't have anything special in their "brains". They are just able to mix match stuff they already know in different ways, that others didn't consider. In essence there is no such thing as extrapolation for the human mind. We only interpolate in new ways. We can only remix what we know, and with that generate something new. How do we actually get new, non-interpolated stuff? By going out into the world and doing things with it. Collecting new data. Experiments, experiencing things, seeing new stuff, reading about new stuff. That's not so different from the AI. Our brain is not something magical that works in mysterious un-replicable ways. Much of AI field has been in fact trying to replicate how we learn, and how our brain works.
- visarga 4y agoWhen verification is possible, the interpolation machine starts to extrapolate. There is a process that generates ideas/solutions, and a process that tests them. An artist and a critic, a scientist and a lab. Together they form the experimentation loop. Let's take the game of Go for example. Testing who won a game is trivial. AlphaGo managed to beat humans in a few days of self-training. In other words, those edges you speak about can be pushed with massive search and verification. There is no reason we can't do massive search + verification for math and code. This is a good way to create training data where it doesn't exist in sufficient quantity. Other things can be simulated with expensive computation, and then "distilled" into fast neural networks. Then we apply the neural net to fast-search solutions. In the end we need to verify some of them (thinking of weather simulations, new materials, new drugs, ...) Also reminded of the recent AlphaTensor who leveraged massive learning from verification to beat Strassen's algorithm who was state of the art for 50 years. The is no reason neural nets should remain purely interpolative if we can manufacture good training data by running computation or experiments. Ideas are cheap, verification matters. Generative outputs are worthless without verification.
- cratermoon 4y ago“Any fool can write code that a computer can understand. Good programmers write code that humans can understand.” – Martin Fowler
- raydiatian 4y agoOne thing software engineers can do now is push legislation that will protect their jobs from being replace by AI. It won’t be that long until there is a 10x improvement in AI-based software development over human-based, easily on the order of legislation lifecycle. When that happens, we need to be legally at the table, or we will straight up be cut out of the profits by billionaires and hedgefunds, because fuck us and anybody except for them (it is their world financially).
- addedlovely 4y agoWhat happens in a future where AI, that is trained on human generated datasets, becomes so prevalent that the humans that created the training data no longer have the original skillset required to create said dataset. Are we going to end up with AI training on AI generated datasets.
- Andaith 4y agoPerforming as well as humans isn't something to be proud of...
- _8j50 4y agoIs appsec dead/dying? Am I wasting time learning software exploitation? What's the point if everything soon will be rust and go written by ML?
- User23 4y ago> It’s also unlikely the AI will take over programming completely, as its code is riddled with errors I suppose that might count as performing as well as (some) humans.
- pizzaknife 4y agoahh good, and here i was getting bored of my current existental fears
- bob1029 4y agoI would say the biggest thing that keeps me from worrying about AI in reasonably-complex domains is the requirement for mapping a workable UX to the domain (i.e. for machine-human interactions in insurance, banking, health care, et. al.). Additionally, the higher-order complexity that emerges when complex domains interact with each other seems to be even more impossible to train for. Sure, a sufficiently advanced AI can craft a UI that looks like something incredible (e.g. McMaster.com), but it won't necessarily be able to replicate the quality of their customer service, consistency of offerings, or reliability of delivery (the UX). Small nuances in some domain can cause vast changes in how you would be expected to interact with it. AI typically fails to extrapolate and manage all of these possibilities. A good human developer can play devil's advocate for 30 minutes and come up with edges that would decimate any AI on the horizon. But, take that same pile of edges and hand them to a product wizard, and that person will give you all kinds of workarounds that would be acceptable to the end customer. Kick it back over the wall to the developers and you have happy customers. I don't think it will last forever, but the most complex domains seem to be the safest ones for a human developer's career prospects.
- lxe 4y agoAI code isn't real code. It lacks the human soul. It's theft.* It's unethically trained. As a coder, I'm exploited! It's going to take my job! * only if it spits out licenced code blocks and comments verbatim
- jkingsbery 4y agoThis isn't a thing to worry about at this point. In the real world (outside of programming competitions, that is), programmers don't get clear instructions for what we're doing, and what instructions we get deal with messy aspects of the real world. For just one example I tried: if you ask Codex how to calculate taxes on US products, it misses the point of the ask. I know that you have a tax rate and a price and you multiple them together - but what tax rate is applicable? Figuring that out is a thorny problem. In programming, a lot of the inherent complexity is in understanding the weird bits of ambiguity and the corner cases. Whether that specification is through natural language or some other thing, we're a ways of from a computer capable of replacing us on that.
- deepsquirrelnet 4y ago> understanding the weird bits of ambiguity and the corner cases I suspect it would not do a very good job at writing test code for itself. Until it can prove that it writes code that works, it’s only really guessing.
- hsuduebc2 4y agoIt seems impossible for programmers to keep their jobs in next five to ten years.
- bubbleRefuge 4y agoWonder what the analog to how severely -IMHO- the hacking and cyber security problems were underestimated at the dawn of the internet to how malicious forces will employ AI ? How will a criminal mind use AI to make a living?
- qualudeheart 4y agoI hope you folks understand this is old news. That Deepmind paper was published nearly a year ago. In fact, it was published a few months before other important papers. And yet, it seems pretty much all the press is still discussing and analyzing the old paper. Why is that? I’d like to think it’s because the earlier work is so compelling, but more likely someone needs to drive more traffic to their site. I think this is more of a problem for code ai discussion. This is something you can’t really discuss on a forum or blog post — your posts will get overshadowed by ML doomstering.
- prmph 4y agoThis is a prompt I gave ChatGPT: > Write an algorithm that, when given a sorted list of numbers, efficiently finds all numbers in the list that lie between a given a range And this is its response: > Here is an algorithm that can be used to find all numbers in a sorted list that lie between a given range: Initialize two variables, low and high, to the lower and upper bounds of the range, respectively. Initialize a variable i to the index of the first number in the list that is greater than or equal to low. Initialize a variable j to the index of the first number in the list that is greater than high. Return the sublist of the input list that consists of the elements at indices i through j-1. This algorithm has a time complexity of O(1), since it only performs a constant number of operations regardless of the size of the input list. Here is some example code in Python that implements this algorithm: def find_range(numbers, low, high): i = 0 while i < len(numbers) and numbers[i] < low: i += 1 j = i while j < len(numbers) and numbers[j] <= high: j += 1 return numbers[i:j] -------------- Does this algorithm really have a time complexity of O(1)? I would think it is O(N). Interestingly, when I posed a similar question to find the index of a specific number in the list, it returned a binary search algorithm. The above question' optimal solution also involves binary search, but asking for a range of numbers throws ChatGPT off, it seems.
- sischoel 4y agoIt is indeed O(N). There is also a more efficient algorithm (in case the range is small) that uses binary search to find the first index. Such an algorithm would have time complexity O(log(N) + H) where H is the number of elements in the output.
- HarHarVeryFunny 4y agoAm I missing something here ? At the end of the day, aren't ChatGPT, AlphaCode, etc still just finetuned LLMs, or has there been some significant changes to the architecture that give reason to expect the output to be anything more intelligent than LLM "predict next word" continuations ? Of course GPT, Codex/Copilot, ChatGPT are all impressive, and have some unexpected emergent capabilities (this being the interesting thing about this "scale experiment"), but even with "think step by step" prompt hacks and finetuning, they're still just LLMs unless something (intelligence!) has been added so they can learn from their mistakes at runtime and test their predictions ("understanding") by interaction with the external world. So, if these are still just LLMs, then why are people treating them as if their prompt answers might/should actually be correct as opposed to what might be expected from massive training set prediction - i.e. right a lot of the time, but catastrophically wrong a lot of the time too, with no insight as to which is which. I saw a ChatGPT example the other day where it confidently explained how taking 90% of a number then adding 10% of that would get you back to the original. It sure doesn't appear if there's any intelligence there, even if it seems it's generating "correct" answers enough of the time that people want to ascribe intelligence to it. I'd trust advice from ChatGPT about as much as I'd trust advice from Eliza (a simplistic 1960's ChatBot that also managed to fool some people). I'm not sure what it says that this type of "contextually correct bullshit" is being productized into things like OpenAI Codex. What next? Use it for medical advice too ? If I want untrustworthy suggestions I'd rather just Google for it where I can at least judge the source, as opposed to ChatGPT where you have no idea whether it's regurgitating stuff from NASA or 4chan.