9 ms·
Solving Advent of Code Puzzles with GitHub Copilot: Day 1
- have_faith 5y agoIt’s going to be interesting how services like copilot affect the quality of programmers who start their studies with it available as a tool. It will also have an interesting knock on effect on hiring? Someone could have a portfolio almost entirely generated but not a literal copy of anything else.
- mehphp 5y agoI suspect it would in increase their chance of getting an interview, but they would struggle during the live coding portions.
- NilsIRL 5y agoIt would be interesting to see if it's possible to just get Copilot to give you other people's solutions as often people upload their solutions to GitHub. The comment to trigger Copilot would be something like the following: # Advent day1 part1
- nunodonato 5y agoNo. It doesnt fetch new code from github or anywhere else. the model has been trained and uses the data of that training.
- Epskampie 5y agoIt’s interesting to see the mix of local identifiers and public code that copilot is able to generate. In the end I disabled it because I had to keep stopping to check its suggestions, really breaking the programming flow. But I imagine with some content and UI improvements it will be invaluable in the future.
- da39a3ee 5y agoYes, I find it quite fun as long as I can turn it on and off with a single keypress. Then you can have it mostly off, and consult it only when you want to.
- Zababa 5y agoI've tried it with Go and it's impressive. However, as you said, having to check generated code is tedious. I feel like I can trust Copilot's code less than what a colleague would produce. So then my coding sessions become code review sessions. I don't want most of my coding to be like that.
- thomasqbrady 5y agoIsn’t this working by finding someone else’s solution to the day 1 challenge? As in, since the OP copied and pasted the prompt, it was relatively easy for Copilot to match to the correct solution (also how it knew the window size for the second half was 3)?
- callmekit 5y agoI would think the today's code is not a part of the Copilot yet...
- fartcannon 5y agoSurprise! Everything everyone types into Microsoft software is recorded and uploaded!
- callmekit 5y agoI think you need to actually get the new data from Github and finetune the model. Maybe it's done very frequently, but unlikely.
- jonathankoren 5y agoI'd bet good money, they aren't ingesting, retraining, and pushing production copilot models on a subdaily basis.
- a-dub 5y agoi think that's kind of the point behind gpt style models, updates are cheap ... not only in terms of source material but also feedback on which suggestions are accepted.
- speedgoose 5y agoTraining the AI that often and fast would be impressive. But I don't think it's the case because they didn't brag about it.
- lokedhs 5y agoI can't help but think that whenever copilot is able to help with a solution, it's because the language itself doesn't allow you to express solutions concisely. I looked at the solutions in the article, and very little of the code is actually part of solving the problem. There are a few bytes of relevant information in each line. I usually use Common Lisp, and whenever I find myself typing boilerplate of a type that copilot would be able to help with, it's because I haven't used the right abstractions. As an extreme example of this, I solved day 1 in KAP (my APL dialect) and for the second problem, my solution was 8 characters long. There was no real need to have anything type that out for me.
- jakear 5y agoAgreed, even with trivial JS constructs parts 1 and 2 can be solved (including parsing) in 58 chars (and only O(n), not O(n*w) like Copilot's solution). The copilot prompt alone for the just the logic part of part one was 98 chars. Surprise, programming languages are more efficient at encoding logic than English.
- thomasahle 5y agoIt's a question of whether copilot can be "creative". I'm mit sure, but it did manage to solve some of the harder leetcode questions here https://youtu.be/FHwnrYm0mNc https://youtu.be/FHwnrYm0mNc and I'm curious how it will fare on the later advent of code problems.
- callmekit 5y agoThe exact answers to leetcode questions are probably included many times in the Copilot dataset, in many programming languages. Fresh Advent of code problems are much more interesting.
- YeGoblynQueenne 5y agoCopilot is trained on examples of programs, so it will not be able to produce programs unlike any it has seen before. This will not be easy to see in any kind of coding exercise. It is very hard to come up with coding problems with solutions that have never been written before, at least for a coding competition like Advent of Code. For example, we don't know of polynomial time solutions to the Travelling Salesman problem but it wouldn't make sense to have that as a coding challenge in Advent of Code, or Leetcode, or anywhere else. Since nobody has ever produced a program solving such a problem, Copilot will not be able to produce a solution either, but it's not designed to do that anyway.
- jeffbee 5y agoI don't know ... the video at least is straight up bullshit, right? The terminal at the bottom of the editor session shows that the user has already run "copilot_solution_puzzle_two.py" with the desired answer before he types anything.
- callmekit 5y agoThat's already mentioned in the blog post: the author first wrote a solution in a normal way, and only then decided to try copilot.
- FreddieV4 5y agoHey, thanks for your comment. I actually tried to make an attempt to write / run the code just once, and it printed the correct answer, but I forgot to record it. So I deleted all of the code and tried typing _again_ from scratch to record it, and ran the program _again_, and it yielded the same result (the second run is what's shown in the video) This is also something I'm curious about regarding Copilot. I mentioned that "GitHub Copilot has a list of possible solutions to code completions, so I wonder if it’s just luck that the suggested solution was the correct one". I'm going to test further to see if it just always produces the same code completion, or there's some bit of randomness to the completion.
- YeGoblynQueenne 5y ago>> This is also something I'm curious about regarding Copilot. I mentioned that "GitHub Copilot has a list of possible solutions to code completions, so I wonder if it’s just luck that the suggested solution was the correct one". The Codex model on which Copilot is based had about 30% accuracy on the first solution to a coding problem, but 70% when it was allowed to generate 100 solutions and choose the one that passed unit tests. On the other hand, when the best-of-100 solution was chosen according to the probability assigned to it by the model it scored 45% [1]. So it's kiind of luck. Basically, Copilot has no way to tell whether it's giving you a right or wrong answer, other than to select the answer with the highest probability according to its training set which usually means the most common answer in its training set. So the probability that the answer you get is the "right" answer depends on the probability that the right anwser is the most common answer to the problem you give it. If that makes sense? __________ [1] https://arxiv.org/abs/2107.03374 https://arxiv.org/abs/2107.03374 See Figure 1.
- lostcolony 5y agoRegardless of how it found it, or what it was basing it on, puzzle two is missing the key "aha" with such a sliding window type of problem. Which is that two of the values are the same in both windows you're comparing; you don't actually need to sum anything, just compare the unique values from each (i-3 and i, respectively). If Copilot is trained on a corpus of average devs...I do worry about what this says about the state of our field.
- wk_end 5y agoMeh. The problem is so small and simple and straightforward, you don't need any clever "aha" moment. If anything, being clever just makes the code slightly less obvious, because the reader needs to get that "aha" moment too. ...and my solution did actually do things the way you described, so I'm not just being defensive about it :P
- alexfoo 5y agoThese types of puzzles can help educate people as to how things might need to be coded to work in more "real world" situations. At my previous job we used them as the basis for discussion and education amongst the team (hoping to do something similar at current company). For example, taking 2021_day1, the naive solution that reads in all of the values and then iterates through calculating and comparing sums is all well and good, it satisfies the end goal of producing a solution. What if you had 1,000,000,000 values in the input? * A program that reads in and stores all of the values before iterating through them now consumes a non-insignificant amount of memory. What if you had to check the differences between the sums of 100,000 consecutive terms? * A program that calculates and compares the sums (rather than just comparing the two non-common terms) will be doing a lot more unnecessary computation. What if you had to take a seemingly endless input stream and report the running totals at 1,000,000 iteration intervals? * A program that reads all of the input before iterating through it is now unsuitable. ..etc.. Programming is not just about getting the answer, it's knowing where you can do the bare minimum to do that, and knowing under what conditions your bare minimum solution will become impractical and what you would have to do to avoid this. There's nothing stopping people using AoC as a toy to collect all of the stars (surprisingly few people have all 304 stars, it was under 700 last time Eric answered that question in July 2021). But it can also be used for a lot more than that.
- a-dub 5y agowould be more interesting to see when it starts suggesting anything relevant in the 15s,30s,45s,1m,10m,1h intervals after a puzzle is released...
- nunodonato 5y agoits completely irrelevant
- oezi 5y agoI can't even get into Github's Copilot pilot program. Are they still adding new users?
- varjag 5y agoThey do, but I believe the queue is sorted by users' GH activity.
- rkachowski 5y agoI decided to use rust yesterday for advent of code to learn something new and had copilot enabled. Copilot suggested `into_iter` instead of `iter` and my very first rust experience was a battle with the borrow checker trying to figure out wth is a move.
- jazzyjackson 5y agoyea i imagine having a robot write code in a language you’re not already an expert in would be painful In my opinion the danger of automation is that anyone who knows less than the machine is not going to recognize a mistake - plus, a robot can do the easy things first and now you have a talent pool problem: no one is paying juniors to learn if a robot can do the work of a junior
- fjfaase 5y agoFor the second puzzle, there is no need to calculate a sum of three elements. I only realized this myself after having thought about it a bit longer.
- seba_dos1 5y ago> How can I come up with solutions to puzzles more quickly? If that's the goal, an obvious solution may be to not write useless comments, typing and verbose function names :) Solving day 1 in Python is 10 lines of code without even trying to golf it.
- charcircuit 5y agoHe copied and pasted that comment from the problem.
- seba_dos1 5y agoThere's no such text in the problem - see https://adventofcode.com/2021/day/1 https://adventofcode.com/2021/day/1. That comment describes the solution, not the problem. The program's output is also annotated; you don't do that if you try to be fast, you already know what the output is gonna mean.
- ricwo 5y agoHey! For folks who are doing this year's Advent of Code in Jupyter Notebook: I'm one of the founders of https://cogram.com https://cogram.com, a coding assistant for data scientists. Cogram brings Copilot-like features to Jupyter Notebook. You can use Cogram by writing ordinary code (and Cogram will complete it), or by writing comments (in which case Cogram completes the code that matches the comment). We've also launched a web app that generates SQL statements from descriptions in simple language. We're just starting out and would love it if you could try it out, give feedback, and spread the word if you like. It doesn't cost anything -- we have a generous free tier (1000 completions / month).
- jstx1 5y agoI get that the point is Copilot and not the code but sum([arr[i] > arr[i - 3] for i in range(3, len(arr))]) will solve part 2. (Obviously for the first part subtract 1 and start the range at 1.) You don't need functions (but that's kind of an opinion, so whatever). More importantly, to check if a rolling sum is increasing you don't need to compute that sum. I wonder if Copilot can ever figure that out.
- WesleyJohnson 5y agoI haven't dug into this too deeply, but how does checking the current arr index against the value 3 positions back work to correctly calculate the answer?
- WesleyJohnson 5y agoAnother poster had the same thing, but explained it a bit further. I get it now. Clever!
- agsht 5y agoWhat is with the recent "grassroots" Copilot postings that go uncontracdicted in the comment section? Are the critics worn out and now the code laundering machine is pushed by sheer volume of postings?