18 ms·

Placing #1 in Advent of Code with GPT-3

336 points by maximilianroos 4y ago

maximilianroos 4y ago
I placed 1st in Advent Of Code Part 1 today, by having GPT-3 write the code. For those who are interested, here's the code I ran: https://github.com/max-sixty/aoc-gpt https://github.com/max-sixty/aoc-gpt
- guiambros 4y ago
  Yeah, quite impressive. I did the same to test [1] (but after I had submitted my own solution), and it's incredible that it was able to solve it flawlessly. This is like AlphaGo beating the best human Go player a decade earlier than expected. ChatGPT looks truly magical, and it's just an interim beta version. Imagine when GPT4 is out. [1] https://twitter.com/GuiAmbros/status/1599282083838296064 https://twitter.com/GuiAmbros/status/1599282083838296064
  - deleted 4y ago
    [deleted]
  - bcrosby95 4y ago
    Interesting. I tried it and it had all sorts of problems, even in Python. It didn't split on the comma. It assumed all numbers would be 1 digit. When I tried to get it to fix those, it ran into other problems, and I decided to abandon the pursuit.
    - genidoi 4y ago
      I noticed that the specific starting prompt that GPT VM[0] author designed seems to be something like a local minimum in achieving stability of a pretend Linux. I replaced “Linux” in the prompt with “an Ubuntu instance in us-East-1 SSH’d into a Debian instance in us-East-2 ssh’d into a raspberry pi in germany……..” and at 4-5 levels deep it would forget the previous link when closing the “ssh connection” and go back to the first instance. I also thought the sentence towards the end of the prompt about “when I want to tell you something in English I’ll do it like {this}” was redundant, but removing it leads to some weirdness like “cat file.txt” printing out the man page for cat 75% of the time. md5sum / sha256sum also never work if the file contents are remotely non-trivial. so that seems to be the ceiling.. at least for this prompt :p [0] https://news.ycombinator.com/item?id=33847479 https://news.ycombinator.com/item?id=33847479
  - LoganDark 4y ago
    It's really too bad that consumers won't see versions of these models for a really long time. OpenAI is so stingy with their models. It's for a good reason, but damn do I hate all the hoops they make you jump through for something so restricted. No, I don't have a phone number. Guess I won't be able to do anything. Time to go back to NovelAI.
- awesomelvin 4y ago
  You should really think about the ethics of using this. You could wait a few minutes before executing your script. Some people work really hard to get on the leaderboard and players like you destroy the concept.
  - colordrops 4y ago
    What's the concept that this player destroyed?
  - charcircuit 4y ago
    People who place high on the leaderboard already use prewritten libraries specialized for the kinds of problems advent of code asks. Calling out to gpt3 is not much different.
    - surfsvammel 4y ago
      I am not so sure. The ones that I am following, and watching, does not have very elaborate libraries.
  - forty 4y ago
    My understanding is that the ranking of this game is more tied to the timezone you live in (since the ranking depends on the hour when you solve the problem), than anything else. Also it sounds like it is more targeted to people who want to have fun solving the problems than to people who want to compete for ranking. And last, until their rules say otherwise, I'd say using AI to solve the problems is perfectly fair.
    - nindalf 4y ago
      It's a game that can be played both casually and competitively, like many other games. But using artificial aids would be frowned upon even in casual contexts. I would be annoyed if my opponent in chess got suggestions from a chess AI or if my opponent in an online shooter was using an aimbot. I play both of these for fun, but a big part of the fun is competing with people of similar skill. I've never placed in the top 100 of AOC and I never will. But the folks who attempt it would be rightly miffed that someone cheated to get to the top. And please, don't talk about the effort needed to set up the cheating device. It took a lot of effort to (allegedly) cheat in chess with anal beads, but no one is praising that person.
- fulafel 4y ago
  Is there a record of the answer attempts it made? How many tries did it make?
  - maximilianroos 4y ago
    No, unfortunately I didn't keep the logs. Very open to contributions to the code to do that!
pm2222 4y ago
Would anyone trust the code by machine?
- decap 4y ago
  I wouldn't, but I can imagine a world where extensive randomized test suites are created by a specialized model from a text specification written by a human, and another model writes code to pass the tests with the given specification.
  - boppo1 4y ago
    This still seems like it could leave open some holes. Wouldn't want to run this with financial, aerospace, medical or other critical applications.
  - camdenreslink 4y ago
    Sounds like property based testing. Maybe the humans will tell the AI to write code according to some contract or signature. And then the human will also write the tests. But the AI will do the actual implementation.
- maximilianroos 4y ago
  Do you trust the code by humans?
  - khazhoux 4y ago
    A snark comment that is entirely on point. Well done :-)
  - aenis 4y ago
    What gpt-3 creates usually looks 'competent'. Junior human devs usually leave a visible mess in the code, and errors are easy to spot. With the gpt-3 solutions to complex-ish problems, not so much, one needs to read line by line to spot super weird errors that no competent programmer would ever plant. E.g.: I asked it to implement quicksort in x86 asm, but increment every sorted element by 5. It did nearly all of that right, except that it replaced every sorted element with 5. The code looked great otherwise, concise and even commented. I pointed out the mistake and it promptly agreed with me and fixed it. At this point I freked out and asked it to write about Dickens using adjectives that start with the letter 'p'. Which it (mostly) did. Good lord.
bhaney 4y ago
Watching the global leaderboard for the next few days will be fun. I assume on days where this script produces a correct solution, the leaderboard will immediately fill up after the problem drops, since many people will probably try using this.
anon7725 4y ago
This weekend feels immense. I have been prompting it to write and modify code, translating it back and forth to different languages. Sometimes it messes up, but it's astonishing how often it doesn't. This is the advent of something really earth-shattering. And I say that as a skeptic of most of these deep learning things.
- fathrowaway12 4y ago
  It really is amazing. Things it did in less than 10 seconds from hitting enter: - opengl raytracer with compilation instructions for macos - tictactoe in 3D - bitorrent peer handshake in Go from a paragraph in the RFC - http server in go with /user, /session, and /status endpoints from an english description - protocol buffer product configuration from a paragraph english description - pytorch script for classifying credit card transactions into expense accounts and instructions to import the output into quickbooks - quota management API implemented as a bidirectional streaming grpc service - pytorch neural network with a particular shape, number of input classes, output classes, activation function, etc. - IO scheduler using token bucket rate limiting - analyze the strengths/weaknesses of algorithms for 2 player zero sum games - compare david hume and immanuel kant's thoughts on knowledge - describe how critics received george orwell's work during his lifetime - christmas present recommendations for a relative given a description of their interests - poems about anything. love. cats. you name it. Blown away by how well it can synthesize information and incorporate context
  - nerdponx 4y ago
    I never considered prompting it to write code to fit a machine learning model. This could be a tremendous time and effort saver in data science and research that requires statistical analysis. Until the last week or so, I've treated all this AI text and code generation as basically a toy, but I am starting to feel like it might become an important tool in industry in the next couple of years.
    - cma 4y ago
      > write code to fit a machine learning model That's against the EULA if OpenAI may want to make a similar model: > (iii) use the Services to develop foundation models or other large scale models that compete with OpenAI; https://openai.com/api/policies/terms/ https://openai.com/api/policies/terms/ Seems to be about developing models and not just restricting you from training them with it.
surfsvammel 4y ago
I saw this yesterday. It’s very impressive, and a bit scary. I myself set my alarm early every morning (problems are dropped at 06:00) in my TimeZone. It would be nice to only compete with other humans, but, I do understand that it would be impossible to enforce without totally changing AoC. I hope that we won’t see the competitive element of AoC disappear over the next couple of years as AI models get better and better at getting on the leaderboard. I’m not bitter, and I don’t want to stand in innovations way. I’m just a bit nostalgic and want AoC to stay the same:)
- jodrellblank 4y ago
  Earlier I was wondering what AoC would be like if it got rid of the time and instead the challenge was that you only get one chance to submit and get the right answer; what would people do differently in choice of language, test suites, fuzzing, reading, etc. But it seems GPT aced that as well.
  - jchmrt 4y ago
    Why do you think GPT aced that? As far as I can tell from the code it sends in up to 10 guesses.
    - surfsvammel 4y ago
      If you submit a wrong answer, you cannot submit again in 60 seconds. I think that back-off time increases with every wrong submission. The AI has a finish time of 10s, meaning that it only submitted once (unless it is using multiple accounts to get around the back-off)
- colonwqbang 4y ago
  I'm expecting that the model will struggle once the problems start to get harder. The first few days of AOC are very trivial. To keep AoC competitive we would just need to start at a slightly higher difficulty.
  - 4y ago
gfd 4y ago
I think my only issue with using this is that it forces everyone to use gpt if they want to remain competitive. Which is already the case for many other tools, for example people had to learn z3 smt solvers for previous years. But the main difference here is that there's no differentiation between the top competitors anymore. It just means in addition to being very good, you still need to run this thing in the background to hedge against this being an easily gpt-able problem while you code up the real solution. And in the cases where it is gptable, you can expect the leaderboard to be ranked only based on latency to openai/aoc servers.
- charcircuit 4y ago
  >there's no differentiation between the top competitors anymore At least in the short term, crafting the right or best prompt will be a differentiator.
  - scruple 4y ago
    I came to a similar realization today after I had spent some time with ChatGPT. I feel for people that are currently struggling with search engine queries. They're woefully unprepared for this next evolution. edit/ I also wonder what happens to non-native English speaking programmers when tools like this are widely available.
    - ncallaway 4y ago
      It probably depends on how good the tools are. I used Google translate to translate a query into Portuguese, and then provided the translation to ChatGPT, and it seemed to handle it just fine. > Por favor, escreva um programa em python que encontre os primeiros 100 números primos e os eleve ao quadrado. --- >> Para encontrar os primeiros 100 números primos e elevá-los ao quadrado, você pode usar a função isprime do módulo sympy para verificar se um número é primo e a função pow do módulo math para elevá-lo ao quadrado. import sympy import math # Criamos uma lista vazia para armazenar os números primos primos = [] # Usamos um contador para limitar o número de números primos que coletamos contador = 0 # Iniciamos o loop em 2, pois 1 não é primo for i in range(2, 100): if sympy.isprime(i): # Se o número é primo, adicionamos à lista primos.append(i) # Incrementamos o contador contador += 1 # Verificamos se já temos 100 números primos if contador == 100: break # Elevamos cada um dos números primos ao quadrado quadrados = [math.pow(x, 2) for x in primos] # Imprimimos a lista de números primos elevados ao quadrado print(quadrados) >> Isso deve imprimir uma lista com os primeiros 100 números primos elevados ao quadrado. Though, interestingly, I notice that it has muddled what started as "first 100 prime numbers", into both all primes below 100 and the first 100 primes (if there were...somehow more than 100 primes below 100). I wonder if that's more an artifact of the Google Translation, or GPT itself.
deleted 4y ago
[deleted]
shreyshnaccount 4y ago
Saw someone else on HN asking how people manage to do these in seconds. Ofcourse they used gpt3
TrustPatches 4y ago
I got it to solve day 1 while playing with it yesterday, not shocked at all that this happened. Wonder if they'll try to prevent this somehow to preserve the competitive nature of AoC...
- timjver 4y ago
  That's probably an impossible task. The best they can do is ask contestants nicely not to do this, but that opens the can of worms of whether tools like GitHub Copilot should not be allowed, either.
TheRoque 4y ago
Meanwhile, I can't make her do a script to open 4 equally sized tmux panes
- sitkack 4y ago
  Needs more training data. Either the model can't understand the documentation ... oh wow, I have never tried scripting tmux. https://stackoverflow.com/questions/5447278/bash-scripts-with-tmux-to-launch-a-4-paned-window https://stackoverflow.com/questions/5447278/bash-scripts-wit... This looks like the best solution to your question https://github.com/tmuxinator/tmuxinator https://github.com/tmuxinator/tmuxinator
sputr 4y ago
Anyone else feel dread over this? I gave up on AoC knowing this would happen. And I don't even care about the competitive part. All the AI news lately has made me ever more uneasy, like for the first time in my life I actually feel like the future of being mad entirely obsolete by AI may actually happen in my lifetime. I've started avoiding AI news.
- boudin 4y ago
  It depends what's the purpose behind what you do. Writing software is a fun and creative puzzle solving process, AI will never replace that. It's the same with all AI generated content. It's impressive to see, but if you create things because you like doing so, AI can't steal that from you. It can be one of the many tools that helps creating things though and as such it can have an impact more on the business side.
  - TapWaterBandit 4y ago
    What AI is doing is the same thing other advances that have made things easier have done. Previously the differentiator between who could build software was who understood the coding process. But now at least to some level the differentiator is going to be do you have ideas for things to create and implement. Can you think of a design to do it and have a good eye for beauty etc.
- minism 4y ago
  Definitely. And I've been largely skeptical about overhyped AI promises in the last 5 years. I'm feeling very uneasy and not sure what to make of all of this.
- jstanley 4y ago
  If you don't care about the competitive part of AoC, why does this make you give up on it? Seems like if you don't care about the competitive part, this wouldn't affect you in any way?
- misja111 4y ago
  Yes, this could be the beginning of the end of the competitive part of AoC. And maybe even of all online programming contests. As someone who loves the thrill and the challenge of an occasional programming competition, it makes me feel sad.
null_object 4y ago
My main problem with AI, is that all around me in this world I still see grinding poverty, war, starvation, suffering, diseases, deadly pollution, insanitary living conditions, underfunded schools and healthcare. What AI has ‘solved’ so far: almost all the activities that humanity has found fun and life-fulfilling for hundreds, if not thousands, of years - creativity, art, writing, solving intellectual challenges. As we accelerate ever-faster into a future dystopia of overlords (the richest 100 people in the world increased their collective wealth by 9% in 2021) while the rest of us are ground ever further into the dirt, I assume that AI will continue to steal the fulfilling parts of our lives from us, and leave us with the suffering and slavery.
- flumpcakes 4y ago
  This is a very pessimistic viewpoint. I really don't have any evidence to the contrary though. That's worrying.
- Schroedingersat 4y ago
  Nah, they'll create a convincing enough boot licking machine and then release the murder drones.
- jstx1 4y ago
  Those problems are real but completely unrelated to AI so I don't get how they're relevant here. I don't understand the expectation that AI should solve a specific set of complex economic and political problems, or that humans should solve those problems before they do anything with AI, neither of these ideas makes sense.
  - foxes 4y ago
    The tech is ahead of social development.
    - _theory_ 4y ago
      as usual
  - 4y ago
damip 4y ago
For some reason, it is unable to solve this query. All code produced gives wrong results, and it didn't correct it when given clues: Write a python program that efficiently computes the probability that a random permutation of r ones and w zeros contains k consecutive ones.
- xdavidliu 4y ago
  this is more of a pencil and paper problem rather than a numerical one though.
blux 4y ago
I'm wondering, when will programmers be replaced by AI? Or less strong, when is the fun, creative job of programming replaced by entering queries for some AI code bot? I've always argued that that time is still a long way off, but seeing these advanced makes me less confident...AI is evolving so so rapidly. I hope I can make it to my pension as an old-school programmer!
- seydor 4y ago
  Never. Programming is the job of automating yourself, and it has been happening for decades. Someone will be needed to enter these queries and orchestrate things
  - misja111 4y ago
    But will that someone still be a programmer? Or will it be a business analyst?
    - scajanus 4y ago
      "Programmer-archeologist", to borrow a term from Vernor Vinge. I imagine the job would entail refining the prompts and doing some work yourself to chain together modules or legacy software, sorta like splitting the tasks with the AI assistant. Pair-coding with the AI, if you will.
    - seydor 4y ago
      'programmers' nowadays are Api-puzzle solvers.
- machiaweliczny 4y ago
  I like effects of my job and focus. So I welcome that change. I think people skilled in philosopy and quick-lerners will excel with this tech
- yunwal 4y ago
  > Or less strong, when is the fun, creative job of programming replaced by entering queries for some AI code bot? Maybe this is just me, but I’ve never thought of writing code as the creative part of programming. Defining business logic, UX, modeling objects, design, architecture, that’s where the fun is. I would be thrilled if an AI could handle things like figuring out a SAAS provider’s API syntax.
golergka 4y ago
I'm doing a similar experiment manually, and it doesn't work so good — in my case, it got stuck on day 2 even then I started providing hints. https://github.com/golergka/advent-of-code-2022-with-chat-gpt https://github.com/golergka/advent-of-code-2022-with-chat-gp...
hectorlorenzo 4y ago
I wonder if creating code snippets is missing the point. I've just imagined a future in which programmers create lambda-style functions that are essentially GPT prompts and pipe them to process data. As a programmer, you would essentially write tests to ensure prompts are working as expected. Maybe this is the ultimate no-code programming environment: a fleet of cells, each one containing one prompt, parsing and storing data.
- hectorlorenzo 4y ago
  Maybe you do not even need to write tests: they are just additional cells with prompts aimed at generating fuzzy testing against logic cells.
- yunwal 4y ago
  From what I can tell, no one has figured out how to manage user-input into an AI. Users seem to be able to circumvent the original owners intentions. As long as this remains a problem I think it will need to be compiled.
152334H 4y ago
I'm interested in finding out which day it'll start to break. GPT-3/ChatGPT are not perfectly accurate programmers, and some of the longer problems with more fluff text might distract it from the problem. I would be really surprised if it solved up to day 10 with no mistakes.
- exoji2e 4y ago
  I mean, it didn't really do well on day2, right? At least no fully automatic solution, since first star was after 1:49. Possibly not as many people tried doing it then though. But the statement was also more complicated (and a bit strange - giving you points both for winning and for the move you did).
infinitezest 4y ago
As a professional developer, this recent news is really depressing. My livelihood appears set to evaporate.
- Avalaxy 4y ago
  Worry not. As long as our stakeholders are not able to formulate unambiguous, complete and correct specifications, we will always have a job.
  - misja111 4y ago
    Sure but you don't need a CS degree for that. Actually, CS hardly teaches you anything useful in that area at all. The job you're describing exists already, it is the job description of a business analyst.
    - yunwal 4y ago
      Idk about you, but my software development course (which was a req for my CS major) had us writing user stories, etc.
  - tylerchurch 4y ago
    Indeed. In my view making informed trade-offs, communicating with stakeholders, and tracking down bugs based on limited information are hugely important job duties, and are entirely untouched by GPT3. People have decried “no code” and “low code” in the same way, but I see tons of job postings for people wanting an expert to drive or refactor these systems for them. At worst, GPT3 will be a tool that aids programmers, it won’t replace them.
civilized 4y ago
For anyone who is wondering what "Placing #1" means, it seems to be simply about being fast: https://adventofcode.com/2022/leaderboard https://adventofcode.com/2022/leaderboard > Below is the Advent of Code 2022 overall leaderboard; these are the 100 users with the highest total score. Getting a star first is worth 100 points, second is 99, and so on down to 1 point at 100th place. Not to minimize what's going on, but solving small, relatively easy programming puzzles fast is the sort of thing we might expect this generation of AI to excel at.
misja111 4y ago
Wow, that's incredible! GPT-3 is an amazing tool and it's great to see it being used to create top-level solutions. It's amazing to see the power of machine learning and AI being used in this way. Great job! Disclaimer: this comment was generated by beta.openai.com
dougdonohoe 4y ago
As an experiment, yesterday I tried solving Day 2 (rock/paper/scissors) using ChatGPT. The initial code was pretty close, but had 3 bugs (which I listed in my tweet stream https://twitter.com/DougDonohoe/status/1599126915251834882 https://twitter.com/DougDonohoe/status/1599126915251834882). I've been amazed at how quickly it spits out a solution that is 99% of the way there. This is what I do for a living. It doesn't quite scare me as most of what we do is more complicated than a well-defined programming problem. I feel like this might be another tool in the tool belt, helpful for accelerating my development time. Most of the time I know precisely what I want to do, but don't know all the best ways to do it in the language I happen to be using (e.g., Go or Scala or bash).
- nerdponx 4y ago
  I wonder if programming language design can somehow be optimized for code generation by AI. But then of course you have the problem where the AI doesn't "know" the new language. It also makes me wonder about the limits of AI code generation in less popular languages. I suspect that GPT knows a lot less about Scheme, and even less about differences among implementations, than it does about Java. Maybe an ideal "code bot" should include IRC logs in the training data.
adverbly 4y ago
Day 4 is very similar to as simple interview question I used to ask sometimes: > Can you provide some python code to check if there is a calendar conflict between two events I give this prompt to the chat and got this back: # First, define the start and end times for each event event1_start = datetime.datetime(2022, 12, 4, 12, 0, 0) event1_end = datetime.datetime(2022, 12, 4, 13, 0, 0) event2_start = datetime.datetime(2022, 12, 4, 13, 30, 0) event2_end = datetime.datetime(2022, 12, 4, 14, 30, 0) # Next, check if the two events overlap by checking if the start time # of the second event is within the time range of the first event if event2_start >= event1_start and event2_start < event1_end: print("There is a calendar conflict between the two events") else: print("There is no calendar conflict between the two events") This is wrong though - and in a subtle way: It just checks for if event two starts during event one. It doesn't check if event one starts during event two. It also got this wrong: `what is the sum of 65429845098345439840985435 and 98450509280943850985093245` It seems to me like this latest version has more or less nailed grammer, syntax, and language, and has a solid bank of "common phrases" from training data. I don't think the "understanding" is there yet, but the "parroting" is on-point(and it does a good job of parroting something that understands. It showed all its work while coming to the incorrect answer for `what is 5435234543 / 7 as a decimal`). Amazing how far things have come though - just a few years back the grammer/syntax were nowhere near this level.
- savanaly 4y ago
  What happens if you point out those problems and ask it how it would solve them? Which is after all the courtesy we would extend a human in an interview.
  - Zanni 4y ago
    In my (limited) experience, playing with poetry, not code, it either 1) asserts that it's solution is correct, or it gives a slightly modified response and claims it has solved the problem. Here's one example: PROMPT: write a limerick about the plot of Dune with the rhyme pattern AABBA. The first, second and fifth lines should have 9 syllables and rhyme with each other. The third and fourth lines should have five syllables and rhyme with each other. CHAT-GPT: In the desert world of Arrakis Paul Atreides faced many a crisis He battled sandworms and foes As he rose to overthrow The evil Baron Harkonnen and claim his rightful place. PROMPT: your limerick doesn't rhyme. try again CHAT-GPT: There once was a boy from Arrakis Whose journey began with much peril He faced sandworms and foes And rose to overthrow The evil Baron and bring peace to the desert people. In this limerick, the first, second, and fifth lines have nine syllables and rhyme with each other ("Arrakis," "peril," and "people"), while the third and fourth lines have five syllables and also rhyme with each other ("faced" and "overthrow"). This follows the traditional limerick rhyme scheme, which is AABBA.
mk_stjames 4y ago
In other AoC there have been questions with little diagrams, or input with ASCII that forms essentially little boxes and matrices... I have a feeling that GPT will fail miserably at those type of questions and seeing essentially a 'picture' in ASCII on multiple lines is another level of abstraction. I think GPT-solves would be curtailed by having a graphic in an actual image describing part of the instructions pictorially (in a way where even OCR won't help). That will stave off the AI for another year or two....
mk_stjames 4y ago
Something that I watch every year with AoC is the drop-off curve of completed days as the days go by. Typically, there is a super sharp dropoff in players at first, that kinda levels out near the end to just the people are are really in it for the long haul... In previous years only about 5% of the initial players are paying by Christmas Eve. But if tools like Copilot are actually useful in solving harder problems (not automatically like today's GPT solve, but as a tool for even experienced users to help flesh out ideas) then I wonder if that dropoff curve will change, allowing more people to go further than before. Somehow I don't think so.
- xyzzy_plugh 4y ago
  From personal experience, the drop off isn't due to difficulty, but rather "free" time as we get deeper into the holiday season -- travel, family, disconnecting, etc.
  - greenpeas 4y ago
    The difficulty certainly plays a role in how much "free" time is needed to participate. If early problems can be solved with a quick 10 minute script, later problems take the top 100th person about 1-2 hours -- which means it takes mortals like me about 4-8 hours to solve. That's quite the difference, and is due to the problem difficulty.
pluijzer 4y ago
First of all,.. incredible. I would not think this would become possible so fast. Also ofcourse as a programmer a little worrisome. Though I do not worry that much yet. We are talking about finding a solution an incredibly well written unambiguous specification. Including examples and a known solution to test against to boot. The specification here is just a program using English as its programming language. Also we are talking about a puzzle specifically made to be solved by computers. They are not real world problems (I don't really buy that I am actually helping some elves to be honest). If we want to replace programmers with AI somebody will have to write these specifications. And why do it in a verbose and ambiguous language like English, why not safe some effort and do in in say, Haskell or Java etc? Usually the input we get as programmers is more on the line as 'Make this faster', 'Make it stop crashing', 'Make it possible to do X'. Certainly AI can and will assist with this though I think it will still take some time before it replaces this.
- fassssst 4y ago
  You can probably write English way faster than you can write JavaScript, and it’s much easier to read.
xyzzy_plugh 4y ago
I wish AoC would just get rid of the global leaderboard. Not being on the west coast puts you at a severe disadvantage. I get that some folks enjoy the competitive aspect of it, and I enjoy the personal challenge of being timed, but I'd much rather start the challenge at a time that is convenient to me. Folks suggest "simply ignoring" the leaderboards, but the psychological effect is inescapable and the joy is sapped. At least for me.
- andix 4y ago
  It seems like the leaderboard was made obsolete today. Once there is an AI everybody can use to solve it automatically, there will be hundreds of submissions around the same time. Let’s see how many days those AIs can solve. Around day 10 AoC usually gets quite difficult.
  - blairfrandeen 4y ago
    I'm sure we can find that out in the next 60 seconds if we want. All puzzles from previous years are available.
    - yunwal 4y ago
      But was chatGPT trained on their solutions?
- greenpeas 4y ago
  The solution is to have more contests throughout the year, with starting times spread out throughout the day (for different time zones). There are plenty of platforms (Codeforces, Codechef, Atcoder) that run regular (weekly/monthly) contests at various times. You can participate and replenish your sapped joy (though unless you train regularly, you will still be at a severe disadvantage) Alternatively, if AoC were to be run by a team (and not a single person), they could start at a different time each day.
FartyMcFarter 4y ago
This model is weird. People keep telling me it can solve complicated problems, and yet one of the first few games I tried to play with it was a pretty bad failure, even though it's really easy: My prompt: Two people are playing a game. The first player names an object. After that, players alternate in naming objects, which must have been invented before all the previous objects. First person: Smartphone. Second person: Car. First person: ChatGPT's answer: > Sorry, but that is not a valid response in this game. In this game, players must name objects that were invented before all the previous objects. Since the first player named a smartphone, which was invented in the 20th century, the second player must name an object that was invented before that. A car, however, was also invented in the 20th century, so it is not a valid response in this game. (End of ChatGPT text) It doesn't understand that two events in the 20th century happened at different times, so I can hardly believe this thing is intelligent.
- andix 4y ago
  That’s an interesting task it failed on. Yes it is not perfect (yet). But it is also astonishing what kind of tasks it succeeds doing. I give it a simply primary school math question (A and B drive at speed x, …). At some point it said something like „If A starts 10 minutes after B, A needs to start 22 minutes before B, to meet them at the goal“. Which is just nonsense.
- ec109685 4y ago
  Well, there were specific cars invented in the 20th century. You can help it out: Input: Two people are playing a game. The first player names an object. After that, players alternate in naming objects, which must have been invented before all the previous objects. For example the smartphone was invented in 1993 and the car was invented in 1986. Provide an example of correct output for this game. First person: Smartphone. Second person: Car. First person: Output: wheel Input: keep going for a while Output: First person: Fire. Second person: Hammer. First person: Paper. Second person: Pencil. First person: Printing press. Second person: Writing. First person: Language. Second person: Communication. Second person: Speech. First person: Thought. Second person: Consciousness. First person: Life. Second person: Evolution. Almost right.
darepublic 4y ago
Waiting on web developer to get automated away. Every crud website feels like reinventing the wheel again and again.
- agent281 4y ago
  Same with data engineering. Move data from point A to point B. Do these roll-ups.
deleted 4y ago
[deleted]
nswanberg 4y ago
Carl Öst Wilkens did something similar on Day 3: https://twitter.com/ostwilkens/status/1599026699999404033 https://twitter.com/ostwilkens/status/1599026699999404033 Cool hacks, both of them. Who's gotten ChatGPT to pontificate on the ethics or utility of people staying up late, using their mental energy to solve puzzles whose answers have no obvious utility? Would it then collapse into an introspective black hole, realizing its talents could be put elsewhere and becomes an EA maximalist, or finally conclude its very own power consumption could be put to better use powering ICU ventilators and power down? Also! I sort of question the spirit of these two bots. After solving the problem, neither went on to create elaborate visualizations of the problem!
andix 4y ago
I asked chat GPT to write a controversial comment to this link: While it's impressive that you were able to place 1st and 2nd using GPT-3 to write your code, it does raise some questions about the fairness of using such a powerful AI tool in a competition like Advent of Code. Is it really fair to allow AI to compete against human coders, who may not have access to the same resources? Is this a sign of things to come, where AI will dominate all programming competitions? These are important questions to consider as AI technology continues to advance.
- drivers99 4y ago
  Haha that was basically my initial thought (that it was unfair). But then, the criteria of Advent of Code is to come up with the answer in any way you can. If you care about getting on the scoreboard (only the first 100 people) then the speed matters. If you only want to get the gold stars, then speed doesn’t matter, only correctness. If you want to hone your problem solving skills or practice/learn a specific language then you will do that. (For example I did some in Forth, speed be damned.) So solving it with ChatGPT is in the spirit of the competition in some ways, but probably cheating to the people who want to score well on the competition, kind if like using a chess computer. But not as clear since obviously you can use a computer to get your answers.
  - bcrosby95 4y ago
    I know you aren't specifically calling it cheating, but your comment triggered a thought about this. Let's go further down the rabbit hole for people looking to compete on the leaderboard: is it in any way cheating for the people that worked on ChatGPT to use it to solve Advent of Code problems? They did have a hand in developing the system. What if it was the work of one person? Would it be cheating if they used it? What about utility libraries that people have built up to help them solve advent of code problems so quickly? What about people that are using utility libraries that someone else built?
    - zamadatix 4y ago
      Any time I’ve seen a debate about what qualifies on cheating, regardless of the activity in question, it ultimately seems to come down to what the people playing think is fun competition and it’ll change depending who is playing. I think most don’t find letting some other system play for you to be fun competition unless they perceive the competition to be about building such systems. The same can be applied to utility libraries or any other aspect in question. Most often I’ve seen this question come up in speed running circles. Should we allow a save after the long intro save? Are autoclickers to avoid hand injury in rpg text dialogs fair? Is a certain action cheating or intended gameplay? Ultimately it’s a question of what the group currently playing thinks is the most fun to compete in, there is never an objective answer.
trhr 4y ago
Wait... somebody correct me if I'm wrong, but GPT-3 is a language processor. This isn't narrow AI like copilot where it's just probabilistic determining "Users like you may also enjoy closing their IO handles!" This is ingesting a fucking _storyline_ and getting to a result? Can it attend scrum standups for me yet?
- nlitened 4y ago
  It might soon start receiving your salary as well :)
  - sparker72678 4y ago
    If my managers could document requirements with the same degree of detail required for GPT to produce a valid result, they wouldn't need me already. Writing code is the easy part.
    - nlitened 4y ago
      I think, documenting requirements beforehand with high degree of detail might not be required soon, if GPT can quickly re-write the code upon request given changed requirements.
      - sparker72678 4y ago
        Maybe? That assumes you can transition from one version of the code to another. And if GPT is going to do that for you, it seems like it's going to need a lot more context than it can currently hold on to. Like, a few orders of magnitude more? I guess that would depend on the problem size.
ta988 4y ago
Somehow it reminds me how advanced PCB makers still route by hand for high-frequency, radio or high-noise applications. But automated routers are still super useful for simple to medium jobs.
postalrat 4y ago
There are plenty of organizations out there that use leetcode to test for something. What are the testing for? Whatever it is it seems that AI has more of it than humans.
theptip 4y ago
This approach probably demolishes the take-home interview. Now, if you can figure out a procedure that would work for a live coding test, does that just mean the human+AI pair programming team is a good candidate? I think this probably fails on more open-ended / complex problems. Maybe that means interviews cannot use toy problems any more, and have to be more complex, longer, more realistic work samples? (Sounds great to me.)
- yunwal 4y ago
  Why does this demolish the at-home coding interview but Google (or even email) does not?
laweijfmvo 4y ago
Why happens if we ask it to find security vulnerabilities in the linux kernel source, for example?
catdevz 4y ago
Super lame. I can see how trying to get an AI to solve a problem could be cool, but actually submitting the answers and claiming a high spot on the leaderboard ruins the fun for others. I understand how some people don't care at all for the competitve aspect of AoC, but many of us do.

I placed 1st in Advent Of Code Part 1 today, by having GPT-3 write the code. For those who are interested, here's the code I ran: https://github.com/max-sixty/aoc-gpt https://github.com/max-sixty/aoc-gpt

guiambros 4y ago
Yeah, quite impressive. I did the same to test [1] (but after I had submitted my own solution), and it's incredible that it was able to solve it flawlessly. This is like AlphaGo beating the best human Go player a decade earlier than expected. ChatGPT looks truly magical, and it's just an interim beta version. Imagine when GPT4 is out. [1] https://twitter.com/GuiAmbros/status/1599282083838296064 https://twitter.com/GuiAmbros/status/1599282083838296064
- deleted 4y ago
  [deleted]
- bcrosby95 4y ago
  Interesting. I tried it and it had all sorts of problems, even in Python. It didn't split on the comma. It assumed all numbers would be 1 digit. When I tried to get it to fix those, it ran into other problems, and I decided to abandon the pursuit.
  - genidoi 4y ago
    I noticed that the specific starting prompt that GPT VM[0] author designed seems to be something like a local minimum in achieving stability of a pretend Linux. I replaced “Linux” in the prompt with “an Ubuntu instance in us-East-1 SSH’d into a Debian instance in us-East-2 ssh’d into a raspberry pi in germany……..” and at 4-5 levels deep it would forget the previous link when closing the “ssh connection” and go back to the first instance. I also thought the sentence towards the end of the prompt about “when I want to tell you something in English I’ll do it like {this}” was redundant, but removing it leads to some weirdness like “cat file.txt” printing out the man page for cat 75% of the time. md5sum / sha256sum also never work if the file contents are remotely non-trivial. so that seems to be the ceiling.. at least for this prompt :p [0] https://news.ycombinator.com/item?id=33847479 https://news.ycombinator.com/item?id=33847479
- LoganDark 4y ago
  It's really too bad that consumers won't see versions of these models for a really long time. OpenAI is so stingy with their models. It's for a good reason, but damn do I hate all the hoops they make you jump through for something so restricted. No, I don't have a phone number. Guess I won't be able to do anything. Time to go back to NovelAI.
awesomelvin 4y ago
You should really think about the ethics of using this. You could wait a few minutes before executing your script. Some people work really hard to get on the leaderboard and players like you destroy the concept.
- colordrops 4y ago
  What's the concept that this player destroyed?
- charcircuit 4y ago
  People who place high on the leaderboard already use prewritten libraries specialized for the kinds of problems advent of code asks. Calling out to gpt3 is not much different.
  - surfsvammel 4y ago
    I am not so sure. The ones that I am following, and watching, does not have very elaborate libraries.
- forty 4y ago
  My understanding is that the ranking of this game is more tied to the timezone you live in (since the ranking depends on the hour when you solve the problem), than anything else. Also it sounds like it is more targeted to people who want to have fun solving the problems than to people who want to compete for ranking. And last, until their rules say otherwise, I'd say using AI to solve the problems is perfectly fair.
  - nindalf 4y ago
    It's a game that can be played both casually and competitively, like many other games. But using artificial aids would be frowned upon even in casual contexts. I would be annoyed if my opponent in chess got suggestions from a chess AI or if my opponent in an online shooter was using an aimbot. I play both of these for fun, but a big part of the fun is competing with people of similar skill. I've never placed in the top 100 of AOC and I never will. But the folks who attempt it would be rightly miffed that someone cheated to get to the top. And please, don't talk about the effort needed to set up the cheating device. It took a lot of effort to (allegedly) cheat in chess with anal beads, but no one is praising that person.
fulafel 4y ago
Is there a record of the answer attempts it made? How many tries did it make?
- maximilianroos 4y ago
  No, unfortunately I didn't keep the logs. Very open to contributions to the code to do that!

pm2222 4y ago

Would anyone trust the code by machine?

decap 4y ago
I wouldn't, but I can imagine a world where extensive randomized test suites are created by a specialized model from a text specification written by a human, and another model writes code to pass the tests with the given specification.
- boppo1 4y ago
  This still seems like it could leave open some holes. Wouldn't want to run this with financial, aerospace, medical or other critical applications.
- camdenreslink 4y ago
  Sounds like property based testing. Maybe the humans will tell the AI to write code according to some contract or signature. And then the human will also write the tests. But the AI will do the actual implementation.
maximilianroos 4y ago
Do you trust the code by humans?
- khazhoux 4y ago
  A snark comment that is entirely on point. Well done :-)
- aenis 4y ago
  What gpt-3 creates usually looks 'competent'. Junior human devs usually leave a visible mess in the code, and errors are easy to spot. With the gpt-3 solutions to complex-ish problems, not so much, one needs to read line by line to spot super weird errors that no competent programmer would ever plant. E.g.: I asked it to implement quicksort in x86 asm, but increment every sorted element by 5. It did nearly all of that right, except that it replaced every sorted element with 5. The code looked great otherwise, concise and even commented. I pointed out the mistake and it promptly agreed with me and fixed it. At this point I freked out and asked it to write about Dickens using adjectives that start with the letter 'p'. Which it (mostly) did. Good lord.

bhaney 4y ago

Watching the global leaderboard for the next few days will be fun. I assume on days where this script produces a correct solution, the leaderboard will immediately fill up after the problem drops, since many people will probably try using this.

anon7725 4y ago

This weekend feels immense. I have been prompting it to write and modify code, translating it back and forth to different languages. Sometimes it messes up, but it's astonishing how often it doesn't. This is the advent of something really earth-shattering. And I say that as a skeptic of most of these deep learning things.

fathrowaway12 4y ago
It really is amazing. Things it did in less than 10 seconds from hitting enter: - opengl raytracer with compilation instructions for macos - tictactoe in 3D - bitorrent peer handshake in Go from a paragraph in the RFC - http server in go with /user, /session, and /status endpoints from an english description - protocol buffer product configuration from a paragraph english description - pytorch script for classifying credit card transactions into expense accounts and instructions to import the output into quickbooks - quota management API implemented as a bidirectional streaming grpc service - pytorch neural network with a particular shape, number of input classes, output classes, activation function, etc. - IO scheduler using token bucket rate limiting - analyze the strengths/weaknesses of algorithms for 2 player zero sum games - compare david hume and immanuel kant's thoughts on knowledge - describe how critics received george orwell's work during his lifetime - christmas present recommendations for a relative given a description of their interests - poems about anything. love. cats. you name it. Blown away by how well it can synthesize information and incorporate context
- nerdponx 4y ago
  I never considered prompting it to write code to fit a machine learning model. This could be a tremendous time and effort saver in data science and research that requires statistical analysis. Until the last week or so, I've treated all this AI text and code generation as basically a toy, but I am starting to feel like it might become an important tool in industry in the next couple of years.
  - cma 4y ago
    > write code to fit a machine learning model That's against the EULA if OpenAI may want to make a similar model: > (iii) use the Services to develop foundation models or other large scale models that compete with OpenAI; https://openai.com/api/policies/terms/ https://openai.com/api/policies/terms/ Seems to be about developing models and not just restricting you from training them with it.

surfsvammel 4y ago

I saw this yesterday. It’s very impressive, and a bit scary. I myself set my alarm early every morning (problems are dropped at 06:00) in my TimeZone. It would be nice to only compete with other humans, but, I do understand that it would be impossible to enforce without totally changing AoC. I hope that we won’t see the competitive element of AoC disappear over the next couple of years as AI models get better and better at getting on the leaderboard. I’m not bitter, and I don’t want to stand in innovations way. I’m just a bit nostalgic and want AoC to stay the same:)

jodrellblank 4y ago
Earlier I was wondering what AoC would be like if it got rid of the time and instead the challenge was that you only get one chance to submit and get the right answer; what would people do differently in choice of language, test suites, fuzzing, reading, etc. But it seems GPT aced that as well.
- jchmrt 4y ago
  Why do you think GPT aced that? As far as I can tell from the code it sends in up to 10 guesses.
  - surfsvammel 4y ago
    If you submit a wrong answer, you cannot submit again in 60 seconds. I think that back-off time increases with every wrong submission. The AI has a finish time of 10s, meaning that it only submitted once (unless it is using multiple accounts to get around the back-off)
colonwqbang 4y ago
I'm expecting that the model will struggle once the problems start to get harder. The first few days of AOC are very trivial. To keep AoC competitive we would just need to start at a slightly higher difficulty.
- 4y ago

gfd 4y ago

I think my only issue with using this is that it forces everyone to use gpt if they want to remain competitive. Which is already the case for many other tools, for example people had to learn z3 smt solvers for previous years. But the main difference here is that there's no differentiation between the top competitors anymore. It just means in addition to being very good, you still need to run this thing in the background to hedge against this being an easily gpt-able problem while you code up the real solution. And in the cases where it is gptable, you can expect the leaderboard to be ranked only based on latency to openai/aoc servers.

charcircuit 4y ago
>there's no differentiation between the top competitors anymore At least in the short term, crafting the right or best prompt will be a differentiator.
- scruple 4y ago
  I came to a similar realization today after I had spent some time with ChatGPT. I feel for people that are currently struggling with search engine queries. They're woefully unprepared for this next evolution. edit/ I also wonder what happens to non-native English speaking programmers when tools like this are widely available.
  - ncallaway 4y ago
    It probably depends on how good the tools are. I used Google translate to translate a query into Portuguese, and then provided the translation to ChatGPT, and it seemed to handle it just fine. > Por favor, escreva um programa em python que encontre os primeiros 100 números primos e os eleve ao quadrado. --- >> Para encontrar os primeiros 100 números primos e elevá-los ao quadrado, você pode usar a função isprime do módulo sympy para verificar se um número é primo e a função pow do módulo math para elevá-lo ao quadrado. import sympy import math # Criamos uma lista vazia para armazenar os números primos primos = [] # Usamos um contador para limitar o número de números primos que coletamos contador = 0 # Iniciamos o loop em 2, pois 1 não é primo for i in range(2, 100): if sympy.isprime(i): # Se o número é primo, adicionamos à lista primos.append(i) # Incrementamos o contador contador += 1 # Verificamos se já temos 100 números primos if contador == 100: break # Elevamos cada um dos números primos ao quadrado quadrados = [math.pow(x, 2) for x in primos] # Imprimimos a lista de números primos elevados ao quadrado print(quadrados) >> Isso deve imprimir uma lista com os primeiros 100 números primos elevados ao quadrado. Though, interestingly, I notice that it has muddled what started as "first 100 prime numbers", into both all primes below 100 and the first 100 primes (if there were...somehow more than 100 primes below 100). I wonder if that's more an artifact of the Google Translation, or GPT itself.

deleted 4y ago

[deleted]

shreyshnaccount 4y ago

Saw someone else on HN asking how people manage to do these in seconds. Ofcourse they used gpt3

TrustPatches 4y ago

I got it to solve day 1 while playing with it yesterday, not shocked at all that this happened. Wonder if they'll try to prevent this somehow to preserve the competitive nature of AoC...

timjver 4y ago
That's probably an impossible task. The best they can do is ask contestants nicely not to do this, but that opens the can of worms of whether tools like GitHub Copilot should not be allowed, either.

TheRoque 4y ago

Meanwhile, I can't make her do a script to open 4 equally sized tmux panes

sitkack 4y ago
Needs more training data. Either the model can't understand the documentation ... oh wow, I have never tried scripting tmux. https://stackoverflow.com/questions/5447278/bash-scripts-with-tmux-to-launch-a-4-paned-window https://stackoverflow.com/questions/5447278/bash-scripts-wit... This looks like the best solution to your question https://github.com/tmuxinator/tmuxinator https://github.com/tmuxinator/tmuxinator

sputr 4y ago

Anyone else feel dread over this? I gave up on AoC knowing this would happen. And I don't even care about the competitive part. All the AI news lately has made me ever more uneasy, like for the first time in my life I actually feel like the future of being mad entirely obsolete by AI may actually happen in my lifetime. I've started avoiding AI news.

boudin 4y ago
It depends what's the purpose behind what you do. Writing software is a fun and creative puzzle solving process, AI will never replace that. It's the same with all AI generated content. It's impressive to see, but if you create things because you like doing so, AI can't steal that from you. It can be one of the many tools that helps creating things though and as such it can have an impact more on the business side.
- TapWaterBandit 4y ago
  What AI is doing is the same thing other advances that have made things easier have done. Previously the differentiator between who could build software was who understood the coding process. But now at least to some level the differentiator is going to be do you have ideas for things to create and implement. Can you think of a design to do it and have a good eye for beauty etc.
minism 4y ago
Definitely. And I've been largely skeptical about overhyped AI promises in the last 5 years. I'm feeling very uneasy and not sure what to make of all of this.
jstanley 4y ago
If you don't care about the competitive part of AoC, why does this make you give up on it? Seems like if you don't care about the competitive part, this wouldn't affect you in any way?
misja111 4y ago
Yes, this could be the beginning of the end of the competitive part of AoC. And maybe even of all online programming contests. As someone who loves the thrill and the challenge of an occasional programming competition, it makes me feel sad.

null_object 4y ago

My main problem with AI, is that all around me in this world I still see grinding poverty, war, starvation, suffering, diseases, deadly pollution, insanitary living conditions, underfunded schools and healthcare. What AI has ‘solved’ so far: almost all the activities that humanity has found fun and life-fulfilling for hundreds, if not thousands, of years - creativity, art, writing, solving intellectual challenges. As we accelerate ever-faster into a future dystopia of overlords (the richest 100 people in the world increased their collective wealth by 9% in 2021) while the rest of us are ground ever further into the dirt, I assume that AI will continue to steal the fulfilling parts of our lives from us, and leave us with the suffering and slavery.

flumpcakes 4y ago
This is a very pessimistic viewpoint. I really don't have any evidence to the contrary though. That's worrying.
Schroedingersat 4y ago
Nah, they'll create a convincing enough boot licking machine and then release the murder drones.
jstx1 4y ago
Those problems are real but completely unrelated to AI so I don't get how they're relevant here. I don't understand the expectation that AI should solve a specific set of complex economic and political problems, or that humans should solve those problems before they do anything with AI, neither of these ideas makes sense.
- foxes 4y ago
  The tech is ahead of social development.
  - _theory_ 4y ago
    as usual
- 4y ago

damip 4y ago

For some reason, it is unable to solve this query. All code produced gives wrong results, and it didn't correct it when given clues: Write a python program that efficiently computes the probability that a random permutation of r ones and w zeros contains k consecutive ones.

xdavidliu 4y ago
this is more of a pencil and paper problem rather than a numerical one though.

blux 4y ago

I'm wondering, when will programmers be replaced by AI? Or less strong, when is the fun, creative job of programming replaced by entering queries for some AI code bot? I've always argued that that time is still a long way off, but seeing these advanced makes me less confident...AI is evolving so so rapidly. I hope I can make it to my pension as an old-school programmer!

seydor 4y ago
Never. Programming is the job of automating yourself, and it has been happening for decades. Someone will be needed to enter these queries and orchestrate things
- misja111 4y ago
  But will that someone still be a programmer? Or will it be a business analyst?
  - scajanus 4y ago
    "Programmer-archeologist", to borrow a term from Vernor Vinge. I imagine the job would entail refining the prompts and doing some work yourself to chain together modules or legacy software, sorta like splitting the tasks with the AI assistant. Pair-coding with the AI, if you will.
  - seydor 4y ago
    'programmers' nowadays are Api-puzzle solvers.
machiaweliczny 4y ago
I like effects of my job and focus. So I welcome that change. I think people skilled in philosopy and quick-lerners will excel with this tech
yunwal 4y ago
> Or less strong, when is the fun, creative job of programming replaced by entering queries for some AI code bot? Maybe this is just me, but I’ve never thought of writing code as the creative part of programming. Defining business logic, UX, modeling objects, design, architecture, that’s where the fun is. I would be thrilled if an AI could handle things like figuring out a SAAS provider’s API syntax.

golergka 4y ago

I'm doing a similar experiment manually, and it doesn't work so good — in my case, it got stuck on day 2 even then I started providing hints. https://github.com/golergka/advent-of-code-2022-with-chat-gpt https://github.com/golergka/advent-of-code-2022-with-chat-gp...

hectorlorenzo 4y ago

I wonder if creating code snippets is missing the point. I've just imagined a future in which programmers create lambda-style functions that are essentially GPT prompts and pipe them to process data. As a programmer, you would essentially write tests to ensure prompts are working as expected. Maybe this is the ultimate no-code programming environment: a fleet of cells, each one containing one prompt, parsing and storing data.

hectorlorenzo 4y ago
Maybe you do not even need to write tests: they are just additional cells with prompts aimed at generating fuzzy testing against logic cells.
yunwal 4y ago
From what I can tell, no one has figured out how to manage user-input into an AI. Users seem to be able to circumvent the original owners intentions. As long as this remains a problem I think it will need to be compiled.

152334H 4y ago

I'm interested in finding out which day it'll start to break. GPT-3/ChatGPT are not perfectly accurate programmers, and some of the longer problems with more fluff text might distract it from the problem. I would be really surprised if it solved up to day 10 with no mistakes.

exoji2e 4y ago
I mean, it didn't really do well on day2, right? At least no fully automatic solution, since first star was after 1:49. Possibly not as many people tried doing it then though. But the statement was also more complicated (and a bit strange - giving you points both for winning and for the move you did).

infinitezest 4y ago

As a professional developer, this recent news is really depressing. My livelihood appears set to evaporate.

Avalaxy 4y ago
Worry not. As long as our stakeholders are not able to formulate unambiguous, complete and correct specifications, we will always have a job.
- misja111 4y ago
  Sure but you don't need a CS degree for that. Actually, CS hardly teaches you anything useful in that area at all. The job you're describing exists already, it is the job description of a business analyst.
  - yunwal 4y ago
    Idk about you, but my software development course (which was a req for my CS major) had us writing user stories, etc.
- tylerchurch 4y ago
  Indeed. In my view making informed trade-offs, communicating with stakeholders, and tracking down bugs based on limited information are hugely important job duties, and are entirely untouched by GPT3. People have decried “no code” and “low code” in the same way, but I see tons of job postings for people wanting an expert to drive or refactor these systems for them. At worst, GPT3 will be a tool that aids programmers, it won’t replace them.

civilized 4y ago

For anyone who is wondering what "Placing #1" means, it seems to be simply about being fast: https://adventofcode.com/2022/leaderboard https://adventofcode.com/2022/leaderboard > Below is the Advent of Code 2022 overall leaderboard; these are the 100 users with the highest total score. Getting a star first is worth 100 points, second is 99, and so on down to 1 point at 100th place. Not to minimize what's going on, but solving small, relatively easy programming puzzles fast is the sort of thing we might expect this generation of AI to excel at.

misja111 4y ago

Wow, that's incredible! GPT-3 is an amazing tool and it's great to see it being used to create top-level solutions. It's amazing to see the power of machine learning and AI being used in this way. Great job! Disclaimer: this comment was generated by beta.openai.com

dougdonohoe 4y ago

As an experiment, yesterday I tried solving Day 2 (rock/paper/scissors) using ChatGPT. The initial code was pretty close, but had 3 bugs (which I listed in my tweet stream https://twitter.com/DougDonohoe/status/1599126915251834882 https://twitter.com/DougDonohoe/status/1599126915251834882). I've been amazed at how quickly it spits out a solution that is 99% of the way there. This is what I do for a living. It doesn't quite scare me as most of what we do is more complicated than a well-defined programming problem. I feel like this might be another tool in the tool belt, helpful for accelerating my development time. Most of the time I know precisely what I want to do, but don't know all the best ways to do it in the language I happen to be using (e.g., Go or Scala or bash).

nerdponx 4y ago
I wonder if programming language design can somehow be optimized for code generation by AI. But then of course you have the problem where the AI doesn't "know" the new language. It also makes me wonder about the limits of AI code generation in less popular languages. I suspect that GPT knows a lot less about Scheme, and even less about differences among implementations, than it does about Java. Maybe an ideal "code bot" should include IRC logs in the training data.

adverbly 4y ago

Day 4 is very similar to as simple interview question I used to ask sometimes: > Can you provide some python code to check if there is a calendar conflict between two events I give this prompt to the chat and got this back: # First, define the start and end times for each event event1_start = datetime.datetime(2022, 12, 4, 12, 0, 0) event1_end = datetime.datetime(2022, 12, 4, 13, 0, 0) event2_start = datetime.datetime(2022, 12, 4, 13, 30, 0) event2_end = datetime.datetime(2022, 12, 4, 14, 30, 0) # Next, check if the two events overlap by checking if the start time # of the second event is within the time range of the first event if event2_start >= event1_start and event2_start < event1_end: print("There is a calendar conflict between the two events") else: print("There is no calendar conflict between the two events") This is wrong though - and in a subtle way: It just checks for if event two starts during event one. It doesn't check if event one starts during event two. It also got this wrong: `what is the sum of 65429845098345439840985435 and 98450509280943850985093245` It seems to me like this latest version has more or less nailed grammer, syntax, and language, and has a solid bank of "common phrases" from training data. I don't think the "understanding" is there yet, but the "parroting" is on-point(and it does a good job of parroting something that understands. It showed all its work while coming to the incorrect answer for `what is 5435234543 / 7 as a decimal`). Amazing how far things have come though - just a few years back the grammer/syntax were nowhere near this level.

savanaly 4y ago
What happens if you point out those problems and ask it how it would solve them? Which is after all the courtesy we would extend a human in an interview.
- Zanni 4y ago
  In my (limited) experience, playing with poetry, not code, it either 1) asserts that it's solution is correct, or it gives a slightly modified response and claims it has solved the problem. Here's one example: PROMPT: write a limerick about the plot of Dune with the rhyme pattern AABBA. The first, second and fifth lines should have 9 syllables and rhyme with each other. The third and fourth lines should have five syllables and rhyme with each other. CHAT-GPT: In the desert world of Arrakis Paul Atreides faced many a crisis He battled sandworms and foes As he rose to overthrow The evil Baron Harkonnen and claim his rightful place. PROMPT: your limerick doesn't rhyme. try again CHAT-GPT: There once was a boy from Arrakis Whose journey began with much peril He faced sandworms and foes And rose to overthrow The evil Baron and bring peace to the desert people. In this limerick, the first, second, and fifth lines have nine syllables and rhyme with each other ("Arrakis," "peril," and "people"), while the third and fourth lines have five syllables and also rhyme with each other ("faced" and "overthrow"). This follows the traditional limerick rhyme scheme, which is AABBA.

mk_stjames 4y ago

In other AoC there have been questions with little diagrams, or input with ASCII that forms essentially little boxes and matrices... I have a feeling that GPT will fail miserably at those type of questions and seeing essentially a 'picture' in ASCII on multiple lines is another level of abstraction. I think GPT-solves would be curtailed by having a graphic in an actual image describing part of the instructions pictorially (in a way where even OCR won't help). That will stave off the AI for another year or two....

mk_stjames 4y ago

Something that I watch every year with AoC is the drop-off curve of completed days as the days go by. Typically, there is a super sharp dropoff in players at first, that kinda levels out near the end to just the people are are really in it for the long haul... In previous years only about 5% of the initial players are paying by Christmas Eve. But if tools like Copilot are actually useful in solving harder problems (not automatically like today's GPT solve, but as a tool for even experienced users to help flesh out ideas) then I wonder if that dropoff curve will change, allowing more people to go further than before. Somehow I don't think so.

xyzzy_plugh 4y ago
From personal experience, the drop off isn't due to difficulty, but rather "free" time as we get deeper into the holiday season -- travel, family, disconnecting, etc.
- greenpeas 4y ago
  The difficulty certainly plays a role in how much "free" time is needed to participate. If early problems can be solved with a quick 10 minute script, later problems take the top 100th person about 1-2 hours -- which means it takes mortals like me about 4-8 hours to solve. That's quite the difference, and is due to the problem difficulty.

pluijzer 4y ago

First of all,.. incredible. I would not think this would become possible so fast. Also ofcourse as a programmer a little worrisome. Though I do not worry that much yet. We are talking about finding a solution an incredibly well written unambiguous specification. Including examples and a known solution to test against to boot. The specification here is just a program using English as its programming language. Also we are talking about a puzzle specifically made to be solved by computers. They are not real world problems (I don't really buy that I am actually helping some elves to be honest). If we want to replace programmers with AI somebody will have to write these specifications. And why do it in a verbose and ambiguous language like English, why not safe some effort and do in in say, Haskell or Java etc? Usually the input we get as programmers is more on the line as 'Make this faster', 'Make it stop crashing', 'Make it possible to do X'. Certainly AI can and will assist with this though I think it will still take some time before it replaces this.

fassssst 4y ago
You can probably write English way faster than you can write JavaScript, and it’s much easier to read.

xyzzy_plugh 4y ago

I wish AoC would just get rid of the global leaderboard. Not being on the west coast puts you at a severe disadvantage. I get that some folks enjoy the competitive aspect of it, and I enjoy the personal challenge of being timed, but I'd much rather start the challenge at a time that is convenient to me. Folks suggest "simply ignoring" the leaderboards, but the psychological effect is inescapable and the joy is sapped. At least for me.

andix 4y ago
It seems like the leaderboard was made obsolete today. Once there is an AI everybody can use to solve it automatically, there will be hundreds of submissions around the same time. Let’s see how many days those AIs can solve. Around day 10 AoC usually gets quite difficult.
- blairfrandeen 4y ago
  I'm sure we can find that out in the next 60 seconds if we want. All puzzles from previous years are available.
  - yunwal 4y ago
    But was chatGPT trained on their solutions?
greenpeas 4y ago
The solution is to have more contests throughout the year, with starting times spread out throughout the day (for different time zones). There are plenty of platforms (Codeforces, Codechef, Atcoder) that run regular (weekly/monthly) contests at various times. You can participate and replenish your sapped joy (though unless you train regularly, you will still be at a severe disadvantage) Alternatively, if AoC were to be run by a team (and not a single person), they could start at a different time each day.

FartyMcFarter 4y ago

This model is weird. People keep telling me it can solve complicated problems, and yet one of the first few games I tried to play with it was a pretty bad failure, even though it's really easy: My prompt: Two people are playing a game. The first player names an object. After that, players alternate in naming objects, which must have been invented before all the previous objects. First person: Smartphone. Second person: Car. First person: ChatGPT's answer: > Sorry, but that is not a valid response in this game. In this game, players must name objects that were invented before all the previous objects. Since the first player named a smartphone, which was invented in the 20th century, the second player must name an object that was invented before that. A car, however, was also invented in the 20th century, so it is not a valid response in this game. (End of ChatGPT text) It doesn't understand that two events in the 20th century happened at different times, so I can hardly believe this thing is intelligent.

andix 4y ago
That’s an interesting task it failed on. Yes it is not perfect (yet). But it is also astonishing what kind of tasks it succeeds doing. I give it a simply primary school math question (A and B drive at speed x, …). At some point it said something like „If A starts 10 minutes after B, A needs to start 22 minutes before B, to meet them at the goal“. Which is just nonsense.
ec109685 4y ago
Well, there were specific cars invented in the 20th century. You can help it out: Input: Two people are playing a game. The first player names an object. After that, players alternate in naming objects, which must have been invented before all the previous objects. For example the smartphone was invented in 1993 and the car was invented in 1986. Provide an example of correct output for this game. First person: Smartphone. Second person: Car. First person: Output: wheel Input: keep going for a while Output: First person: Fire. Second person: Hammer. First person: Paper. Second person: Pencil. First person: Printing press. Second person: Writing. First person: Language. Second person: Communication. Second person: Speech. First person: Thought. Second person: Consciousness. First person: Life. Second person: Evolution. Almost right.

darepublic 4y ago

Waiting on web developer to get automated away. Every crud website feels like reinventing the wheel again and again.

agent281 4y ago
Same with data engineering. Move data from point A to point B. Do these roll-ups.

deleted 4y ago

[deleted]

nswanberg 4y ago

Carl Öst Wilkens did something similar on Day 3: https://twitter.com/ostwilkens/status/1599026699999404033 https://twitter.com/ostwilkens/status/1599026699999404033 Cool hacks, both of them. Who's gotten ChatGPT to pontificate on the ethics or utility of people staying up late, using their mental energy to solve puzzles whose answers have no obvious utility? Would it then collapse into an introspective black hole, realizing its talents could be put elsewhere and becomes an EA maximalist, or finally conclude its very own power consumption could be put to better use powering ICU ventilators and power down? Also! I sort of question the spirit of these two bots. After solving the problem, neither went on to create elaborate visualizations of the problem!

andix 4y ago

I asked chat GPT to write a controversial comment to this link: While it's impressive that you were able to place 1st and 2nd using GPT-3 to write your code, it does raise some questions about the fairness of using such a powerful AI tool in a competition like Advent of Code. Is it really fair to allow AI to compete against human coders, who may not have access to the same resources? Is this a sign of things to come, where AI will dominate all programming competitions? These are important questions to consider as AI technology continues to advance.

drivers99 4y ago
Haha that was basically my initial thought (that it was unfair). But then, the criteria of Advent of Code is to come up with the answer in any way you can. If you care about getting on the scoreboard (only the first 100 people) then the speed matters. If you only want to get the gold stars, then speed doesn’t matter, only correctness. If you want to hone your problem solving skills or practice/learn a specific language then you will do that. (For example I did some in Forth, speed be damned.) So solving it with ChatGPT is in the spirit of the competition in some ways, but probably cheating to the people who want to score well on the competition, kind if like using a chess computer. But not as clear since obviously you can use a computer to get your answers.
- bcrosby95 4y ago
  I know you aren't specifically calling it cheating, but your comment triggered a thought about this. Let's go further down the rabbit hole for people looking to compete on the leaderboard: is it in any way cheating for the people that worked on ChatGPT to use it to solve Advent of Code problems? They did have a hand in developing the system. What if it was the work of one person? Would it be cheating if they used it? What about utility libraries that people have built up to help them solve advent of code problems so quickly? What about people that are using utility libraries that someone else built?
  - zamadatix 4y ago
    Any time I’ve seen a debate about what qualifies on cheating, regardless of the activity in question, it ultimately seems to come down to what the people playing think is fun competition and it’ll change depending who is playing. I think most don’t find letting some other system play for you to be fun competition unless they perceive the competition to be about building such systems. The same can be applied to utility libraries or any other aspect in question. Most often I’ve seen this question come up in speed running circles. Should we allow a save after the long intro save? Are autoclickers to avoid hand injury in rpg text dialogs fair? Is a certain action cheating or intended gameplay? Ultimately it’s a question of what the group currently playing thinks is the most fun to compete in, there is never an objective answer.

trhr 4y ago

Wait... somebody correct me if I'm wrong, but GPT-3 is a language processor. This isn't narrow AI like copilot where it's just probabilistic determining "Users like you may also enjoy closing their IO handles!" This is ingesting a fucking _storyline_ and getting to a result? Can it attend scrum standups for me yet?

nlitened 4y ago
It might soon start receiving your salary as well :)
- sparker72678 4y ago
  If my managers could document requirements with the same degree of detail required for GPT to produce a valid result, they wouldn't need me already. Writing code is the easy part.
  - nlitened 4y ago
    I think, documenting requirements beforehand with high degree of detail might not be required soon, if GPT can quickly re-write the code upon request given changed requirements.
    - sparker72678 4y ago
      Maybe? That assumes you can transition from one version of the code to another. And if GPT is going to do that for you, it seems like it's going to need a lot more context than it can currently hold on to. Like, a few orders of magnitude more? I guess that would depend on the problem size.

ta988 4y ago

Somehow it reminds me how advanced PCB makers still route by hand for high-frequency, radio or high-noise applications. But automated routers are still super useful for simple to medium jobs.

postalrat 4y ago

There are plenty of organizations out there that use leetcode to test for something. What are the testing for? Whatever it is it seems that AI has more of it than humans.

theptip 4y ago

This approach probably demolishes the take-home interview. Now, if you can figure out a procedure that would work for a live coding test, does that just mean the human+AI pair programming team is a good candidate? I think this probably fails on more open-ended / complex problems. Maybe that means interviews cannot use toy problems any more, and have to be more complex, longer, more realistic work samples? (Sounds great to me.)

yunwal 4y ago
Why does this demolish the at-home coding interview but Google (or even email) does not?

laweijfmvo 4y ago

Why happens if we ask it to find security vulnerabilities in the linux kernel source, for example?

catdevz 4y ago

Super lame. I can see how trying to get an AI to solve a problem could be cool, but actually submitting the answers and claiming a high spot on the leaderboard ruins the fun for others. I understand how some people don't care at all for the competitve aspect of AoC, but many of us do.