12 ms·

ChatGPT does Advent of Code 2023

185 points by luu 3y ago

inglor 3y ago
AoC questions this year were _deliberately written_ to be confusing to LLMs, it's not failing because it's worse it's failing because the questions were written to make it hader for models :] Edit: apparently not, the author is just really good at coming up with ai adverse puzzles. When testing ChatGPT did much better on last year’s puzzles.
- sanxiyn 3y ago
  That's fascinating! Where can I read more about it?
  - cormacrelf 3y ago
    [flagged]
    - usrme 3y ago
      The article actually links to a Reddit post[1] where the creator of Advent of Code says that 2023's puzzles weren't engineered to be more difficult for LLMs. --- [1]: https://old.reddit.com/r/adventofcode/comments/18bp8id/why_does_aoc_care_about_llms/kc646i4/ https://old.reddit.com/r/adventofcode/comments/18bp8id/why_d...
- sd9 3y ago
  Except the article says the opposite: Some people have even speculated that the problems this year were deliberately formulated to foil ChatGPT, but Eric actually denied that this is the case. Citing the author of AoC: Here are things LLMs didn't influence: The story. The puzzles. The inputs. I did the same thing this year that I do every year: I picked 25 puzzle ideas that sounded interesting to me, wrote them up, and then calibrated them based on betatester feedback. https://old.reddit.com/r/adventofcode/comments/18bp8id/why_does_aoc_care_about_llms/kc646i4/ https://old.reddit.com/r/adventofcode/comments/18bp8id/why_d...
- anonzzzies 3y ago
  I thought they were as well, because there were nuances which made them harder for LLMs. When I tried on 1 dec, gpt4 couldn't solve day 1 part 2. And I tripped over exactly the same thing ; when I parsed it correctly and then explained to the llm, it did solve it. But no where near as fast as me with my bag of horrible hacking aoc lib... It's interesting it turns out this year was not written with gpt in mind.
- adamors 3y ago
  Here's the creator of AoC saying the exact opposite on Reddit https://old.reddit.com/r/adventofcode/comments/18bp8id/why_does_aoc_care_about_llms/kc646i4/ https://old.reddit.com/r/adventofcode/comments/18bp8id/why_d... > Here are things LLMs didn't influence: > The story. > The puzzles. > The inputs. > I don't have a ChatGPT or Bard or whatever account, and I've never even used an LLM to write code or solve a puzzle, so I'm not sure what kinds of puzzles would be good or bad if that were my goal. Fortunately, it's not my goal - my goal is to help people become better programmers, not to create some kind of wacky LLM obstacle course. I'd rather have puzzles that are good for humans than puzzles that are both bad for humans and also somehow make the speed contest LLM-resistant. > I did the same thing this year that I do every year: I picked 25 puzzle ideas that sounded interesting to me, wrote them up, and then calibrated them based on betatester feedback. If you found a given puzzle easier or harder than you expected, please remember that difficulty is subjective and writing puzzles is tricky.
  - raincole 3y ago
    Good example of "the best way to get a correct answer on the internet is to post a wrong one."
    - fragmede 3y ago
      Poe's law strikes again!
  - maximus-decimus 3y ago
    > calibrated them based on betatester feedback. At least one betatester might very well have been using chatgpt.
  - imjonse 3y ago
    He may not have consciously written them to confuse LLMs but even without using any he probably knows they get more confused on reasoning tasks stated in not very clear text. At least for a few problem descriptions I couldn't help feeling that the statement was a lot more complex than it could have been and not only because of the story woven around it. Of course it could have been there to confuse the humans :)
- radres 3y ago
  ofc, chatgpt can do better in previous years given that solutions discussed all over the internet and could've train on them.
  - mtlmtlmtlmtl 3y ago
    Yeah there are literally Reddit megathreads with hundreds of people sharing their code.
- deleted 3y ago
  [deleted]
- danielbln 3y ago
  I think you should maybe be less confident in your statements if you actually don't really know. Highlighting _deliberately written_ when it was anything but.. well, that's confident bullshitting, LLM style, ironically.
- deleted 3y ago
  [deleted]
- blibble 3y ago
  > Edit: apparently not, the author is just really good at coming up with ai adverse puzzles. aka new unique problems that aren't in its training set
gumballindie 3y ago
"but it will get better" - person that parrots openai's pitch deck.
- criddell 3y ago
  It would be surprising if it didn’t, no?
  - gumballindie 3y ago
    It would, but it might take couple more winters.
  - tovej 3y ago
    It will get incrementally better at being a word calculator and looking up text, but LLMs aren't going to magically gain critical thinking skills or impactful intelligence capabilities.
    - randyrand 3y ago
      humans magically did. I can’t see why we would be special.
      - lewhoo 3y ago
        Then again, I think the opposite is (at this moment) equally unfalsifiable. Humans have on average 86 billion neurons, sperm whales have bigger brains and their average neuron count is 200 billion. African elephants have 257 billion neurons. There are some reasons to believe we are kind of special.
        gumballindie 3y ago
        To me it sounds as if ai workers are trying to oversimply how humans and brains generally speaking work to fit their narrative. A bit among the lines of "if my grandmother had wheels, she would be a bicycle". Emulating neurons doesn't make intelligence.
lancebeet 3y ago
My impression of the 2023 AoC was that Wastl has spent considerable effort on making it less LLM-friendly this year. Some days, this seems to have been done by adding extra conditions and complications which make the task more difficult for LLMs to parse. Other tasks required studying the input data which is difficult to achieve with an unsupervised LLM. Finally, the first couple of days seemed a lot more difficult this year than previous years, possibly to deter chatgpt users from filling up the leaderboard right away (though this year December started with a weekend which could also be a contributing factor).
- qsort 3y ago
  > considerable effort on making it less LLM-friendly this year Wastl himself denied that this is the case[1]. This is a lie. > Other tasks required studying the input data Always been the case for AoC, how is that different from other years? We have a clear example of a set of tasks that state-of-the-art LLMs cannot perform. We are doing science, for once. Why do we need to get into full conspiracy mode? [1] https://old.reddit.com/r/adventofcode/comments/18bp8id/why_does_aoc_care_about_llms/kc646i4/ https://old.reddit.com/r/adventofcode/comments/18bp8id/why_d...
  - lancebeet 3y ago
    I apologize. It was simply my subjective impression of the AoC this year, not a statement of objective fact. I didn't intend to insinuate that Wastl is a liar.
- bnprks 3y ago
  > Other tasks required studying the input data I also felt there were more problems than usual this year that could not easily be solved without looking at the input for special cases not alluded to in the problem descriptions. (As someone who has solved all 25 for the past 3 years). An extreme example was this year's day 20 circuit-simulating problem, which was made far easier by having the given circuit split up into a few independent chunks that are only connected at the start + end. (I suspect it might be NP-complete without this feature) It's a slightly different kind of problem solving to think "what makes this particular input easier than the general version of this problem", and one that I'd naively assume LLMs are less skilled at.
km3r 3y ago
Seeing the other comments, it still leaves me the question if this is a limit of scale or a limit of technology? How much more can LLMs scale up over the next decade?
- ChatGTP 3y ago
  Probably a lot considering people are pouring practically unlimited resources into their development and scaling?
  - WJW 3y ago
    This in itself proves nothing. Consider how much resources have been poured in making rockets better over the last few decades and yet we still can barely make it to Mars. It is also quite possible that there are fundamental limits to scaling LLMs that we have not discovered yet. You won't find a way to let airplanes fly to the moon no matter how many resources you pour into it. To make it there, you need better methods (ie rockets in this case) that look nothing like the first few steps to get off the ground.
    - robryan 3y ago
      LLMs fairly quickly got to consuming a significant portion of all available data. I can't see them improving significantly just by throwing more data at the problem.
deleted 3y ago
[deleted]
amne 3y ago
I have an observation not related to chatgpt but to the debugging skills mentioned in the article: indeed I've always felt that most teaching is done on perfect working code. I've never seen an exercise for developing debugging skills. For example: "This Djikstra implementation is finding the wrong path." and work with students to pinpoint the off-by-one error. I think it would reveal so much more about the implementation details than just explaining how it works. It could be its own topic to explore race conditions, edge case studies and so on.
- kypro 3y ago
  Debugging is by far the most difficult thing when working with AI algorithms because the output is rarely deterministic and that makes it hard to tell the difference between actual bugs and a bad selection of hyperparameters. You're also typically working huge arrays of numbers so unless you have some way to visualise what's happening it's also difficult to pin point where the algorithm is doing something it shouldn't be. My main criticism of learning about algorithms using a framework is that they handhold so much with optimisation techniques and accessible APIs that you can know very little about what's actually happening and build something that works. When writing AI algorithms just knowing how to implement in theory is rarely ever enough. The difficulty is always in the debugging and optimisation techniques.
Closi 3y ago
> I think it's as close to the perfect benchmark as you can get. Depends what you are benchmarking for... If you are benchmarking the ability of the solution to solve LEETCODE challenges, that is different to the ability of GPT4 to assist everyday programmers knock out business logic or diagnose bugs. My experience of GPT4 is that it's significantly better at the latter than GPT3.5. Additionally, the real test is for me is "Can an average programmer using GPT4 as a tool solve Advent of Code faster than an equally-skilled programmer without an LLM?".
- nirvael 3y ago
  >diagnose bugs Literally says in the article that GPT's main drawback is that it can't debug in the same way a human can.
  - okasaki 3y ago
    It can if you use the python interpreter.
  - Closi 3y ago
    Fixing bugs in code it writes itself is very different to diagnosing (i.e. identifying, not fixing) bugs in my own code. When stuck I often paste code into ChatGPT and ask why it doesn't work, and it will often help me quickly identify the error and propose a fix.
    - tnel77 3y ago
      I have done the same thing. “I am seeing the following behavior, but I expected <X>. What am I missing?” That line has often quickly solved some bugs that would have otherwise taken some time to debug. Very handy!
      - DaiPlusPlus 3y ago
        Can you share some examples of this? I haven’t had much luck with ChatGPT correctly identifying issues because (in my case, at least) they stem from other parts of a large codebase, and (last time I checked) I couldn’t paste more than a few kilobytes of code into ChatGPT. One example are bugs caused by precondition violations, which ChatGPT can’t diagnose without also being given the code to all of the incoming call-sites, which means you end-up solving the problem yourself before you’ve even explained the issue to ChatGPT - so (to me, at least) my use of ChatGPT is more akin to rubber-duck-debugging[1] than anything else. [1]: https://en.wikipedia.org/wiki/Rubber_duck_debugging https://en.wikipedia.org/wiki/Rubber_duck_debugging
JacobiX 3y ago
I believe this presents a good challenge, as the article mentioned: the problems are unique :) None of them are found in ChatGPT's training dataset.
lloydatkinson 3y ago
Of the few I attempted this year I actually used ChatGPT to help me decipher just what the hell the waffling, rambling, windy, unclear, and ultimately superfluous challenge requirements were. I didn’t have the solutions generated by ChatGPT to be clear, I used a prompt along the lines of “take this text and extract it’s requirements and generate bullet points, also make the example inputs and outputs clear”. I did the same for the previous year too, when ChatGPT hadn’t been out for long. I find that a lot more enjoyable and less tedious.
- genman 3y ago
  While not all people appear to appreciate your comment then I can relate to it. Not everyone enjoys this kind of story and world building and would like to get to the point already.
  - matsemann 3y ago
    People (including me) can sometime solve the tasks quicker than others can read them. Just read the sentence above the example, then skim the explanation below and go for it. It's a gamble sometimes, but it's entirely feasible to skip all the story (it often do contain hints, though).
    - genman 3y ago
      Looking at the scoreboard, it must have been like this as some people have submitted solutions way faster I could have humanly read all the story.
    - mtlmtlmtlmtl 3y ago
      This is how I read almost anything except fiction. Start at the end and work my way backwards until the conclusion makes sense.
  - deleted 3y ago
    [deleted]
scrapheap 3y ago
What would be really interesting would be to repeated attempt to get get ChatGPT to solve the problems. By that I mean try to solve the Day 1 problem at the point of release, then try to solve it in a fresh ChatGPT session the next day, and then get it to solve it again the day after that, and so for the next couple of months. Do that for each day that it runs for and then see what patterns emerge. Part of me expects it to get better at solving each problem as the rest of us write our solutions and then make them available on the web in some form for it harvest - but it would be interesting to see if that is the case.
- kypro 3y ago
  ChatGPT doesn't learn in real-time so in theory there should be no difference until a new model is released which has been trained on new data.
  - scrapheap 3y ago
    In theory, but what if we did see a difference :)
    - kolinko 3y ago
      No, it's practice. The model doesn't change day by day. Kind of like saying "let's watch the same youtube video every day and see if it changes" - it won't, that's not how youtube works, there is no point trying.
      - kevindamm 3y ago
        On the one hand, I suspect OpenAI does stealth updates of the model sometimes, or adjustments to the chat product. On the other hand, even move-fast-and-break-things mentality knows that some of December should be in production freeze, emergency-only updates.
        kolinko 3y ago
        Most people have that suspicion, myself included. But the API models stay verifiably same, and also OpenAI would have little reason to lie about not updating ChatGPT. Otoh, I can easily imagine helper/censorship models and chat’s system prompt being updated. System prompt doesn’t change capabilities much though, and the censor model behaviour doesn’t change the output, just cuts it off when it discovers copyright or other violations (the case of chat not being able to recite the Dune poem, DALLe not being able to produce certain images etc)
tisc 3y ago
What strikes me about ChatGPT is the blatantly wrong answers it can give. I asked ChatGPT to solve a augmented matrix using gaussian elimination, and it failed in this straightforward task spectacularly.
- deleted 3y ago
  [deleted]
- bradleykingz 3y ago
  [dead]
- danielbln 3y ago
  Confident sounding hallucinations have been a thing basically since it launched, it's not really a hot take (not that it has to be). If you subscribe to ChatGPT+ you can just ask GPT-4 to verify the approach by doing a web search, that often works quite well.
  - drivers99 3y ago
    Perfect example of "confident sounding hallucinations", I was just googling (a moment ago) why olive oil caused a burning sensation. It turns out there's a substance called oleocanthol[1] and there are receptors for that mainly in the throat. But while googling it, I see on Quora (which made it to the previews on Google) an "Assistant - Bot" response that is completely wrong: "Drinking olive oil can cause a burning sensation in the back of your throat due to its high fat content[...]"[2] It wouldn't be notable if someone specifically asked ChatGPT, knowing its limitation, but using it to automatically populate Quora and Google with it is pretty bad. People are using LLM to fill the web with BS. [1] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3073417/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3073417/ [2] https://www.quora.com/Why-does-drinking-olive-oil-burn-the-back-of-your-throat-Is-it-possible-to-drink-in-large-quantities https://www.quora.com/Why-does-drinking-olive-oil-burn-the-b...
    - jiggawatts 3y ago
      GPT4 answers this question correctly. I have no idea why in 2024 people are still lumping all LLMs together, as if they're all identical clones of the same thing. It's like saying vehicular travel will never work because you didn't like the handling of the cheapest car you can buy.
IanCal 3y ago
> ChatGPT never did this: its debugging skills are completely non-existent. If it encounters an error it will simply rewrite entire functions, or more often the entire program, from scratch. Well, it's going to need to rewrite functions to add debug due to not having edit capabilities, but I tried this and it absolutely added debug info which it then used to debug issues: In an inner part it added: debug_info.append({ 'Hand': hand, 'Bid': bid, 'Type': hand_info[0], 'Sorted Hand': hand_info[1], 'Rank': rank, 'Score': score }) I don't know quite what's happening but I feel like people constantly say it can't do something and the very first thing I try (just asking it to do the thing) usually works. I gave it the hands in the problem statement and the expected result, and the explanation as to why (copy pasted). It ran the code, looked at the debug output, identified the problem and rewrote the function. I'm not saying it immediately solved the problem, but it easily added debug information, ran the code, looked at the output and interpreted it.
- spacecadet 3y ago
  Same. I have pre-defined "comments", "error handling", "unit tests" and more, and I get it all. Plus if I target an area and ask for revision, i get just that... Tech perps both miss understand "AI" and think they have mastered it by default.
  - causal 3y ago
    "think they have mastered it by default" - yeah I see this a lot. I think its ease of use lures people into a false sense of mastery.
- deleted 3y ago
  [deleted]
- fragmede 3y ago
  taking HN comments that claim ChatGPT can't do a thing, pasting them into ChatGPT, and then linking the reply where it does the thing, usually doesn't get me any points, in case someone wants to build a bot that does that for points.
wokwokwok 3y ago
The comments about debugging make me sad. It shows that people fundamentally do not understand the tools they are using. Given a book of numbers, here are two tasks: 1) copy out the entire book, but replace every prime number with 7. 2) write down the list of prime numbers in the book. Which one is easier? LLMs have to generate tokens one at a time, and it’s very very difficult to perfectly generate a set of input tokens except for some tokens. Since you are almost certainly randomising the probabilities to some degree (that’s what temperature does), you’re also asking for both deterministic and random outputs. TLDR: ask LLMs what is wrong with the code. Ask for a diff. Don’t ask for an LLM to refactor, bug fix or annotate code… That’s extremely naive usage. Back to my stupid analogy: “please copy out this book, but fix the numbers which are ‘wrong’” I can hardly complain when I get terrible results can I?
- jprete 3y ago
  No-one fundamentally understands LLMs. It's inherent to the technology.
  - wokwokwok 3y ago
    Don't be daft. If you don't understand that an LLM generates output token-by-token, and that as a result of randomizing the token output probabilities that you cannot generate an error free copy of the input you've invested so little time in understanding as to be farcical. There's 'wow, these are complicated and I don't fully understand them' ...and there's, 'What this. It shiny. Not worky. Make some random change to prompt and pray to LLM gods' Come on, make an effort.
    - jiggawatts 3y ago
      I think you're being unfairly downvoted. People generally are terrible at using tools, especially tools that they perceive as "black boxes". I've seen an IT professional type this into Google: "Why did my PC crash?" I couldn't believe that after two decades of using web search technology, he still hadn't figured out how to extract value from a text index using specific and relevant key words. Similarly, very few developers know what a database index does or whether they need one or not. My pet theory is that NoSQL databases become popular because many of them automatically index every column, making it feel like a magic black box instead of an evil black box. LLMs are not only black boxes, but they're soooo fundamentally different and new that a lot of people are really struggling to wrap their brains around it. Just in this thread, today, there are people that are complaining about the direct equivalent of "my poorly thought out Google search didn't work, so Google is bad."
tromp 3y ago
> Problems start very easy on day 1 (sometimes as easy as just asking for a program that sums all numbers in the input) and progress towards more difficult ones, but they never get very hard: a CS graduate should be able to solve all problems, except maybe 1 or 2, in a couple of hours each. I think this wildly overestimates the programming skills of the average CS graduate. My estimate of the fraction of CS graduates able to do that is closer to 1%.
- brightball 3y ago
  I ran a private leaderboard for conference tickets this year and then surveyed participants afterwards. Earning a silver star took less than 1 hour each day for almost everybody on average. The gold star was tougher, between 2-5 on average. https://blog.carolina.codes/p/advent-of-code-ticket-winners https://blog.carolina.codes/p/advent-of-code-ticket-winners
  - thethirdone 3y ago
    Based on the leaderboard screenshot, only 7 people even solved up to day 12. The later days tend to be significantly harder than the early days. Based on top 100 leaderboard times, ~2x as hard for silver star.
    - brightball 3y ago
      Yea, it gets tougher for sure. Everybody who participated seemed to enjoy solving the problems. Many of them have said they plan to solve the rest over the next couple of months, just not on the contest timeline.
    - encomiast 3y ago
      Yeah this…the conclusion I would have drawn is not that the gold star took between 2-5 on average, but rather, 85% of people who self-select as enjoying a programming challenge where not able to finish more than half the tasks. Not that that's bad — seems slightly above par for the leaderboards I was involved in — just that the conclusion is really misleading.
boxed 3y ago
Claim chowder: > The model can unquestionably code. [https://news.ycombinator.com/item?id=38205052 https://news.ycombinator.com/item?id=38205052] Seems pretty clear most of the fantastic results from before was overfitting. Like every other case. It's amazing to me how these models can be caught red handed being overfitted again and again and again, and people don't get the memo.
- rideontime 3y ago
  When that "AI-generated Angry Birds clone" was doing numbers, didn't it turn out that most of the code came straight from tutorials for writing Angry Birds clones?
  - torginus 3y ago
    It might have, however a ton of gamedev related code (like giving adding physics to an object, and having it explode on contact with something else), is either formulaic, or can be glued together from pieces of other code, a feat ChatGPT is definitely capable of. It doesn't require it to develop a sophisticated model of the problem. And the above applies to a lot of prod code as well.
  - el-dude-arino 3y ago
    Yep, because that's all AI is; a parrot. For well-defined problems it works great... BECAUSE THEIR WELL DEFINED. Generative AI is only as good as the dataset that you give it, so for problems that exist in a heavily contrived and parameterized space (like leetcode-style problems) it works really well. But give it a novel problem with intertwining libraries and external dependencies along with custom type and structure definitions, it's going to fall flat. Humans do something AI can't, and that's draw from experience to apply a solution on a novel problem. This is why I'm not terribly worried about AI coming for engineer jobs anywhere in the near future. I use ChatGPT all the time to write me small functions, generate regular expressions, etc. Basically all the drudgery. I'd argue we've hit peak-AI at this point because from this point on, all datasets are going to be colored by AI-generated results. Generative-AI is now on a trajectory were it'll simply regress to the mean of knowledge.
alkonaut 3y ago
Are these problems unique enough that they couldn't have circulated before the AoC where they appear now? I notice that for "novel" code that CoPilot hasn't seen before, it's mostly useless, but when writing a hobby project which is a path tracer (of which there are probably 1000 implementations on GitHub) it's excellent. Which isn't surprising. It has seen the exact same function I'm writing, written 100 times in every language imaginable. There are books on the topic etc.
- monitron 3y ago
  I’m curious…if you’re writing a path tracer for fun, does using Copilot not take the fun out of it? If it’s already been done a thousand times, it feels like the point has got to be to write your version, otherwise why not just copy-paste or pull in a library? If you use Copilot are you not giving up the joy of building it with your own hands? What’s left at that point?
  - alkonaut 3y ago
    Good question. There is quite a lot of boilerplate in graphics, which is basically copy paste/book textbook lookup and ChatGPT is fantastic for this sort of thing. I wouldn't be able to deduce this, it would be a copy paste from github/stackoverflow/physics book/whatever. ChatGPT though is like a magic copy paste that takes the formula from github or the pbrt book and pastes it with my variable names. So you don't do paste and then update the names. // convert color from linear to sRGB let srgb = [some formula] // calculate wavelength dependent index of refraction let nw = [something] But yes, obviously somewhere between this and just saying "Create all the code I need" All the fun would disappear too. For the most part, that wouldn't work either. It would shit out 200 lines of code that doesn't work, and now it instead of taking the boring job of googling a formula from an optics reference replacing the variable names, it's now taking the role of shitty developer and made me the code reviewer. But see that's my day job and it's not what anyone wants.
    - monitron 3y ago
      Thanks so much for the thoughtful and insightful response!
ddp26 3y ago
> I don't pay for ChatGPT Plus, I only have a paid API key so I used instead a command line client, chatgpt-cli and manually ran the output programs. But this is much different, and likely much worse, than paid ChatGPT. Code Interpreter does its own debugging-and-revising loop. It's bizarre to me that this author wouldn't pay $20 one time to evaluate the higher quality product, the one most people would use if they cared about code quality, am I missing something?
- pera 3y ago
  I'm confused, is the gpt-4 model used through the API different to the gpt-4 from ChatGPT Plus? I can't find clear information on their website.
  - dartos 3y ago
    Kind of. ChatGPT is a product built on top of gpt-4. It includes many features that were built on top of the api (not as a a part of it) So it’s the same model, but the chatgpt product has more features than just text generation
    - pera 3y ago
      Interesting, do these extra features have an impact on code generation?
      - asabla 3y ago
        yes they do. And depending on the subject and/or what you're trying to do, it will have a larger impact the just talking to the model
        behnamoh 3y ago
        I stopped my ChatGPT Plus subscription and replaced it with pure GPT-4 API calls because the "product" built around the API just got dumber and dumber over time. - ChatGPT pros: Has some bells and whistles like code interpreter (which I can easily get via Open Interpreter). Has plugins (although I found web browsing to be inferior compared to Poe/Perplexity). - Pure GPT-4 API pros: Is not dumbed down or forced to "forget" things or be lazy in coding. I use the API either programmatically or through Poe.
deleted 3y ago
[deleted]
abbadadda 3y ago
[flagged]
- deleted 3y ago
  [deleted]
feverzsj 3y ago
[flagged]
- deleted 3y ago
  [deleted]
adverbly 3y ago
https://m.youtube.com/@mzikmund https://m.youtube.com/@mzikmund This creator has some excellent videos of chat GPT attempting advent of code. He uses it in a more generous format where he is often giving it multiple attempts and trying to coax the correct answer out of it. He definitely has more success with it than the article, but it is hard to tell how much of that success is due generous assistance and prompting, and so it is hard to know how much the model has actually improved year over year.
- notso411 3y ago
  [dead]
- fragmede 3y ago
  I won't spoil which one it eventually gets choked up on, but it does. But that's besides the point. If I had to sit down and do the Advent of Code challenges with and without ChatGPT's assistance, I know which one I'd choose. Programming's been forever changed, for better or worse, and pretending like it hasn't does no one any favors. I'm not doing LLM unassisted programming anymore, because it's a waste of my, and everyone else's time, and the industry needs to move accordingly.
orzig 3y ago
What prompt engineering do people use in real programming situations?
- deleted 3y ago
  [deleted]
jedberg 3y ago
I feel like ChatGPT is equivalent to an intern. I'd be curious to see how well interns do on AoC for comparison with the same level of help.

185 points by luu 3y ago

AoC questions this year were _deliberately written_ to be confusing to LLMs, it's not failing because it's worse it's failing because the questions were written to make it hader for models :] Edit: apparently not, the author is just really good at coming up with ai adverse puzzles. When testing ChatGPT did much better on last year’s puzzles.

sanxiyn 3y ago
That's fascinating! Where can I read more about it?
- cormacrelf 3y ago
  [flagged]
  - usrme 3y ago
    The article actually links to a Reddit post[1] where the creator of Advent of Code says that 2023's puzzles weren't engineered to be more difficult for LLMs. --- [1]: https://old.reddit.com/r/adventofcode/comments/18bp8id/why_does_aoc_care_about_llms/kc646i4/ https://old.reddit.com/r/adventofcode/comments/18bp8id/why_d...
sd9 3y ago
Except the article says the opposite: Some people have even speculated that the problems this year were deliberately formulated to foil ChatGPT, but Eric actually denied that this is the case. Citing the author of AoC: Here are things LLMs didn't influence: The story. The puzzles. The inputs. I did the same thing this year that I do every year: I picked 25 puzzle ideas that sounded interesting to me, wrote them up, and then calibrated them based on betatester feedback. https://old.reddit.com/r/adventofcode/comments/18bp8id/why_does_aoc_care_about_llms/kc646i4/ https://old.reddit.com/r/adventofcode/comments/18bp8id/why_d...
anonzzzies 3y ago
I thought they were as well, because there were nuances which made them harder for LLMs. When I tried on 1 dec, gpt4 couldn't solve day 1 part 2. And I tripped over exactly the same thing ; when I parsed it correctly and then explained to the llm, it did solve it. But no where near as fast as me with my bag of horrible hacking aoc lib... It's interesting it turns out this year was not written with gpt in mind.
adamors 3y ago
Here's the creator of AoC saying the exact opposite on Reddit https://old.reddit.com/r/adventofcode/comments/18bp8id/why_does_aoc_care_about_llms/kc646i4/ https://old.reddit.com/r/adventofcode/comments/18bp8id/why_d... > Here are things LLMs didn't influence: > The story. > The puzzles. > The inputs. > I don't have a ChatGPT or Bard or whatever account, and I've never even used an LLM to write code or solve a puzzle, so I'm not sure what kinds of puzzles would be good or bad if that were my goal. Fortunately, it's not my goal - my goal is to help people become better programmers, not to create some kind of wacky LLM obstacle course. I'd rather have puzzles that are good for humans than puzzles that are both bad for humans and also somehow make the speed contest LLM-resistant. > I did the same thing this year that I do every year: I picked 25 puzzle ideas that sounded interesting to me, wrote them up, and then calibrated them based on betatester feedback. If you found a given puzzle easier or harder than you expected, please remember that difficulty is subjective and writing puzzles is tricky.
- raincole 3y ago
  Good example of "the best way to get a correct answer on the internet is to post a wrong one."
  - fragmede 3y ago
    Poe's law strikes again!
- maximus-decimus 3y ago
  > calibrated them based on betatester feedback. At least one betatester might very well have been using chatgpt.
- imjonse 3y ago
  He may not have consciously written them to confuse LLMs but even without using any he probably knows they get more confused on reasoning tasks stated in not very clear text. At least for a few problem descriptions I couldn't help feeling that the statement was a lot more complex than it could have been and not only because of the story woven around it. Of course it could have been there to confuse the humans :)
radres 3y ago
ofc, chatgpt can do better in previous years given that solutions discussed all over the internet and could've train on them.
- mtlmtlmtlmtl 3y ago
  Yeah there are literally Reddit megathreads with hundreds of people sharing their code.
deleted 3y ago
[deleted]
danielbln 3y ago
I think you should maybe be less confident in your statements if you actually don't really know. Highlighting _deliberately written_ when it was anything but.. well, that's confident bullshitting, LLM style, ironically.
deleted 3y ago
[deleted]
blibble 3y ago
> Edit: apparently not, the author is just really good at coming up with ai adverse puzzles. aka new unique problems that aren't in its training set

gumballindie 3y ago

"but it will get better" - person that parrots openai's pitch deck.

criddell 3y ago
It would be surprising if it didn’t, no?
- gumballindie 3y ago
  It would, but it might take couple more winters.
- tovej 3y ago
  It will get incrementally better at being a word calculator and looking up text, but LLMs aren't going to magically gain critical thinking skills or impactful intelligence capabilities.
  - randyrand 3y ago
    humans magically did. I can’t see why we would be special.
    - lewhoo 3y ago
      Then again, I think the opposite is (at this moment) equally unfalsifiable. Humans have on average 86 billion neurons, sperm whales have bigger brains and their average neuron count is 200 billion. African elephants have 257 billion neurons. There are some reasons to believe we are kind of special.
      - gumballindie 3y ago
        To me it sounds as if ai workers are trying to oversimply how humans and brains generally speaking work to fit their narrative. A bit among the lines of "if my grandmother had wheels, she would be a bicycle". Emulating neurons doesn't make intelligence.

lancebeet 3y ago

My impression of the 2023 AoC was that Wastl has spent considerable effort on making it less LLM-friendly this year. Some days, this seems to have been done by adding extra conditions and complications which make the task more difficult for LLMs to parse. Other tasks required studying the input data which is difficult to achieve with an unsupervised LLM. Finally, the first couple of days seemed a lot more difficult this year than previous years, possibly to deter chatgpt users from filling up the leaderboard right away (though this year December started with a weekend which could also be a contributing factor).

qsort 3y ago
> considerable effort on making it less LLM-friendly this year Wastl himself denied that this is the case[1]. This is a lie. > Other tasks required studying the input data Always been the case for AoC, how is that different from other years? We have a clear example of a set of tasks that state-of-the-art LLMs cannot perform. We are doing science, for once. Why do we need to get into full conspiracy mode? [1] https://old.reddit.com/r/adventofcode/comments/18bp8id/why_does_aoc_care_about_llms/kc646i4/ https://old.reddit.com/r/adventofcode/comments/18bp8id/why_d...
- lancebeet 3y ago
  I apologize. It was simply my subjective impression of the AoC this year, not a statement of objective fact. I didn't intend to insinuate that Wastl is a liar.
bnprks 3y ago
> Other tasks required studying the input data I also felt there were more problems than usual this year that could not easily be solved without looking at the input for special cases not alluded to in the problem descriptions. (As someone who has solved all 25 for the past 3 years). An extreme example was this year's day 20 circuit-simulating problem, which was made far easier by having the given circuit split up into a few independent chunks that are only connected at the start + end. (I suspect it might be NP-complete without this feature) It's a slightly different kind of problem solving to think "what makes this particular input easier than the general version of this problem", and one that I'd naively assume LLMs are less skilled at.

km3r 3y ago

Seeing the other comments, it still leaves me the question if this is a limit of scale or a limit of technology? How much more can LLMs scale up over the next decade?

ChatGTP 3y ago
Probably a lot considering people are pouring practically unlimited resources into their development and scaling?
- WJW 3y ago
  This in itself proves nothing. Consider how much resources have been poured in making rockets better over the last few decades and yet we still can barely make it to Mars. It is also quite possible that there are fundamental limits to scaling LLMs that we have not discovered yet. You won't find a way to let airplanes fly to the moon no matter how many resources you pour into it. To make it there, you need better methods (ie rockets in this case) that look nothing like the first few steps to get off the ground.
  - robryan 3y ago
    LLMs fairly quickly got to consuming a significant portion of all available data. I can't see them improving significantly just by throwing more data at the problem.

deleted 3y ago

[deleted]

amne 3y ago

I have an observation not related to chatgpt but to the debugging skills mentioned in the article: indeed I've always felt that most teaching is done on perfect working code. I've never seen an exercise for developing debugging skills. For example: "This Djikstra implementation is finding the wrong path." and work with students to pinpoint the off-by-one error. I think it would reveal so much more about the implementation details than just explaining how it works. It could be its own topic to explore race conditions, edge case studies and so on.

kypro 3y ago
Debugging is by far the most difficult thing when working with AI algorithms because the output is rarely deterministic and that makes it hard to tell the difference between actual bugs and a bad selection of hyperparameters. You're also typically working huge arrays of numbers so unless you have some way to visualise what's happening it's also difficult to pin point where the algorithm is doing something it shouldn't be. My main criticism of learning about algorithms using a framework is that they handhold so much with optimisation techniques and accessible APIs that you can know very little about what's actually happening and build something that works. When writing AI algorithms just knowing how to implement in theory is rarely ever enough. The difficulty is always in the debugging and optimisation techniques.

Closi 3y ago

> I think it's as close to the perfect benchmark as you can get. Depends what you are benchmarking for... If you are benchmarking the ability of the solution to solve LEETCODE challenges, that is different to the ability of GPT4 to assist everyday programmers knock out business logic or diagnose bugs. My experience of GPT4 is that it's significantly better at the latter than GPT3.5. Additionally, the real test is for me is "Can an average programmer using GPT4 as a tool solve Advent of Code faster than an equally-skilled programmer without an LLM?".

nirvael 3y ago
>diagnose bugs Literally says in the article that GPT's main drawback is that it can't debug in the same way a human can.
- okasaki 3y ago
  It can if you use the python interpreter.
- Closi 3y ago
  Fixing bugs in code it writes itself is very different to diagnosing (i.e. identifying, not fixing) bugs in my own code. When stuck I often paste code into ChatGPT and ask why it doesn't work, and it will often help me quickly identify the error and propose a fix.
  - tnel77 3y ago
    I have done the same thing. “I am seeing the following behavior, but I expected <X>. What am I missing?” That line has often quickly solved some bugs that would have otherwise taken some time to debug. Very handy!
    - DaiPlusPlus 3y ago
      Can you share some examples of this? I haven’t had much luck with ChatGPT correctly identifying issues because (in my case, at least) they stem from other parts of a large codebase, and (last time I checked) I couldn’t paste more than a few kilobytes of code into ChatGPT. One example are bugs caused by precondition violations, which ChatGPT can’t diagnose without also being given the code to all of the incoming call-sites, which means you end-up solving the problem yourself before you’ve even explained the issue to ChatGPT - so (to me, at least) my use of ChatGPT is more akin to rubber-duck-debugging[1] than anything else. [1]: https://en.wikipedia.org/wiki/Rubber_duck_debugging https://en.wikipedia.org/wiki/Rubber_duck_debugging

JacobiX 3y ago

I believe this presents a good challenge, as the article mentioned: the problems are unique :) None of them are found in ChatGPT's training dataset.

lloydatkinson 3y ago

Of the few I attempted this year I actually used ChatGPT to help me decipher just what the hell the waffling, rambling, windy, unclear, and ultimately superfluous challenge requirements were. I didn’t have the solutions generated by ChatGPT to be clear, I used a prompt along the lines of “take this text and extract it’s requirements and generate bullet points, also make the example inputs and outputs clear”. I did the same for the previous year too, when ChatGPT hadn’t been out for long. I find that a lot more enjoyable and less tedious.

genman 3y ago
While not all people appear to appreciate your comment then I can relate to it. Not everyone enjoys this kind of story and world building and would like to get to the point already.
- matsemann 3y ago
  People (including me) can sometime solve the tasks quicker than others can read them. Just read the sentence above the example, then skim the explanation below and go for it. It's a gamble sometimes, but it's entirely feasible to skip all the story (it often do contain hints, though).
  - genman 3y ago
    Looking at the scoreboard, it must have been like this as some people have submitted solutions way faster I could have humanly read all the story.
  - mtlmtlmtlmtl 3y ago
    This is how I read almost anything except fiction. Start at the end and work my way backwards until the conclusion makes sense.
- deleted 3y ago
  [deleted]

scrapheap 3y ago

What would be really interesting would be to repeated attempt to get get ChatGPT to solve the problems. By that I mean try to solve the Day 1 problem at the point of release, then try to solve it in a fresh ChatGPT session the next day, and then get it to solve it again the day after that, and so for the next couple of months. Do that for each day that it runs for and then see what patterns emerge. Part of me expects it to get better at solving each problem as the rest of us write our solutions and then make them available on the web in some form for it harvest - but it would be interesting to see if that is the case.

kypro 3y ago
ChatGPT doesn't learn in real-time so in theory there should be no difference until a new model is released which has been trained on new data.
- scrapheap 3y ago
  In theory, but what if we did see a difference :)
  - kolinko 3y ago
    No, it's practice. The model doesn't change day by day. Kind of like saying "let's watch the same youtube video every day and see if it changes" - it won't, that's not how youtube works, there is no point trying.
    - kevindamm 3y ago
      On the one hand, I suspect OpenAI does stealth updates of the model sometimes, or adjustments to the chat product. On the other hand, even move-fast-and-break-things mentality knows that some of December should be in production freeze, emergency-only updates.
      - kolinko 3y ago
        Most people have that suspicion, myself included. But the API models stay verifiably same, and also OpenAI would have little reason to lie about not updating ChatGPT. Otoh, I can easily imagine helper/censorship models and chat’s system prompt being updated. System prompt doesn’t change capabilities much though, and the censor model behaviour doesn’t change the output, just cuts it off when it discovers copyright or other violations (the case of chat not being able to recite the Dune poem, DALLe not being able to produce certain images etc)

tisc 3y ago

What strikes me about ChatGPT is the blatantly wrong answers it can give. I asked ChatGPT to solve a augmented matrix using gaussian elimination, and it failed in this straightforward task spectacularly.

deleted 3y ago
[deleted]
bradleykingz 3y ago
[dead]
danielbln 3y ago
Confident sounding hallucinations have been a thing basically since it launched, it's not really a hot take (not that it has to be). If you subscribe to ChatGPT+ you can just ask GPT-4 to verify the approach by doing a web search, that often works quite well.
- drivers99 3y ago
  Perfect example of "confident sounding hallucinations", I was just googling (a moment ago) why olive oil caused a burning sensation. It turns out there's a substance called oleocanthol[1] and there are receptors for that mainly in the throat. But while googling it, I see on Quora (which made it to the previews on Google) an "Assistant - Bot" response that is completely wrong: "Drinking olive oil can cause a burning sensation in the back of your throat due to its high fat content[...]"[2] It wouldn't be notable if someone specifically asked ChatGPT, knowing its limitation, but using it to automatically populate Quora and Google with it is pretty bad. People are using LLM to fill the web with BS. [1] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3073417/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3073417/ [2] https://www.quora.com/Why-does-drinking-olive-oil-burn-the-back-of-your-throat-Is-it-possible-to-drink-in-large-quantities https://www.quora.com/Why-does-drinking-olive-oil-burn-the-b...
  - jiggawatts 3y ago
    GPT4 answers this question correctly. I have no idea why in 2024 people are still lumping all LLMs together, as if they're all identical clones of the same thing. It's like saying vehicular travel will never work because you didn't like the handling of the cheapest car you can buy.

IanCal 3y ago

> ChatGPT never did this: its debugging skills are completely non-existent. If it encounters an error it will simply rewrite entire functions, or more often the entire program, from scratch. Well, it's going to need to rewrite functions to add debug due to not having edit capabilities, but I tried this and it absolutely added debug info which it then used to debug issues: In an inner part it added: debug_info.append({ 'Hand': hand, 'Bid': bid, 'Type': hand_info[0], 'Sorted Hand': hand_info[1], 'Rank': rank, 'Score': score }) I don't know quite what's happening but I feel like people constantly say it can't do something and the very first thing I try (just asking it to do the thing) usually works. I gave it the hands in the problem statement and the expected result, and the explanation as to why (copy pasted). It ran the code, looked at the debug output, identified the problem and rewrote the function. I'm not saying it immediately solved the problem, but it easily added debug information, ran the code, looked at the output and interpreted it.

spacecadet 3y ago
Same. I have pre-defined "comments", "error handling", "unit tests" and more, and I get it all. Plus if I target an area and ask for revision, i get just that... Tech perps both miss understand "AI" and think they have mastered it by default.
- causal 3y ago
  "think they have mastered it by default" - yeah I see this a lot. I think its ease of use lures people into a false sense of mastery.
deleted 3y ago
[deleted]
fragmede 3y ago
taking HN comments that claim ChatGPT can't do a thing, pasting them into ChatGPT, and then linking the reply where it does the thing, usually doesn't get me any points, in case someone wants to build a bot that does that for points.

wokwokwok 3y ago

The comments about debugging make me sad. It shows that people fundamentally do not understand the tools they are using. Given a book of numbers, here are two tasks: 1) copy out the entire book, but replace every prime number with 7. 2) write down the list of prime numbers in the book. Which one is easier? LLMs have to generate tokens one at a time, and it’s very very difficult to perfectly generate a set of input tokens except for some tokens. Since you are almost certainly randomising the probabilities to some degree (that’s what temperature does), you’re also asking for both deterministic and random outputs. TLDR: ask LLMs what is wrong with the code. Ask for a diff. Don’t ask for an LLM to refactor, bug fix or annotate code… That’s extremely naive usage. Back to my stupid analogy: “please copy out this book, but fix the numbers which are ‘wrong’” I can hardly complain when I get terrible results can I?

jprete 3y ago
No-one fundamentally understands LLMs. It's inherent to the technology.
- wokwokwok 3y ago
  Don't be daft. If you don't understand that an LLM generates output token-by-token, and that as a result of randomizing the token output probabilities that you cannot generate an error free copy of the input you've invested so little time in understanding as to be farcical. There's 'wow, these are complicated and I don't fully understand them' ...and there's, 'What this. It shiny. Not worky. Make some random change to prompt and pray to LLM gods' Come on, make an effort.
  - jiggawatts 3y ago
    I think you're being unfairly downvoted. People generally are terrible at using tools, especially tools that they perceive as "black boxes". I've seen an IT professional type this into Google: "Why did my PC crash?" I couldn't believe that after two decades of using web search technology, he still hadn't figured out how to extract value from a text index using specific and relevant key words. Similarly, very few developers know what a database index does or whether they need one or not. My pet theory is that NoSQL databases become popular because many of them automatically index every column, making it feel like a magic black box instead of an evil black box. LLMs are not only black boxes, but they're soooo fundamentally different and new that a lot of people are really struggling to wrap their brains around it. Just in this thread, today, there are people that are complaining about the direct equivalent of "my poorly thought out Google search didn't work, so Google is bad."

tromp 3y ago

> Problems start very easy on day 1 (sometimes as easy as just asking for a program that sums all numbers in the input) and progress towards more difficult ones, but they never get very hard: a CS graduate should be able to solve all problems, except maybe 1 or 2, in a couple of hours each. I think this wildly overestimates the programming skills of the average CS graduate. My estimate of the fraction of CS graduates able to do that is closer to 1%.

brightball 3y ago
I ran a private leaderboard for conference tickets this year and then surveyed participants afterwards. Earning a silver star took less than 1 hour each day for almost everybody on average. The gold star was tougher, between 2-5 on average. https://blog.carolina.codes/p/advent-of-code-ticket-winners https://blog.carolina.codes/p/advent-of-code-ticket-winners
- thethirdone 3y ago
  Based on the leaderboard screenshot, only 7 people even solved up to day 12. The later days tend to be significantly harder than the early days. Based on top 100 leaderboard times, ~2x as hard for silver star.
  - brightball 3y ago
    Yea, it gets tougher for sure. Everybody who participated seemed to enjoy solving the problems. Many of them have said they plan to solve the rest over the next couple of months, just not on the contest timeline.
  - encomiast 3y ago
    Yeah this…the conclusion I would have drawn is not that the gold star took between 2-5 on average, but rather, 85% of people who self-select as enjoying a programming challenge where not able to finish more than half the tasks. Not that that's bad — seems slightly above par for the leaderboards I was involved in — just that the conclusion is really misleading.

boxed 3y ago

Claim chowder: > The model can unquestionably code. [https://news.ycombinator.com/item?id=38205052 https://news.ycombinator.com/item?id=38205052] Seems pretty clear most of the fantastic results from before was overfitting. Like every other case. It's amazing to me how these models can be caught red handed being overfitted again and again and again, and people don't get the memo.

rideontime 3y ago
When that "AI-generated Angry Birds clone" was doing numbers, didn't it turn out that most of the code came straight from tutorials for writing Angry Birds clones?
- torginus 3y ago
  It might have, however a ton of gamedev related code (like giving adding physics to an object, and having it explode on contact with something else), is either formulaic, or can be glued together from pieces of other code, a feat ChatGPT is definitely capable of. It doesn't require it to develop a sophisticated model of the problem. And the above applies to a lot of prod code as well.
- el-dude-arino 3y ago
  Yep, because that's all AI is; a parrot. For well-defined problems it works great... BECAUSE THEIR WELL DEFINED. Generative AI is only as good as the dataset that you give it, so for problems that exist in a heavily contrived and parameterized space (like leetcode-style problems) it works really well. But give it a novel problem with intertwining libraries and external dependencies along with custom type and structure definitions, it's going to fall flat. Humans do something AI can't, and that's draw from experience to apply a solution on a novel problem. This is why I'm not terribly worried about AI coming for engineer jobs anywhere in the near future. I use ChatGPT all the time to write me small functions, generate regular expressions, etc. Basically all the drudgery. I'd argue we've hit peak-AI at this point because from this point on, all datasets are going to be colored by AI-generated results. Generative-AI is now on a trajectory were it'll simply regress to the mean of knowledge.

alkonaut 3y ago

Are these problems unique enough that they couldn't have circulated before the AoC where they appear now? I notice that for "novel" code that CoPilot hasn't seen before, it's mostly useless, but when writing a hobby project which is a path tracer (of which there are probably 1000 implementations on GitHub) it's excellent. Which isn't surprising. It has seen the exact same function I'm writing, written 100 times in every language imaginable. There are books on the topic etc.

monitron 3y ago
I’m curious…if you’re writing a path tracer for fun, does using Copilot not take the fun out of it? If it’s already been done a thousand times, it feels like the point has got to be to write your version, otherwise why not just copy-paste or pull in a library? If you use Copilot are you not giving up the joy of building it with your own hands? What’s left at that point?
- alkonaut 3y ago
  Good question. There is quite a lot of boilerplate in graphics, which is basically copy paste/book textbook lookup and ChatGPT is fantastic for this sort of thing. I wouldn't be able to deduce this, it would be a copy paste from github/stackoverflow/physics book/whatever. ChatGPT though is like a magic copy paste that takes the formula from github or the pbrt book and pastes it with my variable names. So you don't do paste and then update the names. // convert color from linear to sRGB let srgb = [some formula] // calculate wavelength dependent index of refraction let nw = [something] But yes, obviously somewhere between this and just saying "Create all the code I need" All the fun would disappear too. For the most part, that wouldn't work either. It would shit out 200 lines of code that doesn't work, and now it instead of taking the boring job of googling a formula from an optics reference replacing the variable names, it's now taking the role of shitty developer and made me the code reviewer. But see that's my day job and it's not what anyone wants.
  - monitron 3y ago
    Thanks so much for the thoughtful and insightful response!

ddp26 3y ago

> I don't pay for ChatGPT Plus, I only have a paid API key so I used instead a command line client, chatgpt-cli and manually ran the output programs. But this is much different, and likely much worse, than paid ChatGPT. Code Interpreter does its own debugging-and-revising loop. It's bizarre to me that this author wouldn't pay $20 one time to evaluate the higher quality product, the one most people would use if they cared about code quality, am I missing something?

pera 3y ago
I'm confused, is the gpt-4 model used through the API different to the gpt-4 from ChatGPT Plus? I can't find clear information on their website.
- dartos 3y ago
  Kind of. ChatGPT is a product built on top of gpt-4. It includes many features that were built on top of the api (not as a a part of it) So it’s the same model, but the chatgpt product has more features than just text generation
  - pera 3y ago
    Interesting, do these extra features have an impact on code generation?
    - asabla 3y ago
      yes they do. And depending on the subject and/or what you're trying to do, it will have a larger impact the just talking to the model
      - behnamoh 3y ago
        I stopped my ChatGPT Plus subscription and replaced it with pure GPT-4 API calls because the "product" built around the API just got dumber and dumber over time. - ChatGPT pros: Has some bells and whistles like code interpreter (which I can easily get via Open Interpreter). Has plugins (although I found web browsing to be inferior compared to Poe/Perplexity). - Pure GPT-4 API pros: Is not dumbed down or forced to "forget" things or be lazy in coding. I use the API either programmatically or through Poe.

deleted 3y ago

[deleted]

abbadadda 3y ago

[flagged]

deleted 3y ago
[deleted]

feverzsj 3y ago

[flagged]

deleted 3y ago
[deleted]

adverbly 3y ago

https://m.youtube.com/@mzikmund https://m.youtube.com/@mzikmund This creator has some excellent videos of chat GPT attempting advent of code. He uses it in a more generous format where he is often giving it multiple attempts and trying to coax the correct answer out of it. He definitely has more success with it than the article, but it is hard to tell how much of that success is due generous assistance and prompting, and so it is hard to know how much the model has actually improved year over year.

notso411 3y ago
[dead]
fragmede 3y ago
I won't spoil which one it eventually gets choked up on, but it does. But that's besides the point. If I had to sit down and do the Advent of Code challenges with and without ChatGPT's assistance, I know which one I'd choose. Programming's been forever changed, for better or worse, and pretending like it hasn't does no one any favors. I'm not doing LLM unassisted programming anymore, because it's a waste of my, and everyone else's time, and the industry needs to move accordingly.

orzig 3y ago

What prompt engineering do people use in real programming situations?

deleted 3y ago
[deleted]

jedberg 3y ago

I feel like ChatGPT is equivalent to an intern. I'd be curious to see how well interns do on AoC for comparison with the same level of help.