4 ms·

What would be really interesting would be to repeated attempt to get get ChatGPT to solve the problems. By that I mean try to solve the Day 1 problem at the po

What would be really interesting would be to repeated attempt to get get ChatGPT to solve the problems. By that I mean try to solve the Day 1 problem at the point of release, then try to solve it in a fresh ChatGPT session the next day, and then get it to solve it again the day after that, and so for the next couple of months. Do that for each day that it runs for and then see what patterns emerge. Part of me expects it to get better at solving each problem as the rest of us write our solutions and then make them available on the web in some form for it harvest - but it would be interesting to see if that is the case.

kypro 3y ago
ChatGPT doesn't learn in real-time so in theory there should be no difference until a new model is released which has been trained on new data.
- scrapheap 3y ago
  In theory, but what if we did see a difference :)
  - kolinko 3y ago
    No, it's practice. The model doesn't change day by day. Kind of like saying "let's watch the same youtube video every day and see if it changes" - it won't, that's not how youtube works, there is no point trying.
    - kevindamm 3y ago
      On the one hand, I suspect OpenAI does stealth updates of the model sometimes, or adjustments to the chat product. On the other hand, even move-fast-and-break-things mentality knows that some of December should be in production freeze, emergency-only updates.
      - kolinko 3y ago
        Most people have that suspicion, myself included. But the API models stay verifiably same, and also OpenAI would have little reason to lie about not updating ChatGPT. Otoh, I can easily imagine helper/censorship models and chat’s system prompt being updated. System prompt doesn’t change capabilities much though, and the censor model behaviour doesn’t change the output, just cuts it off when it discovers copyright or other violations (the case of chat not being able to recite the Dune poem, DALLe not being able to produce certain images etc)
raincole 3y ago
We don't know how often ChatGPT is getting updated (internally), but I hightly doubt the answer would be daily.
- scrapheap 3y ago
  It doesn't stop us checking daily though does it :)
  - phatfish 3y ago
    It would be interesting to compare the solutions from day 1 to the solutions a year from now (maybe less, but a year would give time for most people that cared to publish their solution and for them to be picked up in re-training). Of course, an LLM will use the solutions that are in its training set. I asked an LLM to explain a pretty generic solution to one of last years AoC questions, and even with no (or very little) reference in the code to the fact it was an AoC solution the LLM used terminology from the question. It wasn't ChatGPT, i think it was Claude.ai. ChatGPT gave an explanation you would expect from just seeing the code without the context it was related to AoC.

kypro 3y ago

ChatGPT doesn't learn in real-time so in theory there should be no difference until a new model is released which has been trained on new data.

scrapheap 3y ago
In theory, but what if we did see a difference :)
- kolinko 3y ago
  No, it's practice. The model doesn't change day by day. Kind of like saying "let's watch the same youtube video every day and see if it changes" - it won't, that's not how youtube works, there is no point trying.
  - kevindamm 3y ago
    On the one hand, I suspect OpenAI does stealth updates of the model sometimes, or adjustments to the chat product. On the other hand, even move-fast-and-break-things mentality knows that some of December should be in production freeze, emergency-only updates.
    - kolinko 3y ago
      Most people have that suspicion, myself included. But the API models stay verifiably same, and also OpenAI would have little reason to lie about not updating ChatGPT. Otoh, I can easily imagine helper/censorship models and chat’s system prompt being updated. System prompt doesn’t change capabilities much though, and the censor model behaviour doesn’t change the output, just cuts it off when it discovers copyright or other violations (the case of chat not being able to recite the Dune poem, DALLe not being able to produce certain images etc)

raincole 3y ago

We don't know how often ChatGPT is getting updated (internally), but I hightly doubt the answer would be daily.

scrapheap 3y ago
It doesn't stop us checking daily though does it :)
- phatfish 3y ago
  It would be interesting to compare the solutions from day 1 to the solutions a year from now (maybe less, but a year would give time for most people that cared to publish their solution and for them to be picked up in re-training). Of course, an LLM will use the solutions that are in its training set. I asked an LLM to explain a pretty generic solution to one of last years AoC questions, and even with no (or very little) reference in the code to the fact it was an AoC solution the LLM used terminology from the question. It wasn't ChatGPT, i think it was Claude.ai. ChatGPT gave an explanation you would expect from just seeing the code without the context it was related to AoC.