3 ms·

I think this is a terrible analysis with a weak conclusion. There's zero mention of how long it took the LLM to write the code vs the human. You have a 300 sec

I think this is a terrible analysis with a weak conclusion. There's zero mention of how long it took the LLM to write the code vs the human. You have a 300 second runtime limit, but what was your coding time limit? The machine spat out code in, what, a few seconds? And how long did your solutions take to write? Advent of code problems take me longer to just read than it takes an LLM to have a proposed solution ready for evaluation. > they didn’t perform nearly as well as I’d expect Is this a joke, though? A machine takes a problem description written as floridly hyperventilated as advent problems are, and, without any opportunity for automated reanalysis, it understands the exact problem domain, it understands exactly what's being asked, correctly models the solution, and spits out a correct single-shot solution on 20 of them in no time flat, often with substantially better running time than your own solutions, and that's disappointing? > a lot of the submissions had timeout errors, which means that their solutions might work if asked more explicitly for efficient solutions. However the models should know very well what AoC solutions entail You made up an arbitrary runtime limit and then kept that limit a secret, and you were surprised when the solutions didn't adhere to the secret limit? > Finally, some of the submissions raised some Exceptions, which would likely be fixed with a human reviewing this code and asking for changes. How many of your solutions got the correct answer on the first try without going back and fixing something?

keyle 2y ago
You raise some good points about the "total of hours spent" but I guess you don't consider training time included. Also there is no need to quote the author's post and have a go at him personally. There are better ways to get your point across by arguing the point made and not the sentences written.
- BugsJustFindMe 2y ago
  > I guess you don't consider training time included In the same way that I don't consider the time an author spends writing a book when saying how long it took me to read the book. OP lost zero time training ChatGPT. Or do you mean how much time OP spent training themself? Because that's a whole new can of worms. How many years should we add to OP's development times because probably they learned how to write code before this exercise?
  - keyle 2y ago
    I was considering the energy used and $, R&D and time spent to train those models, which is colossal compared to the developer and factor that in. Then they don't look so impressive. For reference, I'm not anti-LLM, I use Claude and ChatGPT every day. I'm just raising those if you want to consider all facts.
    - lgas 2y ago
      That cost is amortized over all requests made against all copies of the model though. Fortunately we don't have to re-train the model every time we want to perform inference.
    - bawolff 2y ago
      That's a weird comparison. Would you also count the number of hours it took for someone to write the OS, the editor, etc? The programmer wouldn't be effective without those things.
    - menaerus 2y ago
      And now also consider the amount of time, school, university, practice and $$$ it took to train a software engineer. Repeat for the population of presumably ~30 million software engineers in the world. Time-wise it doesn't scale at all when contrasted to new LLM releases that occur ~once a year per company. $$$-wise is also debatable.
segmondy 2y ago
I did experiment with local LLM running on my machine, most solutions were generated within the time of 30-60seconds. My overhead was really copying part 1 of the problem, generating the code, copying and pasting the code, running it, copying the input data, running it. Entering the result, repeating for part 2 and for most of them that was about 5 minutes from start to finish. If I automated the process, it would probably be 1 minute or less for the problems it could solve. Not the OP, but I was able to zero shot 27 solutions correctly without any back and forth, and 5 more with a little bit back and forth. Using local models.
mvdtnz 2y ago
> it understands the exact problem domain, it understands exactly what's being asked, correctly models the solution It does not "understand" or "model" shit. Good grief you AI shills need to take a breath.
- BugsJustFindMe 2y ago
  I'll use a different word when you demonstrate that humans aren't also stochastic parrots constantly hallucinating reality. Until then, good enough for one is good enough for the other.

keyle 2y ago

You raise some good points about the "total of hours spent" but I guess you don't consider training time included. Also there is no need to quote the author's post and have a go at him personally. There are better ways to get your point across by arguing the point made and not the sentences written.

BugsJustFindMe 2y ago
> I guess you don't consider training time included In the same way that I don't consider the time an author spends writing a book when saying how long it took me to read the book. OP lost zero time training ChatGPT. Or do you mean how much time OP spent training themself? Because that's a whole new can of worms. How many years should we add to OP's development times because probably they learned how to write code before this exercise?
- keyle 2y ago
  I was considering the energy used and $, R&D and time spent to train those models, which is colossal compared to the developer and factor that in. Then they don't look so impressive. For reference, I'm not anti-LLM, I use Claude and ChatGPT every day. I'm just raising those if you want to consider all facts.
  - lgas 2y ago
    That cost is amortized over all requests made against all copies of the model though. Fortunately we don't have to re-train the model every time we want to perform inference.
  - bawolff 2y ago
    That's a weird comparison. Would you also count the number of hours it took for someone to write the OS, the editor, etc? The programmer wouldn't be effective without those things.
  - menaerus 2y ago
    And now also consider the amount of time, school, university, practice and $$$ it took to train a software engineer. Repeat for the population of presumably ~30 million software engineers in the world. Time-wise it doesn't scale at all when contrasted to new LLM releases that occur ~once a year per company. $$$-wise is also debatable.

segmondy 2y ago

I did experiment with local LLM running on my machine, most solutions were generated within the time of 30-60seconds. My overhead was really copying part 1 of the problem, generating the code, copying and pasting the code, running it, copying the input data, running it. Entering the result, repeating for part 2 and for most of them that was about 5 minutes from start to finish. If I automated the process, it would probably be 1 minute or less for the problems it could solve. Not the OP, but I was able to zero shot 27 solutions correctly without any back and forth, and 5 more with a little bit back and forth. Using local models.

mvdtnz 2y ago

> it understands the exact problem domain, it understands exactly what's being asked, correctly models the solution It does not "understand" or "model" shit. Good grief you AI shills need to take a breath.

BugsJustFindMe 2y ago
I'll use a different word when you demonstrate that humans aren't also stochastic parrots constantly hallucinating reality. Until then, good enough for one is good enough for the other.