5 ms·

Competitive in what? Advent of Code as I see it is widely regarded as a chance for people to learn a new programming language or to have fun. No one I know try

by magnio 4y ago

Competitive in what? Advent of Code as I see it is widely regarded as a chance for people to learn a new programming language or to have fun. No one I know try to "compete" in AoC. On the other hand, if ChatGPT provides a competitive edge in CodeForce problems then it would be worrying indeed.

sriram_sun 4y ago
As a pretty senior engineer, I can tell you that Advent of Code or not, Google-Fu is a big part of your effectiveness in picking up new stuff and delivering code. Adding "Prompt-Fu" that equation seems to be the next logical thing to do.
- bluejellybean 4y ago
  Completely agree with this sentiment, and the general idea that this is a pretty monumental moment in time. In a couple of hours of work, I've been able to prompt for something like "Get an nginx server and node.js backend up and running", have it generate a rough outline of todo elements, add color/detail to each outlined todo, and write tickets. The tickets even included a title, what done looks like, how to QA, and additional background info as to how this ticket fits into the greater whole. The only thing I didn't get it to do was the actual code writing because I figured this article, or another like it, would prove out it's capabilities for me. Even if the code comes out looking rough like a junior wrote it, the ability to prompt for edits in the PR review should allow us to resolve most issues. When you feed one level of prompt output into the next using some functional tricks, the ability to layer high-level ideas into the engine become extremely powerful. Even if this isn't 'true' AGI, with a few prompt scaffolding libraries, it won't exactly matter. I'm convinced this is going to radically shake up the entire software field at a process level. Both exciting and scary times ahead.
  - sitkack 4y ago
    So writing code in the near future would not be unlike having manage 10s or potentially thousands of folks (ml models/agents) who have taken two semesters of CS. #EternalHacktoberfest What your thoughts on doing TDD with this, write the unit tests to write the code. But go one step further and have it generate tests from the spec. Repeat until 1=1.
    - bluejellybean 4y ago
      Near future, that seems right, further out I am guessing GPT-4 could enable the level to a couple years of real-world skill instead of two semesters. With the pipeline as such: business need -> tasks/cards outlined -> tests written -> code written -> refactor as changes needed There is still a need for someone to manage the business to task relationship. Let the business person prompt, and the engineer to confirm the prompt output tasks makes sense technically, edit for any errors, and do any modifications needed for specific architectural choices or desired abstractions. With well-enough defined tasks, you can start to write the tests that conform to those tasks. Again, engineer checks the prompt-output, ensuring the tests line up with the card, making any edits as required. With tests written, the code can be written such that it conforms to the tests for correctness, cards for general I/O, and business use-case for domain-specific variables and such. It's the same domain splits that occur in our current day-to-day practice that are cause for pain. Business person and product person have a miscommunication, the wrong tasks/cards get outlined. The task writer and the test writer have a miscommunication, the tests get written poorly and problem the business is trying to solve gets murky. The tests are written poorly so the code is written poorly. The code is written poorly so the product must be refactored. The understanding of each others intent must be had and communicated effectively or the downstream problems will mount quickly. It's in this area where GPT-3 still seems a bit lacking, and maybe GPT-4 will resolve the issue a bit. Another worry I have with this sort of model is the spaghetti and debuggability a misunderstanding may have. When code is written more slowly, as is the case today, one is able to mentor the junior and provide feedback over the process. This not only prevents a huge mess, but allows the team to resolve any initially unsaid misunderstandings. The speed and scale of messes one can now create with this tool is completely massive!
thom 4y ago
Yeah, the race to create the first ugly one-liner to solve the problem has never been interesting to me. Far more fun to play around, explore the correct algorithms and data structures etc. Even on Day 1 this year, you can learn a lot thinking about partial sorting etc. If language models can help me with that then great. I’d be very interested in tools that felt like pairing with the wisest programmer you’d ever met, rather than tools that feel like outsourcing to an admittedly quite fast and clever contractor.
rrss 4y ago
Hi, some of us try to compete in AoC. For me that is part of the fun. In several previous years I have rescheduled things so I can stay up until the problems are released in my time zone to see how high on the leaderboard I could get. It is sad that the AoC leaderboard will now just be filled with GPT entries.
- exoji2e 4y ago
  I mean, that GPT solves problem 1,3,4 of AoC this year does not translate into it solving all problems, not even close. For example, try giving ChatGPT the problem statement of https://adventofcode.com/2021/day/19 https://adventofcode.com/2021/day/19. What comes out is basically gibberish. It seems to completely miss that the scanners report their coordinates in their own coordinate system. But if nothing changes in the rules/community the leaderboards of early and more straight forward problems will probably be filled with GPT solutions.
  - zppln 4y ago
    I could see it be pretty demotivating not standing a chance against AIs for the first couple of days though (and since you don't know beforehand which day they'll break down you still have to put in the work everyday).
    - exoji2e 4y ago
      Yeah, definately. I would like the leaderboard to be for only humans. But it seems futile to try to ban it and enforce it in any meaningful way, without changing the concept. An honorsystem just stating "if you wish to use a languagemodel like gpt, then please wait before the leaderboard is filled" might work, but I'm not sure.
  - deleted 4y ago
    [deleted]
exoji2e 4y ago
A lot of people compete to try to get on the global leader boards, which I find quite fun, also makes me wake up at 05:30 am every day! But yeah, this probably mostly helps for easier problems, and is quite useless when either the problem needs a cleaver construction or some trick to make the time complexity good enough. Deepmind has a project called AlphaCode, where they got in the middle of div2 rounds on codeforces in february 2022 (virtual): https://www.deepmind.com/blog/competitive-programming-with-alphacode https://www.deepmind.com/blog/competitive-programming-with-a... In the blogpost it actually comes up with a linear stack based solution (one of the intended solutions) to a problem where the naive solution is quadratic.

by magnio 4y ago

sriram_sun 4y ago

As a pretty senior engineer, I can tell you that Advent of Code or not, Google-Fu is a big part of your effectiveness in picking up new stuff and delivering code. Adding "Prompt-Fu" that equation seems to be the next logical thing to do.

bluejellybean 4y ago
Completely agree with this sentiment, and the general idea that this is a pretty monumental moment in time. In a couple of hours of work, I've been able to prompt for something like "Get an nginx server and node.js backend up and running", have it generate a rough outline of todo elements, add color/detail to each outlined todo, and write tickets. The tickets even included a title, what done looks like, how to QA, and additional background info as to how this ticket fits into the greater whole. The only thing I didn't get it to do was the actual code writing because I figured this article, or another like it, would prove out it's capabilities for me. Even if the code comes out looking rough like a junior wrote it, the ability to prompt for edits in the PR review should allow us to resolve most issues. When you feed one level of prompt output into the next using some functional tricks, the ability to layer high-level ideas into the engine become extremely powerful. Even if this isn't 'true' AGI, with a few prompt scaffolding libraries, it won't exactly matter. I'm convinced this is going to radically shake up the entire software field at a process level. Both exciting and scary times ahead.
- sitkack 4y ago
  So writing code in the near future would not be unlike having manage 10s or potentially thousands of folks (ml models/agents) who have taken two semesters of CS. #EternalHacktoberfest What your thoughts on doing TDD with this, write the unit tests to write the code. But go one step further and have it generate tests from the spec. Repeat until 1=1.
  - bluejellybean 4y ago
    Near future, that seems right, further out I am guessing GPT-4 could enable the level to a couple years of real-world skill instead of two semesters. With the pipeline as such: business need -> tasks/cards outlined -> tests written -> code written -> refactor as changes needed There is still a need for someone to manage the business to task relationship. Let the business person prompt, and the engineer to confirm the prompt output tasks makes sense technically, edit for any errors, and do any modifications needed for specific architectural choices or desired abstractions. With well-enough defined tasks, you can start to write the tests that conform to those tasks. Again, engineer checks the prompt-output, ensuring the tests line up with the card, making any edits as required. With tests written, the code can be written such that it conforms to the tests for correctness, cards for general I/O, and business use-case for domain-specific variables and such. It's the same domain splits that occur in our current day-to-day practice that are cause for pain. Business person and product person have a miscommunication, the wrong tasks/cards get outlined. The task writer and the test writer have a miscommunication, the tests get written poorly and problem the business is trying to solve gets murky. The tests are written poorly so the code is written poorly. The code is written poorly so the product must be refactored. The understanding of each others intent must be had and communicated effectively or the downstream problems will mount quickly. It's in this area where GPT-3 still seems a bit lacking, and maybe GPT-4 will resolve the issue a bit. Another worry I have with this sort of model is the spaghetti and debuggability a misunderstanding may have. When code is written more slowly, as is the case today, one is able to mentor the junior and provide feedback over the process. This not only prevents a huge mess, but allows the team to resolve any initially unsaid misunderstandings. The speed and scale of messes one can now create with this tool is completely massive!

thom 4y ago

Yeah, the race to create the first ugly one-liner to solve the problem has never been interesting to me. Far more fun to play around, explore the correct algorithms and data structures etc. Even on Day 1 this year, you can learn a lot thinking about partial sorting etc. If language models can help me with that then great. I’d be very interested in tools that felt like pairing with the wisest programmer you’d ever met, rather than tools that feel like outsourcing to an admittedly quite fast and clever contractor.

rrss 4y ago

Hi, some of us try to compete in AoC. For me that is part of the fun. In several previous years I have rescheduled things so I can stay up until the problems are released in my time zone to see how high on the leaderboard I could get. It is sad that the AoC leaderboard will now just be filled with GPT entries.

exoji2e 4y ago
I mean, that GPT solves problem 1,3,4 of AoC this year does not translate into it solving all problems, not even close. For example, try giving ChatGPT the problem statement of https://adventofcode.com/2021/day/19 https://adventofcode.com/2021/day/19. What comes out is basically gibberish. It seems to completely miss that the scanners report their coordinates in their own coordinate system. But if nothing changes in the rules/community the leaderboards of early and more straight forward problems will probably be filled with GPT solutions.
- zppln 4y ago
  I could see it be pretty demotivating not standing a chance against AIs for the first couple of days though (and since you don't know beforehand which day they'll break down you still have to put in the work everyday).
  - exoji2e 4y ago
    Yeah, definately. I would like the leaderboard to be for only humans. But it seems futile to try to ban it and enforce it in any meaningful way, without changing the concept. An honorsystem just stating "if you wish to use a languagemodel like gpt, then please wait before the leaderboard is filled" might work, but I'm not sure.
- deleted 4y ago
  [deleted]

exoji2e 4y ago

A lot of people compete to try to get on the global leader boards, which I find quite fun, also makes me wake up at 05:30 am every day! But yeah, this probably mostly helps for easier problems, and is quite useless when either the problem needs a cleaver construction or some trick to make the time complexity good enough. Deepmind has a project called AlphaCode, where they got in the middle of div2 rounds on codeforces in february 2022 (virtual): https://www.deepmind.com/blog/competitive-programming-with-alphacode https://www.deepmind.com/blog/competitive-programming-with-a... In the blogpost it actually comes up with a linear stack based solution (one of the intended solutions) to a problem where the naive solution is quadratic.