12 ms·

Does programming have a clear reward function? A vague description from a business person is not it. By the time someone (a programmer?) has written a reward fu

Does programming have a clear reward function? A vague description from a business person is not it. By the time someone (a programmer?) has written a reward function that is clear enough, how would that function look compared to a program?

rossamurphy 2y ago
+1
eru 2y ago
> Does programming have a clear reward function? A vague description from a business person isn't it. By the time someone (a programmer?) has written a reward function that is clear enough, how would that function look compared to a program? Well, to give an example: the complexity class NP is all about problems that have quick and simple verification, but finding solutions for many problems is still famously hard. So there are at least some domains where this model would be a step forward.
- thaumasiotes 2y ago
  But in that case, finding the solution is hard and you generally don't try. Instead, you try to get fairly close, and it's more difficult to verify that you've done so.
  - eru 2y ago
    No. Most instances of most NP hard problems are easy to find solutions for. (It's actually really hard to eg construct a hard instance for the knapsack problem. And SAT solvers also tend to be really fast in practice.) And in any case, there are plenty of problems in NP that are not NP hard, too. Yes, approximation is also an important aspect of many practical problems. There's also lots of problems where you can easily specify one direction of processing, but it's hard to figure out how to undo that transformation. So you can get plenty of training data.
    - imtringued 2y ago
      I have a very simple integer linear program and it is really waiting for the heat death of the universe. No, running it as a linear program is still slow. I'm talking about small n=50 taking tens of minutes for a trivial linear program. Obviously the actual linear program is much bigger and scales quadratically in size, but still. N=50 is nothing.
      - eru 2y ago
        Yes, there are also instances of problems in NP that are hard to solve in practice. But here again: solutions to your problem are easy to verify, so it might be interesting to let an AI have a go at solving it.
tablatom 2y ago
Very good point. For some types of problems maybe the answer is yes. For example porting. The reward function is testing it behaves the same in the new language as the old one. Tricky for apps with a gui but doesn't seem impossible. The interesting kind of programming is the kind where I'm figuring out what I'm building as part of the process. Maybe AI will soon be superhuman in all the situations where we know exactly what we want (win the game), but not in the areas we don't. I find that kind of cool.
- martinflack 2y ago
  Even for porting there's a bit of ambiguity... Do you port line-for-line or do you adopt idioms of the target language? Do you port bug-for-bug as well as feature-for-feature? Do you leave yet-unused abstractions and opportunities for expansion that the original had coded in, if they're not yet used, and the target language code is much simpler without? I've found when porting that the answers to these are sometimes not universal for a codebase, but rather you are best served considering case-by-case inside the code. Although I suppose an AI agent could be created that holds a conversation with you and presents the options and acts accordingly.
littlestymaar 2y ago
“A precise enough specification is already code”, which means we'll not run out of developers in the short term. But the day to day job is going to be very different, maybe as different as what we're doing now compared to writing machine code on punchcards.
- mattmanser 2y ago
  Doubtful. This is the same mess we've been in repeatedly with 'low code'/'no code' solutions. Every decade it's 'we don't need programmers anymore'. Then it turns out specifying the problem needs programmers. Then it turns out the auto-coder can only reach a certain level of complexity. Then you've got real programmers modifying over-complicared code. Then everyone realizes they've wasted millions and it would have been quicker and cheaper to get the programmers to write the code in the first place. The same will almost certainly happen with AI generated code for the next decade or two, just at a slightly higher level of program complexity.
  - littlestymaar 2y ago
    > Every decade it's 'we don't need programmers anymore'. Then it turns out specifying the problem needs programmers. I literally refuted this in my comment… That being said, some kind of “no-code” is not necessarily a bad idea, as long as you treat it as just an abstraction for people who actually are programmers, like C versus assembly, or high level languages vs C. In fact I worked for a train manufacturer that had a cool “no code” tool to program automated train control software with automated theorem proving built in, and it was much more efficient than there former Ada implementation especially when you factor the hiring difficulties in.
    - littlestymaar 2y ago
      their*
cs702 2y ago
Programming has a clear reward function when the problem being solving is well-specified, e.g., "we need a program that grabs data from these three endpoints, combines their data in this manner, and returns it in this JSON format." The same is true for math. There is a clear reward function when the goal is well-specified, e.g., "we need a sequence of mathematical statements that prove this other important mathematical statement is true."
- seanthemon 2y ago
  >when the problem being solving is well-specified Phew! Sounds like i'll be fine, thank god for product owners.
  - steveBK123 2y ago
    20 years, number of "well specified" requirements documents I've received: 0.
  - deleted 2y ago
    [deleted]
- danpalmer 2y ago
  I’m not sure I would agree. By the time you’ve written a full spec for it, you may as well have just written a high level programming language anyway. You can make assumptions that minimise the spec needed… but also programming APIs can have defaults so that’s no advantage. I’d suggest that the Python code for your example prompt with reasonable defaults is not actually that far from the prompt itself in terms of the time necessary to write it. However, add tricky details like how you want to handle connection pooling, differing retry strategies, short circuiting based on one of the results, business logic in the data combination step, and suddenly you’ve got a whole design doc in your prompt and you need a senior engineer with good written comms skills to get it to work.
  - cs702 2y ago
    Thanks. I view your comment as orthogonal to mine, because I didn't say anything about how easy or hard it would be for human beings to specify the problems that must be solved. Some problems may be easy to specify, others may be hard. I feel we're looking at the need for a measure of the computational complexity of problem specifications -- something like Kolmogorov complexity, i.e., minimum number of bits required, but for specifying instead of solving problems.
gizmo 2y ago
Much business logic is really just a state machine where all the states and all the transitions need to be handled. When a state or transition is under-specified an LLM can pass the ball back and just ask what should happen when A and B but not C. Or follow more vague guidance on what should happen in edge cases. A typical business person is perfectly capable of describing how invoicing should work and when refunds should be issued, but very few business people can write a few thousand lines of code that covers all the cases.
- discreteevent 2y ago
  > an LLM can pass the ball back and just ask what should happen when A and B but not C What should the colleagues of the business person review before deciding that the system is fit for purpose? Or what should they review when the system fails? Should they go back over the transcript of the conversation with the LLM?
  - ben_w 2y ago
    As an LLM can output source code, that's all answerable with "exactly what they already do when talking to developers".
    - discreteevent 2y ago
      There are two reasons the system might fail: 1) The business person made a mistake in their conversation/specification. In this case the LLM will have generated code and tests that match the mistake. So all the tests will pass. The best way to catch this before it gets to production is to have someone else review the specification. But the problem is that the specification is a long trial-and-error conversation in which later parts may contradict earlier parts. Good luck reviewing that. 2) The LLM made a mistake. The LLM may have made the mistake because of a hallucination which it cannot correct because in trying to correct it the same hallucination invalidates the correction. At this point someone has to debug the system. But we got rid of all the programmers.
      - 2y ago
jimbokun 2y ago
The reward function could be "pass all of these tests I just wrote".
- marcosdumay 2y ago
  Lol. Literally. If you have those many well written tests, you can pass them to a constraint solver today and get your program. No LLM needed. Or even run your tests instead of the program.
  - emporas 2y ago
    Probably the parent assumes that he does have the tests, billions of them. One very strong LLM could generate billions of tests alongside the working code and then train another smaller model, or feed it into the next iteration of training same the strong model. Strong LLMs do exist for that purpose, Nemotron 320B and Llama 3 450B. It would be interesting if a dataset like that would be created like that, and then released as open source. Many LLMs proprietary or not, could incorporate the dataset in their training, and have on the internet hundreds of LLMs suddenly become much better at coding, all of them at once.
- acchow 2y ago
  After much RL, the model will just learn to mock everything to get the test to pass.
- guipsp 2y ago
  You cannot
ryukoposting 2y ago
If we will struggle to create reward functions for AI, then how different is that from the struggles we already face when divvying up product goals into small tasks to fit our development cycles? In other words, to what extent does Agile's ubiquity prove our competence in turning product goals into de facto reward functions?
paxys 2y ago
Exactly, and people have been saying this for a while now. If an "AI software engineer" needs a perfect spec with zero ambiguity, all edge cases defined, full test coverage with desired outcomes etc., then the person writing the spec is the actual software engineer, and the AI is just a compiler.
- satvikpendem 2y ago
  Reminds me of when computers were literally humans computing things (often women). How time weaves its circular web.
- dartos 2y ago
  We’ve also learned that starting off by rigidly defined spec is actually harmful to most user facing software, since customers change their minds so often and have a hard time knowing what they want right from the start.
  - diffxx 2y ago
    This is why most of the best software is written by people writing things for themselves and most of the worst is made by people making software they don't use themselves.
    - TZubiri 2y ago
      True facts: half of the self made software are task trackers.
      - dartos 2y ago
        Sure, and the most performed song in the world is probably hot cross buns or Mary had a little lamb.
- _the_inflator 2y ago
  Exactly. This is what I tell everyone. The harder you work on specs the easier it gets in the aftermath. And this is exactly what business with lofty goals doesn’t get or ignores. Put another way: a fool with a tool… Also look out for optimization the clever way.
airstrike 2y ago
My reward in Rust is often when the code actually compiles...
_the_inflator 2y ago
Full circle but instead of determinism you introduce some randomness. Not good. Also the reasoning is something business is dissonant about. The majority of planning and execution teams stick to processes. I see way more potential automating these than all parts in app production. Business is going to have a hard time, when they believe, they alone can orchestrate some AI consoles.
- deleted 2y ago
  [deleted]
tomrod 2y ago
You can define one based on passed tests, code coverage, other objectives, or weighted combinations without too much loss of generality.
LeifCarrotson 2y ago
I think you could set up a good reward function for a programming assistance AI by checking that the resulting code is actually used. Flag or just 'git blame' the code produced by the AI with the prompts used to produce it, and when you push a release, it can check which outputs were retained in production code from which prompts. Hard to say whether code that needed edits was because the prompt was bad or because the code was bad, but at least you can get positive feedback when a good prompt resulted in good code.
- rfw300 2y ago
  GitHub Copilot's telemetry does collect data on whether generated code snippets end up staying in the code, so presumably models are tuned on this feedback. But you haven't solved any of the problems set out by Karpathy here—this is just bankshot RLHF.
- bee_rider 2y ago
  That could be interesting but it does seem like a much fuzzier and slower feedback loop than the original idea. It also seems less unique to code. You could also have a chat bot write an encyclopedia and see if the encyclopedias sold well. Chat bots could edit Wikipedia and see if their edits stuck as a reward function (seems ethically pretty questionable or at least in need of ethical analysis, but it is possible). The maybe-easy to evaluate reward function is an interesting aspect of code (which isn’t to say it is the only interesting aspect, for sure!)
axus 2y ago
If they get permission and don't mind waiting, they could check if people throw away the generated code or keep it as-is.
consteval 2y ago
There's levels to this. Certainly "compiled" is one reward (although a blank file fits that...) Another is test cases, input and output. This doesn't work on a software-wide scale but function-wide it can work. In the future I think we'll see more of this test-driven development. Where developers formally define the requirements and expectations of a system and then an LLM (combined with other tools) generates the implementation. So instead of making the implementation, you just declaratively say what the implementation should do (and shouldn't).
waldrews 2y ago
There's no reward function in the sense that optimizing the reward function means the solution is ideal. There are objective criteria like 'compiles correctly' and 'passes self-designed tests' and 'is interpreted as correct by another LLM instance' which go a lot further than criteria that could be defined for most kinds of verbal questions.

rossamurphy 2y ago

eru 2y ago

> Does programming have a clear reward function? A vague description from a business person isn't it. By the time someone (a programmer?) has written a reward function that is clear enough, how would that function look compared to a program? Well, to give an example: the complexity class NP is all about problems that have quick and simple verification, but finding solutions for many problems is still famously hard. So there are at least some domains where this model would be a step forward.

thaumasiotes 2y ago
But in that case, finding the solution is hard and you generally don't try. Instead, you try to get fairly close, and it's more difficult to verify that you've done so.
- eru 2y ago
  No. Most instances of most NP hard problems are easy to find solutions for. (It's actually really hard to eg construct a hard instance for the knapsack problem. And SAT solvers also tend to be really fast in practice.) And in any case, there are plenty of problems in NP that are not NP hard, too. Yes, approximation is also an important aspect of many practical problems. There's also lots of problems where you can easily specify one direction of processing, but it's hard to figure out how to undo that transformation. So you can get plenty of training data.
  - imtringued 2y ago
    I have a very simple integer linear program and it is really waiting for the heat death of the universe. No, running it as a linear program is still slow. I'm talking about small n=50 taking tens of minutes for a trivial linear program. Obviously the actual linear program is much bigger and scales quadratically in size, but still. N=50 is nothing.
    - eru 2y ago
      Yes, there are also instances of problems in NP that are hard to solve in practice. But here again: solutions to your problem are easy to verify, so it might be interesting to let an AI have a go at solving it.

tablatom 2y ago

Very good point. For some types of problems maybe the answer is yes. For example porting. The reward function is testing it behaves the same in the new language as the old one. Tricky for apps with a gui but doesn't seem impossible. The interesting kind of programming is the kind where I'm figuring out what I'm building as part of the process. Maybe AI will soon be superhuman in all the situations where we know exactly what we want (win the game), but not in the areas we don't. I find that kind of cool.

martinflack 2y ago
Even for porting there's a bit of ambiguity... Do you port line-for-line or do you adopt idioms of the target language? Do you port bug-for-bug as well as feature-for-feature? Do you leave yet-unused abstractions and opportunities for expansion that the original had coded in, if they're not yet used, and the target language code is much simpler without? I've found when porting that the answers to these are sometimes not universal for a codebase, but rather you are best served considering case-by-case inside the code. Although I suppose an AI agent could be created that holds a conversation with you and presents the options and acts accordingly.

littlestymaar 2y ago

“A precise enough specification is already code”, which means we'll not run out of developers in the short term. But the day to day job is going to be very different, maybe as different as what we're doing now compared to writing machine code on punchcards.

mattmanser 2y ago
Doubtful. This is the same mess we've been in repeatedly with 'low code'/'no code' solutions. Every decade it's 'we don't need programmers anymore'. Then it turns out specifying the problem needs programmers. Then it turns out the auto-coder can only reach a certain level of complexity. Then you've got real programmers modifying over-complicared code. Then everyone realizes they've wasted millions and it would have been quicker and cheaper to get the programmers to write the code in the first place. The same will almost certainly happen with AI generated code for the next decade or two, just at a slightly higher level of program complexity.
- littlestymaar 2y ago
  > Every decade it's 'we don't need programmers anymore'. Then it turns out specifying the problem needs programmers. I literally refuted this in my comment… That being said, some kind of “no-code” is not necessarily a bad idea, as long as you treat it as just an abstraction for people who actually are programmers, like C versus assembly, or high level languages vs C. In fact I worked for a train manufacturer that had a cool “no code” tool to program automated train control software with automated theorem proving built in, and it was much more efficient than there former Ada implementation especially when you factor the hiring difficulties in.
  - littlestymaar 2y ago
    their*

cs702 2y ago

Programming has a clear reward function when the problem being solving is well-specified, e.g., "we need a program that grabs data from these three endpoints, combines their data in this manner, and returns it in this JSON format." The same is true for math. There is a clear reward function when the goal is well-specified, e.g., "we need a sequence of mathematical statements that prove this other important mathematical statement is true."

seanthemon 2y ago
>when the problem being solving is well-specified Phew! Sounds like i'll be fine, thank god for product owners.
- steveBK123 2y ago
  20 years, number of "well specified" requirements documents I've received: 0.
- deleted 2y ago
  [deleted]
danpalmer 2y ago
I’m not sure I would agree. By the time you’ve written a full spec for it, you may as well have just written a high level programming language anyway. You can make assumptions that minimise the spec needed… but also programming APIs can have defaults so that’s no advantage. I’d suggest that the Python code for your example prompt with reasonable defaults is not actually that far from the prompt itself in terms of the time necessary to write it. However, add tricky details like how you want to handle connection pooling, differing retry strategies, short circuiting based on one of the results, business logic in the data combination step, and suddenly you’ve got a whole design doc in your prompt and you need a senior engineer with good written comms skills to get it to work.
- cs702 2y ago
  Thanks. I view your comment as orthogonal to mine, because I didn't say anything about how easy or hard it would be for human beings to specify the problems that must be solved. Some problems may be easy to specify, others may be hard. I feel we're looking at the need for a measure of the computational complexity of problem specifications -- something like Kolmogorov complexity, i.e., minimum number of bits required, but for specifying instead of solving problems.

gizmo 2y ago

Much business logic is really just a state machine where all the states and all the transitions need to be handled. When a state or transition is under-specified an LLM can pass the ball back and just ask what should happen when A and B but not C. Or follow more vague guidance on what should happen in edge cases. A typical business person is perfectly capable of describing how invoicing should work and when refunds should be issued, but very few business people can write a few thousand lines of code that covers all the cases.

discreteevent 2y ago
> an LLM can pass the ball back and just ask what should happen when A and B but not C What should the colleagues of the business person review before deciding that the system is fit for purpose? Or what should they review when the system fails? Should they go back over the transcript of the conversation with the LLM?
- ben_w 2y ago
  As an LLM can output source code, that's all answerable with "exactly what they already do when talking to developers".
  - discreteevent 2y ago
    There are two reasons the system might fail: 1) The business person made a mistake in their conversation/specification. In this case the LLM will have generated code and tests that match the mistake. So all the tests will pass. The best way to catch this before it gets to production is to have someone else review the specification. But the problem is that the specification is a long trial-and-error conversation in which later parts may contradict earlier parts. Good luck reviewing that. 2) The LLM made a mistake. The LLM may have made the mistake because of a hallucination which it cannot correct because in trying to correct it the same hallucination invalidates the correction. At this point someone has to debug the system. But we got rid of all the programmers.
    - 2y ago

jimbokun 2y ago

The reward function could be "pass all of these tests I just wrote".

marcosdumay 2y ago
Lol. Literally. If you have those many well written tests, you can pass them to a constraint solver today and get your program. No LLM needed. Or even run your tests instead of the program.
- emporas 2y ago
  Probably the parent assumes that he does have the tests, billions of them. One very strong LLM could generate billions of tests alongside the working code and then train another smaller model, or feed it into the next iteration of training same the strong model. Strong LLMs do exist for that purpose, Nemotron 320B and Llama 3 450B. It would be interesting if a dataset like that would be created like that, and then released as open source. Many LLMs proprietary or not, could incorporate the dataset in their training, and have on the internet hundreds of LLMs suddenly become much better at coding, all of them at once.
acchow 2y ago
After much RL, the model will just learn to mock everything to get the test to pass.
guipsp 2y ago
You cannot

ryukoposting 2y ago

If we will struggle to create reward functions for AI, then how different is that from the struggles we already face when divvying up product goals into small tasks to fit our development cycles? In other words, to what extent does Agile's ubiquity prove our competence in turning product goals into de facto reward functions?

paxys 2y ago

Exactly, and people have been saying this for a while now. If an "AI software engineer" needs a perfect spec with zero ambiguity, all edge cases defined, full test coverage with desired outcomes etc., then the person writing the spec is the actual software engineer, and the AI is just a compiler.

satvikpendem 2y ago
Reminds me of when computers were literally humans computing things (often women). How time weaves its circular web.
dartos 2y ago
We’ve also learned that starting off by rigidly defined spec is actually harmful to most user facing software, since customers change their minds so often and have a hard time knowing what they want right from the start.
- diffxx 2y ago
  This is why most of the best software is written by people writing things for themselves and most of the worst is made by people making software they don't use themselves.
  - TZubiri 2y ago
    True facts: half of the self made software are task trackers.
    - dartos 2y ago
      Sure, and the most performed song in the world is probably hot cross buns or Mary had a little lamb.
_the_inflator 2y ago
Exactly. This is what I tell everyone. The harder you work on specs the easier it gets in the aftermath. And this is exactly what business with lofty goals doesn’t get or ignores. Put another way: a fool with a tool… Also look out for optimization the clever way.

airstrike 2y ago

My reward in Rust is often when the code actually compiles...

_the_inflator 2y ago

Full circle but instead of determinism you introduce some randomness. Not good. Also the reasoning is something business is dissonant about. The majority of planning and execution teams stick to processes. I see way more potential automating these than all parts in app production. Business is going to have a hard time, when they believe, they alone can orchestrate some AI consoles.

deleted 2y ago
[deleted]

tomrod 2y ago

You can define one based on passed tests, code coverage, other objectives, or weighted combinations without too much loss of generality.

LeifCarrotson 2y ago

I think you could set up a good reward function for a programming assistance AI by checking that the resulting code is actually used. Flag or just 'git blame' the code produced by the AI with the prompts used to produce it, and when you push a release, it can check which outputs were retained in production code from which prompts. Hard to say whether code that needed edits was because the prompt was bad or because the code was bad, but at least you can get positive feedback when a good prompt resulted in good code.

rfw300 2y ago
GitHub Copilot's telemetry does collect data on whether generated code snippets end up staying in the code, so presumably models are tuned on this feedback. But you haven't solved any of the problems set out by Karpathy here—this is just bankshot RLHF.
bee_rider 2y ago
That could be interesting but it does seem like a much fuzzier and slower feedback loop than the original idea. It also seems less unique to code. You could also have a chat bot write an encyclopedia and see if the encyclopedias sold well. Chat bots could edit Wikipedia and see if their edits stuck as a reward function (seems ethically pretty questionable or at least in need of ethical analysis, but it is possible). The maybe-easy to evaluate reward function is an interesting aspect of code (which isn’t to say it is the only interesting aspect, for sure!)

axus 2y ago

If they get permission and don't mind waiting, they could check if people throw away the generated code or keep it as-is.

consteval 2y ago

There's levels to this. Certainly "compiled" is one reward (although a blank file fits that...) Another is test cases, input and output. This doesn't work on a software-wide scale but function-wide it can work. In the future I think we'll see more of this test-driven development. Where developers formally define the requirements and expectations of a system and then an LLM (combined with other tools) generates the implementation. So instead of making the implementation, you just declaratively say what the implementation should do (and shouldn't).

waldrews 2y ago

There's no reward function in the sense that optimizing the reward function means the solution is ideal. There are objective criteria like 'compiles correctly' and 'passes self-designed tests' and 'is interpreted as correct by another LLM instance' which go a lot further than criteria that could be defined for most kinds of verbal questions.