derrak

5 ms·

31.
by derrak 7mo ago
So do you disagree with the parent that “code is just a spec as far as the C compiler is concerned”? Maybe it’s important to agree on what a spec is. For instance, do you agree that a spec can be just as formal as code implementing the spec
32.
by derrak 7mo ago
My cynical take on this sort of research is that we will never use raw LLMs to solve these kinds of reasoning problems and it’s therefore unclear why we bother to test them on these kinds of benchmarks. Modern SAT solvers are completely cra
33.
by derrak 7mo ago
Are there any good systems that somehow enforce consistency between documentation and code? Maybe the problem is fundamentally ill-posed.
34.
by derrak 7mo ago
I see. Then we’re on the same page. My follow up question is: why do we care if the LLM is deterministic? Hypothetically, if we could guarantee a semantic relationship between the input and output we wouldn’t care if the LLM was determinist
35.
by derrak 7mo ago
> if we noted them down, it would have been longer than the code If you are only noting down requirements of what the program is doing rather than how it should do that thing, I would expect that writing the requirements would necessaril
36.
by derrak 7mo ago
> code is rigid and deterministic I really wish people would start defining what they mean when they say code is deterministic. For instance, code is not deterministic in the sense that it admits a single interpretation when compiled. C
37.
by derrak 7mo ago
> two programs can both satisfy the same spec and have completely different The spec should express all relevant constraints. If your spec admits two things and only one is admissible in your mind, your spec is incomplete. > has a mas
38.
by derrak 7mo ago
Why is this a weird definition of determinism? Could you please define what you mean when you say deterministic? A C program does not identify a single assembly program. It identifies a set of assembly programs. This fits the pretty standar
39.
by derrak 7mo ago
> LLMs output the average of their training data I think it follows that if you ask it to implement something, it will implement the average thing. But then it also follows that if you give it the average thing and ask it to improve it b
40.
by derrak 7mo ago
> Creating this spec is often going to be harder than writing a single compliant program. I’m not sure if I agree. A good spec will specify what the program is supposed to do rather than how the program should do that thing. It should be