3 ms·
Thought about automated discovery of laws in an existing codebase? If you can find a law which the existing code obeys, and show it to a human, and if they agr
by skissane 17d ago
Thought about automated discovery of laws in an existing codebase?
If you can find a law which the existing code obeys, and show it to a human, and if they agree, save it. And maybe the AI could make a decent guess as to what kind of laws would appeal to a human versus which wouldn't – a simple law identifying a fundamental constraint the system obeys is good, something really complicated or constraining something coincidentally true isn't
Or some kind of measure of coverage? you'd never want 100% – then your laws would become so complex you couldn't follow or maintain them – but if it is very low, that can be a signal to increase it
- drfloyd51 17d agoDiscover the laws in a current code base? They are usually in the “tests” folder. So anything capable of extracting unit tests is extracting “laws”. One might consider the test name the text of a law. And the AI fills in the details.
- skissane 17d agoI don’t think that’s right. Unit tests check whether a specific code module is implemented correctly given its (implicit) specification. Good “laws” are independent of the structure of the code. Tests may be a useful source for inferring what the laws are, but much of the actual content of the tests aren’t “laws” at all.
- serf 17d agoI think the premise is more that if one is given a grouping/constellation of unit tests that one can derive generalized laws by looking at what behavior those tests were checking for or against. if there are a dozen unit tests trying to determine if some regex can escape a sensitive area, then one can derive a generalized 'don't let the regex escape from here' type rule -- or at least you could theoretically. I'm sure in reality that'd be a big minefield much like harness self-skill-writing has been.
- schlich 17d agoyou might be interested in property-based testing, which somewhat enumerates tests based on invariance and induction
- actionfromafar 17d agoFrom the codebases I have seen the "laws" are scattered between unit tests and lots of little tests and assertions in the actual code, plus cultural norms and "do it like this" patterns. But a good start could be found in the unit tests.