4 ms·

Safety-critical perspective: Specification means requirements. I like EARS [0] syntax for requirements. e.g. "while an error is present, the software shall ig

Safety-critical perspective: Specification means requirements. I like EARS [0] syntax for requirements. e.g. "while an error is present, the software shall ignore keypresses" Requirements are not code at all, they are expectations about what the code does; it is the contract about what developers will be held accountable to. Putting implementation details in requirements is a rookie mistake because it takes agency away from the engineers in finding the best solution. The spec discussed in this article is more akin to the level of detail appropriate in an interface control document (ICD). Very common for a requirement to declare the software shall be compliant to a revision of an ICD. My own thoughts are: like a good systems engineer that recognizes the software engineers know their domain better than they, we should write specification for AI that leaves room for it to be more clever than we ourselves are. What's the point of exhaustive pseudocoding, it's worse coding. Align on general project preferences, set expectations, and concentrate effort on verifying. [0]: https://alistairmavin.com/ears/ https://alistairmavin.com/ears/

zombot 7mo ago
> for AI that leaves room for it to be more clever than we ourselves are. How probable is that given that LLMs output the average of their training data? Your own cleverness would have to be below average for this to hold.
- derrak 7mo ago
  > LLMs output the average of their training data I think it follows that if you ask it to implement something, it will implement the average thing. But then it also follows that if you give it the average thing and ask it to improve it by doing something clever, it may identify a clever improvement if such an improvement has ever been done in an analogous setting that exists in its training set. > Your own cleverness would have to be below average In this case the difference between myself and the LLM is that I’ve never seen the clever thing before (or I cannot recall it as effectively), but the LLM has and the LLM is specialized to recall it

zombot 7mo ago

> for AI that leaves room for it to be more clever than we ourselves are. How probable is that given that LLMs output the average of their training data? Your own cleverness would have to be below average for this to hold.

derrak 7mo ago
> LLMs output the average of their training data I think it follows that if you ask it to implement something, it will implement the average thing. But then it also follows that if you give it the average thing and ask it to improve it by doing something clever, it may identify a clever improvement if such an improvement has ever been done in an analogous setting that exists in its training set. > Your own cleverness would have to be below average In this case the difference between myself and the LLM is that I’ve never seen the clever thing before (or I cannot recall it as effectively), but the LLM has and the LLM is specialized to recall it