I think that what we're losing is a habit of mapping our squishy human desires onto a formal system. Maybe you didn't chose a toolchain that treats it as a for

I think that what we're losing is a habit of mapping our squishy human desires onto a formal system. Maybe you didn't chose a toolchain that treats it as a formal system. There isn't a proof assistant to be seen for miles in any direction. But when the guy says "give me foo" and you say "that's impossible", you're consulting a formal system and translating a proof back into that squishy human desire language. The author is maybe a bit too attached to the idea that the system is an ideal representation of the real world. Given a landscape of systems that might represent it, probably there's one that lets you give the guy what he wants, and probably it's not the one you've built. Spend enough time building something and it changes how you see the world, so that's an expected bias. I agree that it's a loss. The statistical approach used by AI's or by any sufficiently complex system of customer facing employees (which likely isolates the customer from the programmer) tends towards creating responses that are likely to make the guy go away, which is not always the same as responses that gave him what he asked for.