That sounds like a cute analogy, but even though the wooden framework of a house is mostly see-through and somewhat modular, it would actually be quite cumberso

That sounds like a cute analogy, but even though the wooden framework of a house is mostly see-through and somewhat modular, it would actually be quite cumbersome and effortful to say "okay, but for this particular wall I'm going to do something entirely different", and if you do, then it might turn out that some of the other things you'd like to have on top of it, say, insulation or drywall, no longer fit very well or no longer do their job as intended. In that sense, the word "framework" in programming actually maps quite nicely to the original meaning of the term.