I'm not sure what the big mystery is. Management exists to provide an abstraction for information cascade (both up and down) and centralized command and control

I'm not sure what the big mystery is. Management exists to provide an abstraction for information cascade (both up and down) and centralized command and control. The amount of management needed decreases for rote work, and increases for more creative or novel work in competitive industries. In a healthy, well-functioning org a manager will have a reasonably deep understanding of everything going underneath them. This ladders up to where a CEO can get in a room with 5-6 execs and discuss the biggest problems for the company with a group that is A) small enough to have an actual dialogue and B) have as much breadth and depth of knowledge as is reasonably possible for such a small group. Like a balanced search tree, this structure means any necessary detail can theoretically be fished out by going through log(n) layers, which enables scaling to tens of thousands or hundreds of thousands of employees. Of course you could also do this without managers by just having certain individual contributors be the subject matter experts of record, but such systems end up being more complicated because you lack the clarity of every person regardless of role or position having one and exactly one boss. Now this doesn't mean this is how work gets done, it just provides a substrate for decision making. In terms of actual work, more complex (read: non-DAG) networks are always needed (see Conway's Law), but when conflicts or inaction arise the manager hierarchy is what allows problems to be resolved decisively. Now obviously in the real world people are people, and management is often the first place dysfunction shows up. This is partly because of the Peter Principle, partly because it's just easier to hide a lack of competence and play politics when you're in the fog of the middle layers, but that does not invalidate the function of management. The implication of this is that the primary job of senior leadership is bullshit detection.