The problem is that most very rich people have very concentrated positions. The family office might only manage 10% of the wealth with 90% being in one company.

The problem is that most very rich people have very concentrated positions. The family office might only manage 10% of the wealth with 90% being in one company. The investment then has to be structured so that the 10% survive even if the 90% vanish overnight (unlikely but not impossible). The performance on those 10% isn't that important, ownership in the company drives most of the fluctuation in wealth anyway.