Why do groups matter at all? Shouldn't the individual be what we measure? How much is this individual disadvantaged, regardless of their group or set of groups?

Why do groups matter at all? Shouldn't the individual be what we measure? How much is this individual disadvantaged, regardless of their group or set of groups? Adjust for that, rather than arbitrary groupings.