They're also a genuine hassle/pain for anyone who happens to be on the autistic spectrum, or who has trouble with social anxiety in general. As someone in that

They're also a genuine hassle/pain for anyone who happens to be on the autistic spectrum, or who has trouble with social anxiety in general. As someone in that situation, open plan offices are basically a recipe for constant stress and sensory overload. Either way, it's worrying given those in said situation are statistically likely to find it challenging to find employment as it is, and software engineering/web development/programming is an area which is seen as more suitable for them. As for why it's so common? Well there are a few reasons given in other comments, but I suspect a major one is that managers/boss/CEOs tend to be more extraverted than average. So they end up going with office designs that they would have wanted to work in, not ones their employees would work better in.