I'm also from the bay area, and it's quite simple: demand skyrocketed due to tech, and supply couldn't keep up, due to various kinds of regulations. Zoning is

I'm also from the bay area, and it's quite simple: demand skyrocketed due to tech, and supply couldn't keep up, due to various kinds of regulations. Zoning is the most obvious regulation that makes things hard: most land is zoned exclusively for single family homes on large lots. There's been a bit of movement on this from the state level to allow ADU's or duplexes, but not enough, there's usually various legal hurdles that make it harder. Missing middle housing is mostly still illegal. The other thing is all the various processes involved. CEQA makes things take a long time (not that I'm opposed to protecting the environment, but surely there's a way to do it that doesn't take years for an apartment complex). Community meetings and planning commissions can often block new developments for random, arbitrary reasons, like "tall building would cast too many shadows" or "demand on street parking would increase"; these are things that are so obviously inherent to any significant development permitted in areas where you can build taller apartment buildings, and yet they're still used to block more housing.