As a graduate of both Harvard and Caltech, I can confirm that this is true. Even many people in Pasadena, where Caltech is located, are only vaguely aware of it

As a graduate of both Harvard and Caltech, I can confirm that this is true. Even many people in Pasadena, where Caltech is located, are only vaguely aware of its existence, and even many southern Californians confuse it with Cal Poly Pomona. Of course, Caltech is universally known and respected among technical people, but the contrast with Harvard—which, as you say, is known by just about every mother, brother, and child—is unmistakable. (Indeed, it's telling that many references to Caltech in these comments misspell its name. Hell, even many Pasadena city street signs misspell its name. Apparently the Institute is embarking on an ambitious new branding strategy—I suggest that getting Caltech's home town to spell its name right would be a good place to start.)