It matters if you want a career in a 'professional service' i.e. politics, banking, law etc and to some extent the business world purely b/c these institutions

It matters if you want a career in a 'professional service' i.e. politics, banking, law etc and to some extent the business world purely b/c these institutions not only offer you a brand name education but the opportunities (work experience, public speakers from a range of industries, extra-curriculats etc) on offer are far far superior to lower ranked colleges.