I'd go further and say that among the intellectual elite, having a degree is more based on whether or not your interests overlap with things that colleges have

I'd go further and say that among the intellectual elite, having a degree is more based on whether or not your interests overlap with things that colleges have to offer than some default for very smart people. It is not the default now.