You just are not "smart enough" (don't have enough working memory and attention capacity) to fit all (or any) these people thoughts in your "observation window"

You just are not "smart enough" (don't have enough working memory and attention capacity) to fit all (or any) these people thoughts in your "observation window", you already have immense amount of information processed consciously and unconsciously in it and all around it which clouds this function of perception to the point of near-impossibility. Some meditation techniques can let you develop nearly direct perception without ideas but as soon as you want to use this to read someone's mind that already is an idea and makes this impossible.