Statistically, they are right in assuming that you are probably not a speaker of Japanese. Of all white (or black) folk in Japan, perhaps 5% speaks enough Japan

Statistically, they are right in assuming that you are probably not a speaker of Japanese. Of all white (or black) folk in Japan, perhaps 5% speaks enough Japanese to carry an every day conversation. The rest are tourists, business folk, and weeaboos. It never bothered me. Whenever I initiated a conversation people just spoke Japanese. Sometimes people would start in English, but the conversation always effortlessly slid into Japanese when I answered in Japanese. Further down from Tokyo the experience tends to be even simpler at times: either you speak Japanese, or you have no conversations at all.