I got yelled at for taking a picture of "The House That Waffle Built" in the Philippines -- standing on a sidewalk 200 feet away. I've also gotten yelled out fo

I got yelled at for taking a picture of "The House That Waffle Built" in the Philippines -- standing on a sidewalk 200 feet away. I've also gotten yelled out for trying to take a picture of a shark fin in China Town in SF. Is there some cultural faux pas that I'm missing? Or am I just unlucky.