I think you are underselling the empirical evidence in favor of weak Sapir-Whorf. In the 70ies Rosch did some famous experiments in Papa New Guinea with speaker

I think you are underselling the empirical evidence in favor of weak Sapir-Whorf. In the 70ies Rosch did some famous experiments in Papa New Guinea with speakers of a language with impoverished color vocabulary that demonstrated that colors outside the vocabulary remained salient (and were e.g. used to categorize objects). However since then there has been a fair amount of research that shows effects which go beyond what you would seem to allow above (e.g. hemispherical effects on color discrimination, http://www1.icsi.berkeley.edu/~kay/tics2.pdf http://www1.icsi.berkeley.edu/~kay/tics2.pdf has some examples).