You can't use HSV because the apparent differences in H / S / V aren't linear. You need to convert it to a different color system: https://en.wikipedia.org/wik

You can't use HSV because the apparent differences in H / S / V aren't linear. You need to convert it to a different color system: https://en.wikipedia.org/wiki/Munsell_color_system https://en.wikipedia.org/wiki/Munsell_color_system What is interesting about this color system is that it isn't symmetric (your eye is better at differentiating certain hues like blue at more maximal values than others) so more care is needed on the nearest neighbour lookups. If you are using this coloring scheme for labelling colors in a photograph then you also need to adjust for colors that are especially vibrant and overtake the photo. For example this photo: http://1.bp.blogspot.com/-OkTS4Wo34UM/Uwg568lRl2I/AAAAAAAAFiM/_tawBh648b0/s1600/heart-pink-bench-love-wallpaper.jpg http://1.bp.blogspot.com/-OkTS4Wo34UM/Uwg568lRl2I/AAAAAAAAFi... Is best tagged "pink" not "brown".