Agreed, you might be able to get better results brute forcing multiple colors at once at a lower color resolution, then gradient descent the optimal colors from

Agreed, you might be able to get better results brute forcing multiple colors at once at a lower color resolution, then gradient descent the optimal colors from there. One other obvious optimization is to use a more sophisticated metric for "closeness" based on human color perception, not just simple pixel value delta- that could make a huge difference for the skin tones in the final example image.