7 ms·

What Color Is This? Part 2

182 points by jkubicek 6y ago

psychometry 6y ago
Is anyone else surprised that stitchfix has been successful for long enough that they can keep such a skilled and productive data science team around? Especially considering how much churn subscription box companies have, I'm shocked that they're apparently doing so well.
- taytus 6y ago
  This is what funding allows. You don't desperately need to be a super success because you have reserves feeding that growth.
  - mkrazzledazzle 6y ago
    Bruh you just don’t get how funding works
- anindha 6y ago
  The do really well in areas like middle America where the options are limited. I am not surprised they are doing so well.
- mbreese 6y ago
  They need a skilled data science team. That's the only way for them to have been so successful. When you're a small company, using your data science team to optimize every interaction is key. It's not just the image analysis, but also logistics, inventory, shipping, etc.
  - altdatathrow 6y ago
    Earlier this year Stitchfix had around 8,000 employees and 115 data scientists, and operated at about $55k of revenue per employee. They are neither small nor particularly successful. [1] https://www.forbes.com/sites/stevenli1/2020/02/17/stitch-fix-data-science/ https://www.forbes.com/sites/stevenli1/2020/02/17/stitch-fix...
  - IshKebab 6y ago
    Do they? I'm pretty sure an intern (or mTurk) could do this manually and more accurately for much less than the salary of whoever wrote this post.
    - mbreese 6y ago
      I look at posts like this as a very small view into one of the problems they've worked on. This post really offered little by way of rationale for why they are extracting colors. The earlier post offers a bit more, but not much. Instead, I think that a post like this describes how they approach a challenge and how they automate work that would require manual annotation. I actually think of this analysis more as decision support (making manual review easier) as opposed to full-on automation.
- CitrusFruits 6y ago
  I'm not surprised. 1. It's a pretty common misconception that StitchFix is subscription only. A lot of their business is ad hoc just pay for a "fix". 2. Also, as far as I understand it, they are just as much a tech company as they are a retailer, if not more. 3. I know someone who works their, and from what they tell me StitchFix seems to really have their act together and just be well run in general.
pippy 6y ago
I did something similar to this. I wanted "Lazy loaded" SVG images. Generating an average color over an area would render horrific results - an area with equal red green and blue would be gray for example. My solution was to convert a target pixel area into a histogram in HSL color space, select the most populated color area and take an average of that populated area. You could then do smart things like render edges along where the change reached a threshold so you got clean edges. Once you blurred the SVG it was a fraction the size of a down scaled raster image (using imagemagick). There's all kinds of very interesting pitfalls when dealing with color. RGB is great for transmitting data but horrible when applying operations. For example, your eye converts RGB in a way so that yellow essentially becomes a primary. Another is the eye perceives photons in a logarithmic fashion adding two colors and halving them doesn't make an average. This article has some great info in it: https://blog.asmartbear.com/color-wheels.html https://blog.asmartbear.com/color-wheels.html
- deleted 6y ago
  [deleted]
- deleted 6y ago
  [deleted]
- dumpsterdiver 6y ago
  > For example, your eye converts RGB in a way so that yellow essentially becomes a primary. Another is the eye perceives photons in a logarithmic fashion adding two colors and halving them doesn't make an average. Coming from an audio background where we have the fletcher-munson noise curve, I'm really curious to learn more about how the eye perceives light. Does anyone have other sources for this, please?
  - dahart 6y ago
    Oh yeah there are tons of sources, whole books on color physiology and perception. Wikipedia really isn’t a bad place to start - see the cone response curves at the top of this section https://en.wikipedia.org/wiki/Color_vision#Physiology_of_color_perception https://en.wikipedia.org/wiki/Color_vision#Physiology_of_col... A somewhat similar idea to the Fletcher-Munson noise curve is the “just noticeable difference” https://en.wikipedia.org/wiki/Just-noticeable_difference https://en.wikipedia.org/wiki/Just-noticeable_difference which can be mapped across color differences https://en.wikipedia.org/wiki/Color_difference https://en.wikipedia.org/wiki/Color_difference to figure out how people perceive the brightness of a given color https://en.wikipedia.org/wiki/Brightness https://en.wikipedia.org/wiki/Brightness
Waterluvian 6y ago
Is all of this because the images they have of their own products aren’t taken with so much light fill that all shadows are removed? Like without any shadows doesn’t this become a more trivial problem?
- wffurr 6y ago
  If they could control the photo taking process, couldn't they also just have the photo takers label the colors as well?
  - Waterluvian 6y ago
    Good point. Just colour gun boom boom.
  - Agentlien 6y ago
    That reminds me of photogrammetry for environments in games. Sometimes when you take, photos of an environment you include an object of known size and a spectrum of known colours for reference.
helloiloveyou 6y ago
I also had to implement something similar to this and wrote about it: https://www.mikealche.com/software-development/how-to-implement-search-by-color-when-all-you-have-is-a-good-coffee https://www.mikealche.com/software-development/how-to-implem...
im3w1l 6y ago
For the shadows, maybe consider a method where first take a picture with light from one direction, then from a different direction. This could be done automatically by toggling multiple light sources quickly.
- tincholio 6y ago
  Or, using a proper lighting setup, so as to minimize shadows. This would likely solve most of their shadow-related problems.
- contravariant 6y ago
  You could also cluster first purely on colour (using the xy part of the xyY colourspace). For greyscale clothing you might then need to see if some colour has clearly separate shades, but that's going to be tricky without a proper understanding of shadows.
- Atomskun 6y ago
  One could also remove the expected shadow values (and hues) depending on the color of the clothing and the color of the light, at least approximately. Should work as long as there aren't complicated interactions of the material with light that hits it.
ivanbakel 6y ago
A very nice writeup, but (no offense) "tens of seconds" per image? Maybe I haven't understood the technical challenges well enough, but processing an image of 60,000 pixels should surely be much faster than that. What would the cost center of such a process even be - building the superpixels?
- thdrdt 6y ago
  Exacty what I was thinking. Some years ago I also tried this and had great results by just resizing the image first to something like 100x100 pixels.
  - LoSboccacc 6y ago
    resizing is quite disruptive to patterns we got good results for that specific failure mode with random sampling. how much samples to take require some tuning, but you can use the image data itself to facilitate the tuning part by exiting the random sampling when the deviation of sampled pixels is below a factor proportional to samples/totals
- rasz 6y ago
  Probably the result of using python for number crunching.
m12k 6y ago
In game development there are already pretty mature photogrammetry techniques that reverse engineer the lighting and shadow conditions of a scene in order to "subtract" these from objects in the scene so you can get the neutral albedo of their surfaces (this is needed for physically based rendering, so the texture in the game engine is doesn't have highlights or shadows baked into it, and can thus get applied dynamically by the engine instead based on the ingame surroundings). It'd be interesting to see if a technique like that could be applied here too. And on the flipside, if you want to do this using ML, rendered images from a game engine/blender could be used for generating large amounts of training data with perfectly defined "ground truth" colors already known.
- maverwa 6y ago
  I always thought that these techniques only work because they use multiple viewpoints of the same object (with the same lightning), can infer topography from these and then, knowing that, can "figure out" the lighting. Would be nice to know how well this would work with just very few images, maybe even with different positions, let alone a single image.
chris_st 6y ago
Really wish those lovely visualizations in the "Part 1" had pause buttons... some of the transitions are just too fast.

182 points by jkubicek 6y ago

Is anyone else surprised that stitchfix has been successful for long enough that they can keep such a skilled and productive data science team around? Especially considering how much churn subscription box companies have, I'm shocked that they're apparently doing so well.

taytus 6y ago
This is what funding allows. You don't desperately need to be a super success because you have reserves feeding that growth.
- mkrazzledazzle 6y ago
  Bruh you just don’t get how funding works
anindha 6y ago
The do really well in areas like middle America where the options are limited. I am not surprised they are doing so well.
mbreese 6y ago
They need a skilled data science team. That's the only way for them to have been so successful. When you're a small company, using your data science team to optimize every interaction is key. It's not just the image analysis, but also logistics, inventory, shipping, etc.
- altdatathrow 6y ago
  Earlier this year Stitchfix had around 8,000 employees and 115 data scientists, and operated at about $55k of revenue per employee. They are neither small nor particularly successful. [1] https://www.forbes.com/sites/stevenli1/2020/02/17/stitch-fix-data-science/ https://www.forbes.com/sites/stevenli1/2020/02/17/stitch-fix...
- IshKebab 6y ago
  Do they? I'm pretty sure an intern (or mTurk) could do this manually and more accurately for much less than the salary of whoever wrote this post.
  - mbreese 6y ago
    I look at posts like this as a very small view into one of the problems they've worked on. This post really offered little by way of rationale for why they are extracting colors. The earlier post offers a bit more, but not much. Instead, I think that a post like this describes how they approach a challenge and how they automate work that would require manual annotation. I actually think of this analysis more as decision support (making manual review easier) as opposed to full-on automation.
CitrusFruits 6y ago
I'm not surprised. 1. It's a pretty common misconception that StitchFix is subscription only. A lot of their business is ad hoc just pay for a "fix". 2. Also, as far as I understand it, they are just as much a tech company as they are a retailer, if not more. 3. I know someone who works their, and from what they tell me StitchFix seems to really have their act together and just be well run in general.

pippy 6y ago

I did something similar to this. I wanted "Lazy loaded" SVG images. Generating an average color over an area would render horrific results - an area with equal red green and blue would be gray for example. My solution was to convert a target pixel area into a histogram in HSL color space, select the most populated color area and take an average of that populated area. You could then do smart things like render edges along where the change reached a threshold so you got clean edges. Once you blurred the SVG it was a fraction the size of a down scaled raster image (using imagemagick). There's all kinds of very interesting pitfalls when dealing with color. RGB is great for transmitting data but horrible when applying operations. For example, your eye converts RGB in a way so that yellow essentially becomes a primary. Another is the eye perceives photons in a logarithmic fashion adding two colors and halving them doesn't make an average. This article has some great info in it: https://blog.asmartbear.com/color-wheels.html https://blog.asmartbear.com/color-wheels.html

deleted 6y ago
[deleted]
deleted 6y ago
[deleted]
dumpsterdiver 6y ago
> For example, your eye converts RGB in a way so that yellow essentially becomes a primary. Another is the eye perceives photons in a logarithmic fashion adding two colors and halving them doesn't make an average. Coming from an audio background where we have the fletcher-munson noise curve, I'm really curious to learn more about how the eye perceives light. Does anyone have other sources for this, please?
- dahart 6y ago
  Oh yeah there are tons of sources, whole books on color physiology and perception. Wikipedia really isn’t a bad place to start - see the cone response curves at the top of this section https://en.wikipedia.org/wiki/Color_vision#Physiology_of_color_perception https://en.wikipedia.org/wiki/Color_vision#Physiology_of_col... A somewhat similar idea to the Fletcher-Munson noise curve is the “just noticeable difference” https://en.wikipedia.org/wiki/Just-noticeable_difference https://en.wikipedia.org/wiki/Just-noticeable_difference which can be mapped across color differences https://en.wikipedia.org/wiki/Color_difference https://en.wikipedia.org/wiki/Color_difference to figure out how people perceive the brightness of a given color https://en.wikipedia.org/wiki/Brightness https://en.wikipedia.org/wiki/Brightness

Waterluvian 6y ago

Is all of this because the images they have of their own products aren’t taken with so much light fill that all shadows are removed? Like without any shadows doesn’t this become a more trivial problem?

wffurr 6y ago
If they could control the photo taking process, couldn't they also just have the photo takers label the colors as well?
- Waterluvian 6y ago
  Good point. Just colour gun boom boom.
- Agentlien 6y ago
  That reminds me of photogrammetry for environments in games. Sometimes when you take, photos of an environment you include an object of known size and a spectrum of known colours for reference.

helloiloveyou 6y ago

I also had to implement something similar to this and wrote about it: https://www.mikealche.com/software-development/how-to-implement-search-by-color-when-all-you-have-is-a-good-coffee https://www.mikealche.com/software-development/how-to-implem...

im3w1l 6y ago

For the shadows, maybe consider a method where first take a picture with light from one direction, then from a different direction. This could be done automatically by toggling multiple light sources quickly.

tincholio 6y ago
Or, using a proper lighting setup, so as to minimize shadows. This would likely solve most of their shadow-related problems.
contravariant 6y ago
You could also cluster first purely on colour (using the xy part of the xyY colourspace). For greyscale clothing you might then need to see if some colour has clearly separate shades, but that's going to be tricky without a proper understanding of shadows.
Atomskun 6y ago
One could also remove the expected shadow values (and hues) depending on the color of the clothing and the color of the light, at least approximately. Should work as long as there aren't complicated interactions of the material with light that hits it.

ivanbakel 6y ago

A very nice writeup, but (no offense) "tens of seconds" per image? Maybe I haven't understood the technical challenges well enough, but processing an image of 60,000 pixels should surely be much faster than that. What would the cost center of such a process even be - building the superpixels?

thdrdt 6y ago
Exacty what I was thinking. Some years ago I also tried this and had great results by just resizing the image first to something like 100x100 pixels.
- LoSboccacc 6y ago
  resizing is quite disruptive to patterns we got good results for that specific failure mode with random sampling. how much samples to take require some tuning, but you can use the image data itself to facilitate the tuning part by exiting the random sampling when the deviation of sampled pixels is below a factor proportional to samples/totals
rasz 6y ago
Probably the result of using python for number crunching.

m12k 6y ago

In game development there are already pretty mature photogrammetry techniques that reverse engineer the lighting and shadow conditions of a scene in order to "subtract" these from objects in the scene so you can get the neutral albedo of their surfaces (this is needed for physically based rendering, so the texture in the game engine is doesn't have highlights or shadows baked into it, and can thus get applied dynamically by the engine instead based on the ingame surroundings). It'd be interesting to see if a technique like that could be applied here too. And on the flipside, if you want to do this using ML, rendered images from a game engine/blender could be used for generating large amounts of training data with perfectly defined "ground truth" colors already known.

maverwa 6y ago
I always thought that these techniques only work because they use multiple viewpoints of the same object (with the same lightning), can infer topography from these and then, knowing that, can "figure out" the lighting. Would be nice to know how well this would work with just very few images, maybe even with different positions, let alone a single image.

chris_st 6y ago

Really wish those lovely visualizations in the "Part 1" had pause buttons... some of the transitions are just too fast.