3 ms·

Ask HN: Should expert opinion be a bigger part of the Machine Learning world?

3 points by SleekEagle 5y ago

I came across a Twitter thread (linked in comments) which shows some interesting results when trying to recolor historical photos that have been decolored. The recolored photos are a lot drabber than the original (example in comments), and the thread author says that this gives us a skewed view of the past, making us think the past was a lot more boring than it was. I thought of some related questions that I thought could be good for discussion here: - Do you think that expert opinion should be consulted in the Machine Learning process more? If so, where? (perhaps omitting Expert Systems) - Is there too much faith that a result from an ML model is the "right" result? (a phenomenon that maybe isn't specific to ML but a result of human tendencies?) - Do ML practitioners have a responsibility to clearly communicate to the general public the limitations and degree-of-confidence in these systems? - Am I reading too much into this, and this colorization model is just a fun model to play with, and the conclusions of the Twitter thread are too speculative or conjectural? - Is this colorization issue just another form of bias that needs to be ironed out? - The thread concludes by saying that colorization should be left to experts who can use context to pick accurate colors. I think this is too extreme, and that ML systems can incorporate expertise when training, or after during evaluation. Do you think there are any jobs/problems that ML methods could be applied to but should be left to experts (some considerations might be safety, privacy, ethics, etc.) I know that ultimately a lot of these questions can simply boil down to statistics and their interpretation, so I'm not sure exactly where discussion could/should/will lead, but I'm looking forward to hearing your opinions!

SleekEagle 5y ago
Example images: https://imgur.com/a/2MCxNR6 https://imgur.com/a/2MCxNR6 Twitter thread: https://twitter.com/gwenckatz/status/1381652071695351810?lang=en https://twitter.com/gwenckatz/status/1381652071695351810?lan...
jstx1 5y ago
If we're talking about the specific example - DeepAI's API isn't perfect (as the twitter thread demonstrates), that's about it. > Am I reading too much into this Kind of. If you're trying to generalize to all ML problems based on this one example, the discussion is too broad to say anything meaningful. Try to apply your questions to the specific example and you'll see how little sense they make: - consulting experts - who would be the expert when trying to guess the color of a 19th century carpet? - faith in the results - does anyone care about the specific results in this case? I mean it's a cool API but the end result is purely aesthetic, it's not like something important is happening in the world based on the results - do ML practitioners have responsibility to communicate to the general public - no, the people who are writing the ML API have a responsibility to their customers and to the internal product stakeholders in the company. The general public has nothing to do with any of this. And then there's hundreds different configurations of [ML application + person creating it + company they work for + user + real world impact] for which the answers will be completely different.
- SleekEagle 5y ago
  Thanks for the reply! I don't really have a good frame of reference for the model-improvement feedback loop in different industries, so I could very well be generalizing incorrectly. As for the experts part, I assume there is some expert out there who would know enough about the economy/trade at the time that they could glean enough information about materials, dyes, etc. to make an educated guess/ Faith in results - while maybe in this simple example that's valid, what about in cases where ethics are concerned? IIRC there was a police force using an ML model to predict where units should be patrolling and it turned out they were overpolicing primarily black neighborhoods. Same thing with ML systems for employee applicant screening. Communication - I think that this is a fair point, although given how pervasive ML is at this point I think it's good to try make sure the view of the public is not too skewed (like the whole "God particle" fiasco several years ago) Either way, thanks for the reply! You make some good points and I don't think many of my questions were well-framed enough.
PaulHoule 5y ago
My take. I have a rug of the same style as that rug and the colors are pretty close to the reconstructed colors even if the reconstructed colors are "a little drab" I think there are many copies of that rug with the exact same colors but I don't think you can go to a store and find a rug with the same pattern in different colors. The hue of the garment is completely wrong but it might be impossible to tell what the true colors are because I think you could go to a store and find a very similar garment with different hues. In the case of image recoloring the "expert" might well be the person who is supervising the inference process. For many purposes the endpoint might be that that person likes the result. If it's really important to you that the hue is right you need to have an "expert" do research the way people would do research if they were colorizing a movie.
- SleekEagle 5y ago
  Thanks for the reply - sorry if it wasn't clear but by "expert" I meant an expert on e.g. early 20th century clothing or something like that who would confirm or refute the performance of the recolorization algorithm
  - PaulHoule 5y ago
    Sure. If it's important to you that the colors come out like the original you are going to need that expert. That expert might very well tell you that the problem is well-determined in some cases (I don't think you'll ever see that rug pattern with different colors) and in other cases isn't (right now I am wearing a hoodie I bought at Target that comes in I think 7 different colors, so all an expert can tell you is that it could be one of many colors.) That's an unappreciated point in machine learning. In some cases we can classify something accurately, in some cases the answer is always going to be ambiguous. From a practical aspect some answers are critical to the success of the application, in other cases it makes no difference if item X is classified in class A or class B. Developing a training set you can drive yourself batty accurately classifying things that don't matter and still have a training set that's worthless because the important things aren't determined.
    - 5y ago