5 ms·
The author seems to have just learned that most machine learning algos are not "free lunch" but there's a good amount of poking and tuning and some "cheating" n
by raverbashing 7y ago
The author seems to have just learned that most machine learning algos are not "free lunch" but there's a good amount of poking and tuning and some "cheating" needed.
It's part of the learning curve, and I spent quite some time to learn the basics
- flo_hu 7y agoI'm fine with the free lunch thing. But here the cheating is done on the level of how people present the capabilities of the tool. If you ask the algorithm how "SHE is to LOVELY as HE is to X", the reported answer (Bolukbasi 2016) was "BRILLIANT", which in this case suggests a heavy gender-bias. But what the algorithm actually gives for X is: "LOVELY". The authors justed picked the 10th example in the list without clearly stating it.
- raverbashing 7y ago"Cheating" (in practice) usually means "embedding problem/domain specific quirks" In the "King" example, you're adding and subtracting two words that are probably very close already, so if you want to find "something else" besides itself, you need to exclude it. For some problems it might make sense, for some others it might not.
- yorwba 7y ago> The authors justed picked the 10th example in the list without clearly stating it. That's not an accurate description what Bolukbasi et al (2016) [0] did. In particular, they do not list x close to lovely + he - she and then pick arbitrarily from that list. Instead, they explicitly reject that approach (see appendix A), because they're looking for pairs of words that are maximally gendered. They do that by finding x and y such that the angle between x - y and she - he is minimized. Since the task they're solving is different, you can't fault them for getting different results. [0] https://arxiv.org/abs/1607.06520 https://arxiv.org/abs/1607.06520
- flo_hu 7y agoOk, thanks a lot for bringing this up! I will have a closer look at that.
- willvarfar 7y agoJust yesterday NHers were warned to stop using 'free lunch' in conjunction with AI: https://news.ycombinator.com/item?id=20436205 https://news.ycombinator.com/item?id=20436205
- pfortuny 7y agoSo, in some sense we are back to "eliminating outliers"... using an ad-hoc definition of "outlier". Does not look very scientific to me.
- flo_hu 7y agoExactly! I think that was part of the problem for many of the examples that turn out to not-really-work. People pretend they let the work do by an algorithm, but then hand-pick from a list of somewhat close candidates. Which of course happens with a hypothesis (and thereby desired outcome) in mind.
- Ensorceled 7y agoThat's fine if you're using it to make business decisions or embedded in a chatbot, but if you're using your new NLP algorithm to "discover" latent social biases and then make commentary .... then a "good amount of poking and tuning" is, at best, academic fraud.