maxlam

5 ms·

1.
by maxlam 9y ago
That's definitely an interesting idea -- it seems this would allow for boundaries that "change" along with the data (instead of having static boundaries as it is). Would be interested to know how that turns out!
2.
by maxlam 9y ago
Haven't tried this but this is definitely a good idea for visualizing what's going on!
3.
by maxlam 9y ago
Hm what do you mean? I'm not quite seeing how to differentiate with respect to the quantization steps.
4.
by maxlam 9y ago
Absolutely, please do!
5.
by maxlam 9y ago
Training these quantized word vectors has to be done in full precision (so no memory gains during training the word vectors). But when you save them to disk every value is either -1/3 or +1/3 so one could encode the word vectors i
6.
by maxlam 9y ago
Here's the one for man: ['man', 'woman', 'boy', 'handsome', 'stranger', 'gentleman', 'young', 'drunkard', 'devil', 'lonely', 'lady
7.
by maxlam 9y ago
Thanks for the compliments! And thanks also for bringing up the debiasing! It's interesting to see what people care about and any conversation that leads to new ideas is a plus.
8.
by maxlam 9y ago
Hm not sure, would need to think more about this -- definitely an interesting idea though!
9.
by maxlam 9y ago
You can kind of see that words that are similar have similar looking vector values (that's why there are vertical stripes of yellow / black). But you're right in that most of it just looks like noise. I put the picture there
10.
by maxlam 9y ago
Definitely tried to figure out if the dimensions mean anything -- as far as I can tell they don't really mean much :(
11.
by maxlam 9y ago
Interesting, definitely need to think about debiasing. Seems like it won't really work straight out of the box since it'd destroy the 1 bit-ness of the vectors. Though if only a few of the vectors are de-biased then you can still
12.
by maxlam 9y ago
The idea is that you can kind of capture the "meaning" of a word with a sequence of numbers (a vector) -- and then you use these vectors for machine learning tasks to do cool stuff like answer questions! Word2Vec is one of the alg
13.
by maxlam 9y ago
You're definitely right, the quantization function and its values definitely have an impact on performance. For 1 bit I think I tried something like -1/+1, -.5/+.5, -.25/+.25, -.333/+.333. and something like -10&#x2
14.
by maxlam 9y ago
Main reason I did it this way is because Facebook's DrQA (which I evaluate the vectors on for the SQuAD task) uses case sensitive vectors. Was a tough decision between choosing whether to train case sensitive vectors / case insens
15.
by maxlam 9y ago
This might be because "Artist" has an uppercase "A" -- I trained all the word vectors to be case sensitive so "Artist" is not the same as "artist" (which should be closer to "man" than &quot
16.
by maxlam 9y ago
Oops, thought you meant the graphs (with the dotted/solid lines) in the writeup. If you're referring to the image under "Visualizing Quantized Word Vectors" then each row is a word vector (and there are only two colors s
17.
by maxlam 9y ago
Yeah, I should definitely put more detail in the writeup -- thanks for the feedback! What's happening with figure 1a (epochs vs google accuracy) is that as you train for more epochs the full precision loss continues to decrease (dotted
18.
Show HN: Word2Bits – Quantized Word Vectors(github.com)
221 points by maxlam 9y ago|110 comments