3 ms·
Related to w2v, the proof is in the vectors. They work. And, by the way, w2v has had a whole family of variants which were built on top of the original implemen
by visarga 7y ago
Related to w2v, the proof is in the vectors. They work. And, by the way, w2v has had a whole family of variants which were built on top of the original implementation (such as Doc2Vec, Doc2VecC and others).
Related to most ML code, it's not written in C but in Python frameworks that implement whole layers as building blocks. So it's not as hard to read/change. You can take something complex like the transformer and understand it in a few minutes. The most puzzling part with framework code is figuring out the shape of tensors and, sometimes, what each dimension means.
Regarding reproducibility - it's hard to achieve on account of parallelisation. You see, neural nets use floats, and floats are not real numbers. Half the float numbers lie between [-1, 1] and the rest outside. If you do 1e-10+1e10 you get 1e10. So if you add 1000 times 1e-10 to 1e+10 you still get 1e+10, but if you add them up together first, the result is different. So float summation is not commutative. Depending on race conditions the order in which summation occurs could change and the result change as well, even if you use the same random seeds. And if we give up parallelism then we can't run the experiments any more.
- Dylan16807 7y agoFor almost all ML training, it's straightforward to break it into a fixed number of chunks without race conditions. That's if float rounding issues are even a big problem in the first place. If your results are within .1% over a few runs it's reproducible enough for most purposes.