4 ms·
Nice! I recently implemented a C-based numpy solution of LSH to compress / recover cosine similarity[1]. It was my first time writing Numpy C, and it was a lot
by softwaredoug 3y ago
Nice!
I recently implemented a C-based numpy solution of LSH to compress / recover cosine similarity[1]. It was my first time writing Numpy C, and it was a lot of fun to massively improve the performance over pure Python[2].
1- https://github.com/softwaredoug/np-sims https://github.com/softwaredoug/np-sims
2- https://softwaredoug.com/blog/2023/08/22/rand-projections-in-c https://softwaredoug.com/blog/2023/08/22/rand-projections-in...
- deleted 3y ago[deleted]
- eigenvalue 3y agoCool. I looked into using the CFFI stuff in Python, and I found that using PyO3/maturin was much simpler and easier, especially when it came to automatic compilation of wheels. If you're looking to make high performance functions for use in Python, I can't recommend that approach enough, and then you get to use Rust which is amazing. My experience from years ago when I was starting out in Python was that, as soon as you make people compile stuff to use your python library, you've already lost, since most people don't have the tooling/compilers installed and configured correctly, and even if they do, it's just not reasonable to expect people to wait so long for everything to compile just to use a library from pip. It's really nice to be able to start from a fresh Ubuntu install and just make a venv and instantly pip install requirements.txt using only wheels-- the whole process takes seconds and there's no fuss.