3 ms·
I have been benefitted a lot by PyPy. I made a search engine[1] in python for one of my university course project. It had to process 30+ GB of html files. I g
by h43k3r 11y ago
I have been benefitted a lot by PyPy.
I made a search engine[1] in python for one of my university course project. It had to process 30+ GB of html files.
I got a 2x improvement by using PyPy and completed indexing the dataset in 11 hours as compared to projected 22 hours. The only explanation I can think of is that my work was compute intensive and involved a lot of loops and accessing data in dictionaries.
[1] https://github.com/shobhit6993/webGrep https://github.com/shobhit6993/webGrep
- ngoldbaum 11y agoA factor of two doesn't sound that great actually. You can get similar speedups by "compiling" an arbitrary python module using cython. Of course you can usually do much better than a factor of two if you think carefully about type annotations and CPython overhead.
- gtaylor 11y ago> A factor of two doesn't sound that great actually. A factor of two sounds great to me, especially if he didn't have to write Cython-specific code. That's a nice, easy win with a noticeable impact.
- fijal 11y agoit's a factor of two without thinking too hard. You can probably optimize it for pypy too which will get you more benefit. If it's "just" dict lookups there is not really all that much there is to win using cython for example