3 ms·
If your problem is memory-bound then no compiler will help you. Memory-mapping a serialisation of the dict would help, but I'm not sure how you would do that o
by 314 11y ago
If your problem is memory-bound then no compiler will help you.
Memory-mapping a serialisation of the dict would help, but I'm not sure how you would do that outside of an extension lib.
- unhammer 11y ago1m for 11GB sounds IO-bound, but yeah that's still not a use-case where switching to pypy will make a difference. But did you try using it as a service (ie. avoid the startup time) and timing the actual translation? At work, we switched to pypy for the speed. For a spell-checker I made in python, pypy made the dictionary compiling 3x faster and the spelling 2x faster, so it can really matter. On the other hand, pypy's memory usage is often a bit worse, and startup time can also be a bit slower, at least if your program has an otherwise "zero" startup time.
- andor 11y agoNo, I didn't try to optimize this any further. I just checked whether any of the new interpreters is really faster. It's a simple one-off experiment, and compared to real statistical machine translation with Moses on the same data the translation is pretty much instant. Moses needs something on the order of 1600 CPU-hours (50 nodes * 8 threads * 4 hours).