3 ms·
Thanks, I saw the pings on github! Definitely let me know if any of the additional things I mentioned are worth my doing some testing or filing issues/feature r
by zbentley 2mo ago
Thanks, I saw the pings on github! Definitely let me know if any of the additional things I mentioned are worth my doing some testing or filing issues/feature requests, always happy to help out.
Re: the codec option, that's fair; I hadn't thought of that angle. I think your options are 1) stick with what you have, 2) use a codec and require anyone using the codec have Lucen installed, but have it disabled by default, or 3) add a runtime component (like a "with require_lucen_to_be_imported_before_this_code_was_compiled()" context manager or something) that emits an error or warning when someone's import of Lucen happened too late.
Re: threads, that's a fair point about the GIL, but I really think that there are a lot of unappreciated drawbacks to the process model. Just off the top of my head: depending on your OS, the multiprocessing backend used inside ProcessPoolExecutor may reinitialize/reimport all your Python code in the worker, throwing off your profitability analysis due to an extremely expensive startup and causing the memory consumption of Lucen'd code to go crazy; ProcessPoolExecutor suddenly requires everything to be pickleable, which lots of things aren't; PPE can leave orphaned or hung/stuck processes across the stack; extra diligence is required when surfacing exceptions back up from the PPE due to pickleability of the exception stack and everything on it; many many more.
- soumik15630m 2mo agothanks again! will consider these suggestions ... and also feel free to open issues and pr in the repo if you're interested