4 ms·

You were right on both of the issues you seemed unsure.. The issues you flagged are closed by version 1.1.1 - also mentioned in issue threads. On the codec su

You were right on both of the issues you seemed unsure.. The issues you flagged are closed by version 1.1.1 - also mentioned in issue threads. On the codec suggestion ... it seems to go against my design motivation ...A file with "# -- coding: lucen --" doesn't degrade to native python when Lucen isn't installed, it fails to compile: "SyntaxError: encoding problem: lucen", which fails the property that a file with lucen absent will run as native python. But the import order problem is real so i'm investigating about a .pth based hook installation at interpreter startup ... Threads only i'm not taking as on a GIL build thread cannot spedup cpu bound python. Thanks for all the suggestions tho..

zbentley 2mo ago
Thanks, I saw the pings on github! Definitely let me know if any of the additional things I mentioned are worth my doing some testing or filing issues/feature requests, always happy to help out. Re: the codec option, that's fair; I hadn't thought of that angle. I think your options are 1) stick with what you have, 2) use a codec and require anyone using the codec have Lucen installed, but have it disabled by default, or 3) add a runtime component (like a "with require_lucen_to_be_imported_before_this_code_was_compiled()" context manager or something) that emits an error or warning when someone's import of Lucen happened too late. Re: threads, that's a fair point about the GIL, but I really think that there are a lot of unappreciated drawbacks to the process model. Just off the top of my head: depending on your OS, the multiprocessing backend used inside ProcessPoolExecutor may reinitialize/reimport all your Python code in the worker, throwing off your profitability analysis due to an extremely expensive startup and causing the memory consumption of Lucen'd code to go crazy; ProcessPoolExecutor suddenly requires everything to be pickleable, which lots of things aren't; PPE can leave orphaned or hung/stuck processes across the stack; extra diligence is required when surfacing exceptions back up from the PPE due to pickleability of the exception stack and everything on it; many many more.
- soumik15630m 2mo ago
  thanks again! will consider these suggestions ... and also feel free to open issues and pr in the repo if you're interested

zbentley 2mo ago

Thanks, I saw the pings on github! Definitely let me know if any of the additional things I mentioned are worth my doing some testing or filing issues/feature requests, always happy to help out. Re: the codec option, that's fair; I hadn't thought of that angle. I think your options are 1) stick with what you have, 2) use a codec and require anyone using the codec have Lucen installed, but have it disabled by default, or 3) add a runtime component (like a "with require_lucen_to_be_imported_before_this_code_was_compiled()" context manager or something) that emits an error or warning when someone's import of Lucen happened too late. Re: threads, that's a fair point about the GIL, but I really think that there are a lot of unappreciated drawbacks to the process model. Just off the top of my head: depending on your OS, the multiprocessing backend used inside ProcessPoolExecutor may reinitialize/reimport all your Python code in the worker, throwing off your profitability analysis due to an extremely expensive startup and causing the memory consumption of Lucen'd code to go crazy; ProcessPoolExecutor suddenly requires everything to be pickleable, which lots of things aren't; PPE can leave orphaned or hung/stuck processes across the stack; extra diligence is required when surfacing exceptions back up from the PPE due to pickleability of the exception stack and everything on it; many many more.

soumik15630m 2mo ago
thanks again! will consider these suggestions ... and also feel free to open issues and pr in the repo if you're interested