7 ms·
The python overhead is something I've noticed as well in a system that runs a lot of python scripts. Especially with a few more modules imported, the interprete
by fabian2k 6y ago
The python overhead is something I've noticed as well in a system that runs a lot of python scripts. Especially with a few more modules imported, the interpreter and module loading overhead can be quite significant for short running scripts.
Numpy was particularly slow during imports, but I didn't see an easy way to fix this apart from removing it entirely. My impression was that it does a significant amount of work on module loading, without a way around it.
I think the other side of "surprisingly slow" is that computers are generally very fast, and the things we tend to think of as the "real" work can often be faster than this kind of stuff that we don't think about that much.
- nullify88 6y agoI see this alot with Ansible. Its not particularly slow but running it places a bigger burden on laptop cpu and fans than I'd imagined.
- throwdbaaway 6y agoNoticed the same too. It is likely that we are both impacted by the very aggressive default of 1ms for `internal_poll_interval`: https://github.com/ceph/ceph-cm-ansible/pull/308 https://github.com/ceph/ceph-cm-ansible/pull/308
- nullify88 6y agoThanks for the heads up, i'll give that a go :)
- mkj 6y agoHuh, polling with a sleep() rather than a proper event wait seems like a bad code smell there...
- necheffa 6y agoHow so? This is almost certainly what the internals of a packaged event wait would look like.
- mkj 6y agoIt could use a Queue of some kind rather than just pushing onto a deque between threads [1]? Then it would idly wait for the results_thread_main to push results. I guess that might be what Mitogen does (along with other improvements), a faster Ansible strategy [2]. [1] https://github.com/ansible/ansible/blob/becf9416736dc911d3411b92f09512b4dae2955c/lib/ansible/plugins/strategy/__init__.py#L109 https://github.com/ansible/ansible/blob/becf9416736dc911d341... [2] https://github.com/mitogen-hq/mitogen/blob/master/docs/ansible_detailed.rst https://github.com/mitogen-hq/mitogen/blob/master/docs/ansib...