5 ms·
Whatever works for you, but changing 20 lines of code in this to use asyncio will send your throughput through the roof.
by orf 10y ago
Whatever works for you, but changing 20 lines of code in this to use asyncio will send your throughput through the roof.
- revelation 10y agoThe limiting factor for a fast TCP scanner is not your favorite async toolkit.
- orf 10y agoWell, no, of course not. Don't be stupid. The default ThreadPoolExecutor size is the number of cores you have on your laptop. In my case, 8. So that's 8 concurrent connections at once. That's, frankly, shit. Use asyncio (or roll your own toolkit, or just spawn 'nmap') and you get 20000 concurrent connections. Which is better?
- rantanplan 10y agoYou're conflating concurrency with parallelism. Your 20000 concurrent connections with asyncio are not parallel. You can have 20000 threads if you want.
- orf 10y agoThe connections are actually handled by the OS in parallel, it's the callbacks to Python that are not. Why do you need a whole OS thread to handle sending a SYN/ACK? As you well know 20000 threads is not a great way to do anything. Especially in Python.
- the_mitsuhiko 10y agoI agree that threads in Python are not great but 20000 threads is otherwise not an issue.
- fzzzy 10y agoAre you serious? Have you ever tried starting 20k threads?
- fzzzy 10y agoAre you sure about that? Have you ever tried starting 20,000 threads? Try it.
- njharman 10y agoNot using something to deserialize all that blocking I/O waiting on the internet is THE limiting factor.