3 ms·
What is the advantage of this over say using threads? Web scraping is pretty much all IO so you get big wins using threads in Python and Ruby.
by dham 11y ago
What is the advantage of this over say using threads? Web scraping is pretty much all IO so you get big wins using threads in Python and Ruby.
- poooogles 11y agoThreads in Python creates a full posix thread which is very heavyweight compared to using AsyncIO. You should get the same throughput for a far lower resource usage.
- platz 11y agodoes asyncio in python = green threads in python, therefore defeats the GIL?
- takeda 11y agoIt does not defeat GIL, but the way asyncio is working makes GIL irrelevant. In simplest use case you have a single thread (an actual OS thread) and as soon as certain operation blocks (e.g. waiting a response from server) another coroutine is scheduled. The reason why GIL is irrelevant is because only a single coroutine executes at given time[1]. The way it works is similar to cooperative multitasking. The new task that was scheduled will execute until it encounters an operation that will make it block (typically an I/O operation). [1] You actually can spin multiple threads and each could have its own event loop and then schedule coroutines on different threads.
- takeda 11y agoActually because it is all I/O is what makes it more suitable than threads. You don't have all the overhead that comes with threads, but you still get all the concurrency. That's why for example Nginx when it was released was so much faster compared to Apache.