3 ms·
For something as straightforward as downloading files in parallel, why not just use Python threads? Since most of the IO will occur in C without holding the GIL
by falcolas 10y ago
For something as straightforward as downloading files in parallel, why not just use Python threads? Since most of the IO will occur in C without holding the GIL, it seems silly to be forking off entire new processes for this sole purpose.
The processing of the file in Python would hold the GIL, but this could still be resolved with a multiprocessing pool of workers.
In other words, while academically interesting, mixing Elixir and Python for web crawling doesn't make much actual sense.
- deleted 10y ago[deleted]
- klibertp 10y ago> In other words, while academically interesting, mixing Elixir and Python for web crawling doesn't make much actual sense. That's right. It wasn't supposed to be a real project, it's a just a tech demo of sorts. I will have some more real-world uses for the integration in my Raspberry project, I think, but I'm not there yet.