2 ms·
I do both when I'm scaling to millions of requests per day. I also do some cpu bound preprocessing after the io bound async functions end.
by nurettin 6y ago
I do both when I'm scaling to millions of requests per day. I also do some cpu bound preprocessing after the io bound async functions end.
- genidoi 6y agoLook into Go, specifically the colly pkg. Learnt Go just so I could use this library after wrestling with highly parralelized Python web scraping. The 1k/requests/sec/core figure is true and frankly deadly, and Go borrows a lot of useful stuff from Python like array slicing. It was pretty remarkable to go from 32 cores being maxed for 100 concurrent workers to... A 10-15% usage spike on a 4 core XPS :) https://github.com/gocolly/colly https://github.com/gocolly/colly