I worked in a web scraping team for a couple of years and memory was always the bottleneck since we ran hundreds of thousands of instances. Our scrapers were wr

I worked in a web scraping team for a couple of years and memory was always the bottleneck since we ran hundreds of thousands of instances. Our scrapers were written in python and with the interpreter + all the imports we had 40MB reserved before even starting work. Not just that, but they were very sensitive to fluctuations.