3 ms·
Zeroing a 100MB buffer is pegging the CPU? It takes ~30ms to zero such a buffer on my machine, which seems to imply that `iter_content` here is allocating a buf
by deathanatos 10y ago
Zeroing a 100MB buffer is pegging the CPU? It takes ~30ms to zero such a buffer on my machine, which seems to imply that `iter_content` here is allocating a buffer on each iteration of the loop. But if `iter_content` is just returning a bytes object, this would make complete sense.
One could perhaps pass in a bytearray object, or even have it return the same bytearray object each loop, with a very large warning on the tin about how you will want to take a copy if you want to keep it outside the loop's body. (i.e., the returned bytearray is owned by the iterator, and you're merely borrowing it for the loop; this is a potentially confusing API but would reduce the memory allocation churn significantly: you'd only alloc your buffer once, instead of O(download size) times.)
- AstralStorm 10y agoAt gigabit Ethernet speed (in pps) setting big chunks of memory per packet happens to be expensive. At 10GbE you actually may need specialised hardware.