3 ms·
Yeah Python got done dirty here. I tried a million tasks and I saw a peak of about 166MB. I'm still getting used to async in Python, but this seems to work fine
by mrj 3y ago
Yeah Python got done dirty here. I tried a million tasks and I saw a peak of about 166MB. I'm still getting used to async in Python, but this seems to work fine though?
import asyncio
async def do_task():
await asyncio.sleep(10)
async def main(num_tasks: int):
tasks = []
for task_id in range(num_tasks):
task = asyncio.create_task(do_task())
tasks.append(task)
print("spawned tasks")
await asyncio.gather(*tasks)
if __name__ == "__main__":
asyncio.run(main(1_000_000))
- pkolaczk 3y agoThank you! I fixed the code and updated the benchmarks. However, your 166 MB doesn't match my testing (I'm getting about 240M at 100k tasks) and 10x more at 1M. Are you sure you haven't misread your RSS?
- mrj 3y agoNope, that's what it is for me but you're using an older Python. 3.11 has a lot of improvements and not sure why you wouldn't use the latest?
- d0mine 3y agoIf Python 3.11 is available, then asyncio.TaskGroup could be used instead of asyncio.gather() https://docs.python.org/3/library/asyncio-task.html#task-groups https://docs.python.org/3/library/asyncio-task.html#task-gro...
- pkolaczk 3y agoRunning Ubuntu LTS and that was the version from the official repo. Will try a newer version then.