Mostly IO bound. In this case the actual limit was the API rate limiting of the geocoder I was using. A couple of thousand calls per minute. Quite a bit more th

Mostly IO bound. In this case the actual limit was the API rate limiting of the geocoder I was using. A couple of thousand calls per minute. Quite a bit more than what you can do with a single thread but not quite what a decent laptop would be able to do. Python has blocking IO. So a network call blocks the process. So if you have 250ms response times, you are doing 4 requests per second. Without the GIL, threads would be a good way to scale that. With 10 threads you should be able to do 2400 requests per minute. But with the GIL forget about that. With co-routines and non blocking IO, it could all be single threaded. There are some ways to do that with python of course but then you are going to have to use some specialized frameworks and step away from the standard library a bit. Using the shell vs. the multi processing module is the same difference. I've done both with python. I've a slight preference for the multi processing module so I don't have to deal with bash weirdness on top of all the python boiler plate. The first time I did stuff like that with python was probably 13 years ago or so around 2010. Not a lot has changed or progressed on this front since then in python. The GIL made scaling this unnecessarily hard then and it still does. Threads are a no go because of it, the standard library mostly offers blocking IO, and when you go with multiprocessing things like shared memory are very limited so you end up using files or databases for state. Things like node.js, go, or kotlin would handle this type of work load with a lot less fuss. With Kotlin, I'd be using co-routines and some multi threaded scope to launch them in. Or I could build on some of the Java internals. Or a mix of both. I'd be able to write similar code and choose between blocking or non blocking IO. I'd be forking and joining things. Maybe use channels and flows to pass data around and rely on back pressure to keep things progressing at a more or less optimal rate. Not an option for this project as my client just is more python focused and that's fine. But just signaling that python is a bit out of its depth here. Just singling out kotlin here because I just have spend a lot of time with it. If you have a hammer everything seems a nail. I think python could be so much better but that starts with modernizing its internals. Mojo seems like a huge step in the right direction.