2 ms·
It's because threads in Python are only really good for parallel I/O, and are ineffective for CPU bound workloads. This can be a problem for a lot of treading
by heavyset_go 5y ago
It's because threads in Python are only really good for parallel I/O, and are ineffective for CPU bound workloads.
This can be a problem for a lot of treading use cases. If I'm working on an ETL app that parses large amounts of data, either the related CPU bound tasks need to either run sequentially, call out to C extensions, or use multiple processes which incurs an overhead.
It's a pain when you know threads would suit your use case well, but the threading implementation in the language you're working in isn't up to the task.
- Godel_unicode 5y agoWhile it would be nice to have the language be able to do it, the fact that you can't today leads to some tidy separation of concerns for parallelization. For instance, many people I've talked to use things like Spark or dask to get high scale on data processing tasks. That means that all of the management of distributed jobs is handled through an easily googleable framework that your ops team can manage, as opposed to needing to build all of that yourself. I see this as being a nice stopgap solution for those who are too big for single-threaded but not big enough to need Spark.
- deleted 5y ago[deleted]
- ketozhang 5y agoIt's very interesting you mentioned ETL tasks. In ETL batch jobs, a unit in the batch is defined small enough to rarely be CPU bound; rather, as you mentioned, it is I/O bound. In what situation must you define a unit of work to be so heavily CPU bound? To me that's a smell for too large of a unit.
- heavyset_go 5y agoI'm using ETL as shorthand for a "I wrote a script at home that parses my data and puts it in a database, and threads might save time" situation. I wouldn't reach for a thread pool for anything serious.