5 ms·
Design patterns are just that, design patterns. With upsides, downsides, suitable use cases and unsuitable use cases. I was originally going through pointing o
by forgottenpass 10y ago
Design patterns are just that, design patterns. With upsides, downsides, suitable use cases and unsuitable use cases. I was originally going through pointing out all the nonsense that was outright wrong or just ruffled my feathers, but this whole article is punching above it's weight. And boils down to the following:
The author doesn't like using concurrent programing techniques in a single-threaded process.
The author does like a job queue that manages task execution across multiple processes.
This has fuck all to do with the GIL beyond the fact the presence of the GIL might have contributed to the popularity of the former in python and that celery uses multiprocessing over multithreading.
Designing a piece of software is more than cargo-culting the latest fads. Who cares if asyncio exists? Yes, we "really need" it to the extent that people want to design their programs like that. No, you don't "really need" to use it.
- markbnj 10y agoThe title is off I agree, it's not like Celery fixes cpython's problematic data structures. I don't think it was a bad article, though.
- forgottenpass 10y agoI wouldn't have minded reading someones process of learning a new design pattern if it wasn't asserting so many things that are silly or flat out wrong during the process. That's why I originally wrote, "this is a bad article." I edified that bit out because it detracts from my point unless I take the time to go through and fisk the article.
- mdomans 10y agoWell, I'm more about advertising the lockless programming pattern :) Point out the silly, please, I like it when someone actually makes honest effort to teach me something.
- scott_s 10y agoThe title, mainly. Most readers will take that to mean that you fixed the GIL itself - that is, removed it - rather than are presenting a pattern that avoids it. Your title sets up an adversarial relationship with such readers, because their attitude when they start is going to be "no, you didn't", which will cause them to miss your actual message.
- mdomans 10y agoFair point.
- deleted 10y ago[deleted]
- theseatoms 10y agoI'm curious.. in what way are cpython's data structures problematic? Do you just mean as they relate to the GIL design choice, and the downsides/limitations that follow from the overall design?
- coldtea 10y agoHe probably means the internal datastructures of Python the language (interpreter), not the datastructures it exposes to the user (list, map, tuple, etc).
- mdomans 10y agoYup. If you don't have to care about thread safety on API level, e.g. list, problems such as what's going on currently in gilectomy are no longer yours.
- DasIch 10y agoThe GIL has atomic bytecodes as a side effect and this assumption has become part of the API. There is nothing on the datastructure side that can be done to "fix" this. In any case memory access should be considered a memory management problem just like allocating and freeing memory is. It needs to be solved on the language level to create sufficiently high level user experience. You can only do that if you're willing to do things very differently such as Rusts Ownership/Borrowing approach.
- markbnj 10y agoYes, this. Sorry I was away all day and didn't have time to reply.
- mdomans 10y agoHi, I'm the author. Please point out the nonsense :) But honestly, what I'm trying to advocate here is that if we remove the problem of what's executing your tasks and the problem of implicit locking, we can stop worrying about GIL. A very good implementation of that is how Grand Central Dispatch works.
- danpalmer 10y agoI was under the impression that GCD used green threads, which means it's sharing the same address space, and ultimately would be subject to the GIL? When you're memory bound, sharing the address space is important, and Celery might not work well at all.
- mdomans 10y agoDepends on how much memory we're talking about. In regard to GCD - AFAIK it uses classic posix threads. In regard to GIL, if you can indeed guarantee that each task is atomic and lockless, you can use the same approach pyparralel uses
- danpalmer 10y agoI believe GCD actually multiplexes green threads across a pool of posix threads, which wouldn't work for Python because of the GIL.
- richardwhiuk 10y agoCorrect. GCD has dispatch_queues, and each block on the dispatch queue will be executed on one of a pool of posix threads.
- mdomans 10y agoThat's correct, but looking at the Python internals it seems that working around the GIL is possible if having this pattern exposed to end programmer.