5 ms·
Re: Semaphore: Not dropping the connection might mean we might starve others incoming msgs of resources (which might be a good thing) [0]. Also, releasing the s
by ignoramous 11y ago
Re: Semaphore:
Not dropping the connection might mean we might starve others incoming msgs of resources (which might be a good thing) [0]. Also, releasing the semaphore back into the pool in case of failures takes on a happy complication of having to deal with errors beyond one's control.
Re: Queue:
Wouldn't the queue involve locking lest two workers end up trying to work on the same request? To be completely concurrent, I guess one could use a lock-free data structure instead (or implement one on top of something like RocksDB)?
[0] http://ferd.ca/queues-don-t-fix-overload.html http://ferd.ca/queues-don-t-fix-overload.html
[0] http://engineering.voxer.com/2013/09/16/backpressure-in-nodejs/ http://engineering.voxer.com/2013/09/16/backpressure-in-node...
- peterwaller 11y agoSemaphore, I'm suggesting: Block on N-Semaphore, with timeout Do timeout upload Replace N-Semaphore If you don't always replace the semaphore, that's a bug. Queue: I'm just comparing to what the article does. It already has contention on a queue (the chan), it's just the chan-chan-Worker rather than chan-Job. In practice, go channels happily handle millions of messages contending to multiple workers just fine. Consider this test example where you aren't even actually burning any CPU to perform the work: package main func main() { q := make(chan int) for i := 0; i < 10; i++ { go func() { for x := range q { x = x * 10 } }() } for i := 0; i < 1000000; i++ { q <- i } } On my laptop, it runs in 0.333s single core, and it's slightly slower when you set GOMAXPROCS > 1. But not much slower, the total runtime goes to 0.4-0.5s or so. (Measured with go 1.4). As soon as you do any actual work with the messages you are passing around, the overhead of locking will be lost in the noise.