3 ms·
With pipes between threads, the OS is able to suspend the reading thread when the buffer is empty, or suspend the writing thread when the buffer is full, becaus
by kristov 5y ago
With pipes between threads, the OS is able to suspend the reading thread when the buffer is empty, or suspend the writing thread when the buffer is full, because the OS "owns" the buffer and the threads. With distributed systems connected via Kafka for example, there is no "process" overseeing both ends that can suspend or resume hollistically, so its difficult for producers to know and respond to back pressure.
- pjc50 5y agoIsn't the buffer then the TCP one, which has its own backpressure system?
- singron 5y agoTCP will only slow down the writer if the network or the Kafka broker can't keep up. It the Kafka consumer can't keep up, their backlog will grow unhindered until Kafka runs out of disk space. I wish there was an easier built in way to do it, but a producer could read the backlog length of it's consumers and try to throttle itself.