2 ms·
Wasted an opportunity to implement code that would detect and handle poison-pill messages. Those will happen in any system where queue is involved and there alw
by watt 6y ago
Wasted an opportunity to implement code that would detect and handle poison-pill messages. Those will happen in any system where queue is involved and there always needs to be an escape hatch to get rid of them. Deleting the queue is too extreme.
- miahi 6y agoDeleting the queues is an operational decision that I made to be able to put the system back online after the network device was replaced (the important part was the uptime/SLA). From a quick analysis of the logs the percentage of bad messages was ~90% (there was a ~50% chance that the original "touched" bit was 0 so no change was done, but the messages had multiple "touched" bits at fixed intervals). There was an escape hatch, but the conditions to hit it were a bit complex. Implementing new message filtering of this kind at 2AM while the system was down was not feasible.