3 ms·

the situation is caused by a confluence of multiple issues. The biggest red-flag is that they chose to repurpose a flag! Why? Is it really difficult to add a n

by chii 3y ago

the situation is caused by a confluence of multiple issues. The biggest red-flag is that they chose to repurpose a flag! Why? Is it really difficult to add a new flag for a new feature? Even if the technician was careful not to let prod be out of sync, it is possible that the deployment isn't instantaneous, and that the old code could've ran when the repurposed flag was turned on.

atomicnumber3 3y ago
Some of the trickiness that comes from HFT and adjacent things like this is you can be working on tremendously powerful hardware but still be miserly about bits because every extra byte you have to cram into a packet is extra latency. The HFT firm I worked for would "re-use flags" in the sense that each packet literally had an 8-bit section called "flags" (and further down in the message, another 8 bit section called flags2 because of course) and each bit in there was a Boolean that could be on or off - a flag. So we weren't reusing flags as much as we were re-allocating what a high bit at that index in flags meant. We were very conscious of this kind of error though and we managed them like Scrooge counting his farthings.
nijave 3y ago
A couple solutions I've seen before - Config gets generated to a deploy and saved as <version>.json. Code downloads config file matching its own version or fails to start (this is a nice one since rollbacks becomes vary deterministic and code rollback is the same process as config rollback) - Deploy code first and verify it's updated and working correctly before changing feature flag config (this one is still prone to errors without automation) The first one can be done on Kubernetes using kustomize ConfigMap generator. The system I worked on used object storage for the config file and deploy tooling to generate it from a key value store at deploy time

3 ms·

the situation is caused by a confluence of multiple issues. The biggest red-flag is that they chose to repurpose a flag! Why? Is it really difficult to add a n

by chii 3y ago

atomicnumber3 3y ago
Some of the trickiness that comes from HFT and adjacent things like this is you can be working on tremendously powerful hardware but still be miserly about bits because every extra byte you have to cram into a packet is extra latency. The HFT firm I worked for would "re-use flags" in the sense that each packet literally had an 8-bit section called "flags" (and further down in the message, another 8 bit section called flags2 because of course) and each bit in there was a Boolean that could be on or off - a flag. So we weren't reusing flags as much as we were re-allocating what a high bit at that index in flags meant. We were very conscious of this kind of error though and we managed them like Scrooge counting his farthings.
nijave 3y ago
A couple solutions I've seen before - Config gets generated to a deploy and saved as <version>.json. Code downloads config file matching its own version or fails to start (this is a nice one since rollbacks becomes vary deterministic and code rollback is the same process as config rollback) - Deploy code first and verify it's updated and working correctly before changing feature flag config (this one is still prone to errors without automation) The first one can be done on Kubernetes using kustomize ConfigMap generator. The system I worked on used object storage for the config file and deploy tooling to generate it from a key value store at deploy time