5 ms·

The power switches were in locked cages inside colos miles from the nearest Knight employee. The sysadmins could probably power off the boxes remotely, but the

by KMag 3y ago

The power switches were in locked cages inside colos miles from the nearest Knight employee. The sysadmins could probably power off the boxes remotely, but the devs probably didn't have that access. Yes, a "big red button" kill switch was the right answer, but they didn't have that, and it took time to work through several layers of corporate bureaucracy while losing $150,000 per second, time they didn't have. Anyone with experience in the industry knows that, especially the day of/following a software release, "if in doubt, kill and roll back". Being out of the market for even a few hours is fairly easily to come back from, even in places where you are legally required to be in the market for a given percentage of the time in order to qualify for transaction tax breaks on your market making trades. That's why on that day my manger called me after hours to come in early the next morning and test our "big red button" kill switch before the market open.

JumpCrisscross 3y ago
> Being out of the market for even a few hours is fairly easily to come back from Not if you’re dynamically hedging an options book.
- KMag 3y ago
  Fair enough. At Goldman, I don't recall our options and equities trading systems simultaneously having outages, so I think we could always shed risk by reducing our options exposure if the auto-hedger was unable to delta-hedge in the equities market. I'm not actually sure if the relative independence of the options and equity execution systems was intentional. I did some work with connecting the options auto-hedger to the equity execution system, and certainly failures on the delta-one side prevented increasing exposure on the options side. "How long can we be out of the equities market and still be certain of meeting our options market-making obligations with the Hong Kong Exchange?" did come up a couple of times. Depending on exactly where the outage was, there was potentially also the option of manually hedging the options book like the bad-old days. (Execution engines failed, but order management system and exchange connectivity still intact would be one such scenario.)
fl7305 3y ago
Thanks, good answer. I've been in other types of very stressful situations. Simple things like getting hold of the right person or the right tool can be astonishingly time consuming in a tight spot.
heelix 3y ago
Many years back, we had just moved our new AS400 to a freshly build server room as one of those home shopping networks expanded into the rest of the building. It had a 'big red button' and one of the veeps touring the pre-power up event really wanted to push the button. He did. What we quickly discovered is the power line was connected to the entire call center terminal too. Hit the button... and then all of our pagers started going off shortly there after. The next discovery was the fire extinguisher head above the room was not installed right when they turned on the water. Brand new machine soaked in the brackish waster.

by KMag 3y ago

JumpCrisscross 3y ago

> Being out of the market for even a few hours is fairly easily to come back from Not if you’re dynamically hedging an options book.

KMag 3y ago
Fair enough. At Goldman, I don't recall our options and equities trading systems simultaneously having outages, so I think we could always shed risk by reducing our options exposure if the auto-hedger was unable to delta-hedge in the equities market. I'm not actually sure if the relative independence of the options and equity execution systems was intentional. I did some work with connecting the options auto-hedger to the equity execution system, and certainly failures on the delta-one side prevented increasing exposure on the options side. "How long can we be out of the equities market and still be certain of meeting our options market-making obligations with the Hong Kong Exchange?" did come up a couple of times. Depending on exactly where the outage was, there was potentially also the option of manually hedging the options book like the bad-old days. (Execution engines failed, but order management system and exchange connectivity still intact would be one such scenario.)

fl7305 3y ago

Thanks, good answer. I've been in other types of very stressful situations. Simple things like getting hold of the right person or the right tool can be astonishingly time consuming in a tight spot.

heelix 3y ago

Many years back, we had just moved our new AS400 to a freshly build server room as one of those home shopping networks expanded into the rest of the building. It had a 'big red button' and one of the veeps touring the pre-power up event really wanted to push the button. He did. What we quickly discovered is the power line was connected to the entire call center terminal too. Hit the button... and then all of our pagers started going off shortly there after. The next discovery was the fire extinguisher head above the room was not installed right when they turned on the water. Brand new machine soaked in the brackish waster.