4 ms·

Interesting analysis. You can easily imagine each bit of sloppiness and oops happening in many shops. Even many instances of sloppiness combining for emergent

by neilv 3y ago

Interesting analysis. You can easily imagine each bit of sloppiness and oops happening in many shops. Even many instances of sloppiness combining for emergent worse effect isn't unusual. Much less common is for the company to be wiped out by it. > At 10:15, the kill switch was flipped, stopping the company’s trading operations for the day. By early afternoon, many of Knight Capital’s employees had already sent out resumes, Was the patient obviously dead in those first few hours, or had people written it off prematurely when they might've been called to help perform CPR? Maybe there's two problems here: insufficient culture of diligence, and insufficient culture of loyalty.

deleted 3y ago
[deleted]
randmeerkat 3y ago
> insufficient culture of loyalty Loyalty in tech..? Have you not seen the layoffs? Employees are just a line item in someone’s cost center.
- tekla 3y ago
  So many developers are fungible, it makes sense for everyone involved.
- golergka 3y ago
  Quants and HFT developers are one of the best compensated people on Earth. I don't think that someone who's being paid over a million dollars a year to sit in a cubicle and write code should use the same rhetoric as 19th century factory workers.
  - willdr 3y ago
    Then you don't understand the relationship between capital, labour and being an employee vs employer. Yes, they are well-compensated, but they can still have their lives turned upside down by bosses. Granted, their life getting turned upside down probably means less front-row Knicks tickets and not choosing between paying a water bill and a power bill.
    - deleted 3y ago
      [deleted]
  - nly 3y ago
    Best paid on Earth in the US. In other countries we're paid well but about average by ordinary Valley tech standards.
  - mhh__ 3y ago
    A lot of quants are objectively paid very well but not that well (at least pre layoffs) compared to some people who do comparatively little work in tech.
    - golergka 3y ago
      Good. Compensation should never be about the amount of work you do. It should be about your impact and market value.
- neilv 3y ago
  I'm not necessarily blaming labor. For example: a company that needs loyalty, but then does things like the layoffs we've seen recently (or earlier behavior consistent with that thinking), is creating insufficient culture of loyalty.
pclmulqdq 3y ago
I am the author, and everyone I knew who was at Knight at the time had no power to fix anything. After all the trades happened, the quants and tech people couldn't really do anything to unwind them. That was up to the "business" side of the org, who still had no power because they were essentially left looking for a deal to rescue a bankrupt company. Those people actually knew that this was a company-destroying event, and apparently it was a late night for them at work and they were also quietly talking to recruiters. At least one manager actually gave his whole team the afternoon off while senior management worked this out, anticipating that there would be no Knight by the end of the week.
- fl7305 3y ago
  In the movie WarGames, someone suggested "just turn the damn power off", and got a good answer on why that was "a bad idea"™. I'm kinda wondering the same thing here, and can't think of the reason why not?
  - yellowstuff 3y ago
    That's what always struck me about this story. I like to think that if I was in the room we would've turned the system off at 9:31 when we knew it was working abnormally but didn't know why. Instead they let it run for over an hour while they tried to QA it. I've heard that Knight had no kill switch to stop all systems from trading, which seems like by far the biggest mistake they made.
    - pclmulqdq 3y ago
      As far as I know, Knight did not have a "clean" kill switch, but they did have the ability to shut everything down. If they killed trading, that meant killing some processes that would screw up their accounting, which would mean losing the rest of the trading day. Firms I worked at after the fact tried to make sure the kill switch was non-disruptive, and actually did push it once or twice.
  - KMag 3y ago
    The power switches were in locked cages inside colos miles from the nearest Knight employee. The sysadmins could probably power off the boxes remotely, but the devs probably didn't have that access. Yes, a "big red button" kill switch was the right answer, but they didn't have that, and it took time to work through several layers of corporate bureaucracy while losing $150,000 per second, time they didn't have. Anyone with experience in the industry knows that, especially the day of/following a software release, "if in doubt, kill and roll back". Being out of the market for even a few hours is fairly easily to come back from, even in places where you are legally required to be in the market for a given percentage of the time in order to qualify for transaction tax breaks on your market making trades. That's why on that day my manger called me after hours to come in early the next morning and test our "big red button" kill switch before the market open.
Kranar 3y ago
Loyalty had nothing to do with this. The company lost $400 million dollars in about 45 minutes. That's 9 million dollars a minute or 150 thousand dollars a second. I work as a quant and remember that morning vividly. It was as if trucks of free money were falling from the sky to the point that many of us were skeptical that these were actual trades; we were convinced that this was some kind of bug at the exchange and these trades would get broken. Mistakes like this do happen, in fact it's not thaaaat uncommon. What made this situation so unusual is that it just kept going and going.
mhh__ 3y ago
> Maybe there's two problems here: insufficient culture of diligence I know nothing about Knight but I will say that traditional finance often has no ability to think about systems particularly well so the local optimum is usually a rickety shack that has a lot of smart people working on it, and some less smart people making sure problems are being seen to be fixed, but the overall architecture can be extremely brittle or non-existent. It's not like (good) tech companies.

by neilv 3y ago

deleted 3y ago

[deleted]

randmeerkat 3y ago

> insufficient culture of loyalty Loyalty in tech..? Have you not seen the layoffs? Employees are just a line item in someone’s cost center.

tekla 3y ago
So many developers are fungible, it makes sense for everyone involved.
golergka 3y ago
Quants and HFT developers are one of the best compensated people on Earth. I don't think that someone who's being paid over a million dollars a year to sit in a cubicle and write code should use the same rhetoric as 19th century factory workers.
- willdr 3y ago
  Then you don't understand the relationship between capital, labour and being an employee vs employer. Yes, they are well-compensated, but they can still have their lives turned upside down by bosses. Granted, their life getting turned upside down probably means less front-row Knicks tickets and not choosing between paying a water bill and a power bill.
  - deleted 3y ago
    [deleted]
- nly 3y ago
  Best paid on Earth in the US. In other countries we're paid well but about average by ordinary Valley tech standards.
- mhh__ 3y ago
  A lot of quants are objectively paid very well but not that well (at least pre layoffs) compared to some people who do comparatively little work in tech.
  - golergka 3y ago
    Good. Compensation should never be about the amount of work you do. It should be about your impact and market value.
neilv 3y ago
I'm not necessarily blaming labor. For example: a company that needs loyalty, but then does things like the layoffs we've seen recently (or earlier behavior consistent with that thinking), is creating insufficient culture of loyalty.

pclmulqdq 3y ago

I am the author, and everyone I knew who was at Knight at the time had no power to fix anything. After all the trades happened, the quants and tech people couldn't really do anything to unwind them. That was up to the "business" side of the org, who still had no power because they were essentially left looking for a deal to rescue a bankrupt company. Those people actually knew that this was a company-destroying event, and apparently it was a late night for them at work and they were also quietly talking to recruiters. At least one manager actually gave his whole team the afternoon off while senior management worked this out, anticipating that there would be no Knight by the end of the week.

fl7305 3y ago
In the movie WarGames, someone suggested "just turn the damn power off", and got a good answer on why that was "a bad idea"™. I'm kinda wondering the same thing here, and can't think of the reason why not?
- yellowstuff 3y ago
  That's what always struck me about this story. I like to think that if I was in the room we would've turned the system off at 9:31 when we knew it was working abnormally but didn't know why. Instead they let it run for over an hour while they tried to QA it. I've heard that Knight had no kill switch to stop all systems from trading, which seems like by far the biggest mistake they made.
  - pclmulqdq 3y ago
    As far as I know, Knight did not have a "clean" kill switch, but they did have the ability to shut everything down. If they killed trading, that meant killing some processes that would screw up their accounting, which would mean losing the rest of the trading day. Firms I worked at after the fact tried to make sure the kill switch was non-disruptive, and actually did push it once or twice.
- KMag 3y ago
  The power switches were in locked cages inside colos miles from the nearest Knight employee. The sysadmins could probably power off the boxes remotely, but the devs probably didn't have that access. Yes, a "big red button" kill switch was the right answer, but they didn't have that, and it took time to work through several layers of corporate bureaucracy while losing $150,000 per second, time they didn't have. Anyone with experience in the industry knows that, especially the day of/following a software release, "if in doubt, kill and roll back". Being out of the market for even a few hours is fairly easily to come back from, even in places where you are legally required to be in the market for a given percentage of the time in order to qualify for transaction tax breaks on your market making trades. That's why on that day my manger called me after hours to come in early the next morning and test our "big red button" kill switch before the market open.

Kranar 3y ago

Loyalty had nothing to do with this. The company lost $400 million dollars in about 45 minutes. That's 9 million dollars a minute or 150 thousand dollars a second. I work as a quant and remember that morning vividly. It was as if trucks of free money were falling from the sky to the point that many of us were skeptical that these were actual trades; we were convinced that this was some kind of bug at the exchange and these trades would get broken. Mistakes like this do happen, in fact it's not thaaaat uncommon. What made this situation so unusual is that it just kept going and going.

mhh__ 3y ago

> Maybe there's two problems here: insufficient culture of diligence I know nothing about Knight but I will say that traditional finance often has no ability to think about systems particularly well so the local optimum is usually a rickety shack that has a lot of smart people working on it, and some less smart people making sure problems are being seen to be fixed, but the overall architecture can be extremely brittle or non-existent. It's not like (good) tech companies.