4 ms·

What I wonder, following this story this week, is how the software quality controls at a place like Knight compare with those for life-critical systems like tho

by svdad 14y ago

What I wonder, following this story this week, is how the software quality controls at a place like Knight compare with those for life-critical systems like those in, e.g., aviation. On one hand, you'd think the QA in finance would be pretty solid, considering that the survival of the company could be at stake (witness Knight). On the other hand, I have a feeling that even there, people just don't take it that seriously. Would love to hear from anyone with more experience writing software for these industries.

izaidi 14y ago
Unlike high-frequency trading, aviation is highly regulated. In the United States the FAA specifies pretty detailed development standards for avionics software (e.g., DO-178B: http://en.wikipedia.org/wiki/DO-178B http://en.wikipedia.org/wiki/DO-178B). We're unlikely to see similarly strict requirements for financial software anytime soon.
- wpietri 14y ago
  Having talked with people who write life-critical code, the regulation isn't really what makes it safe. Safety comes from good engineering. The regulation just makes it much harder to bring an unsafe product to market, and makes it clearer who to blame when people die.
  - Retric 14y ago
    much harder to bring an unsafe product to market I would assume that's the point of the regulation in the first place? Nothing guarantees great software, but say requiring companies to pay for independent 3rd party testing adds significant barriers.
  - wglb 14y ago
    But don't you think the existence of regulation influences the culture?
    - toomuchtodo 14y ago
      Penalties influence the culture. As we all know, the first lesson in economics is that incentives matter. Sometimes, the right people aren't being incentivized to do the right thing.
- jimmytucson 14y ago
  I don't blame anybody who thinks it's not as regulated as it should be but you might be surprised at how extensive the current requirements are. Storing ~200 million daily transactions in order to fulfill arbitrarily complex reporting specifications from SEC, FINRA, internal audit (rogue trading detection, etc.), not to mention analytics for other trading systems... Oracle/IBM/whatever enterprise big data provider gets the contract must be sending one gigantic fruit basket around Christmas time. EDIT: Here's a little light reading for you http://finra.complinet.com/en/display/display_main.html?rbid=2403&element_id=4430 http://finra.complinet.com/en/display/display_main.html?rbid... (or you could just take sandpaper and rub it against your brain)
AncientPC 14y ago
I can't find the links at the moment, but some HFT guys have responded to relevant Slashdot articles over the years. From what I remember from those accounts, traders often like to tweak algorithms at the last second and there is little to no QA before changes gets pushed live. It's sort of a byproduct of the necessity for speed. Algorithms are quite complex, and even running a basic test suite that takes a few minutes may be deemed unnecessary.
thibaut_barrere 14y ago
I can tell first-hand that it depends on the places: some have fairly clean tests including fuzzy testing on trading automation, while other have, well, "fix it when it fails in production".
ewood 14y ago
I have experience in HFT, there are similarities to market making and I have plenty of colleagues who've worked in market making. Just like any company the culture is largely dependent on those in charge. Founders of these companies fall into three buckets - traders, techies, and mathematicians/physicists - and quality control will generally be a function of the founder mix. Mostly techies: strong software culture, unit testing, realtime monitoring. Mostly scientists: strong algos, software that works, monitoring varies. Mostly traders: risk v reward is the driver, software quality is unimportant unless it affects short-term profit, monitoring is unimportant - dollars and cents are sufficient. Obviously every firm's goals are driven by the goals of those in control. In the case of Knight they are largely a trader driven firm that has arrived late to the algo party. They were looking to get ahead by being one of the first market makers on the NYSE's new retail order matching system and probably cut some corners to get there. From a risk v reward perspective it probably looked like a good bet - with no major competitors customers would flood in and any bugs could be ironed out in live. Unfortunately the 'fat tail' (http://en.wikipedia.org/wiki/Fat_tail http://en.wikipedia.org/wiki/Fat_tail) struck and it may have sunk their company. For a closer look at what went wrong see http://www.zerohedge.com/news/what-happens-when-hft-algo-goes-totally-berserk-and-serves-knight-capital-bill http://www.zerohedge.com/news/what-happens-when-hft-algo-goe...
- jriley 14y ago
  Thanks for link. Given your experience, do you have any thoughts about a small circuit-breaker on every security that trips for, say any 3-standard deviation event? In a quick Google search, it seems circuit breakers do not kick in for first 15 minutes of trading: http://www.nytimes.com/2012/08/02/business/unusual-volume-roils-early-trading-in-some-stocks.html http://www.nytimes.com/2012/08/02/business/unusual-volume-ro...
  - ewood 14y ago
    In derivative markets such as futures there are predefined price and volume limits (in the jargon 'limit up' and 'limit down' - ie: max up and down movements.) These limits exist primarily to prevent market manipulation or cornering a market (buying the entire supply of a commodity) but have also been triggered by trading around events such as the Japan earthquake. Doing a quick search it seems that there are plans to trial these controls on equity markets (http://blogs.law.harvard.edu/corpgov/2012/06/13/limit-up-limit-down-plan-and-circuit-breakers-approved http://blogs.law.harvard.edu/corpgov/2012/06/13/limit-up-lim...) The problem here though was that while some stocks had dramatic price movements that might have triggered a limit control, more heavily traded stocks were able to absorb the additional volume and the price did not move significantly. Knight were not doing anything outside of normal bands, they were buying normal volumes of stocks close to the current market price and selling close to the current price. What they were doing was illogical in a profit sense because they were buying at a high price and selling at a lower price and thus immediately losing money. I think it would be very difficult for an exchange to trap this kind of problem. In all the issue of determining 'normal' trading is very difficult as markets tend to be much noisier than you might expect. The majority of trading occurs near the open and closes of major markets (Hong Kong, London, New York) or data releases (eg US Unemployment) so large spikes in volume and price are a regular occurrence. In equities this is even more difficult as smaller stocks will tend to be more volatile and profit reporting season increases this volatility even further. Markets are ruled by fear and greed, and falsely triggering a limit may cause larger issues than it solves.
njharman 14y ago
> considering that the survival of the company could be at stake It's at stake either way. QA is not risk free. Time spent in QA is opportunity cost lost. Many things are very time to market sensitive. One must balance "perfect" against "shipped". This is even more true in (electronic) financial industry.

by svdad 14y ago

izaidi 14y ago

Unlike high-frequency trading, aviation is highly regulated. In the United States the FAA specifies pretty detailed development standards for avionics software (e.g., DO-178B: http://en.wikipedia.org/wiki/DO-178B http://en.wikipedia.org/wiki/DO-178B). We're unlikely to see similarly strict requirements for financial software anytime soon.

wpietri 14y ago
Having talked with people who write life-critical code, the regulation isn't really what makes it safe. Safety comes from good engineering. The regulation just makes it much harder to bring an unsafe product to market, and makes it clearer who to blame when people die.
- Retric 14y ago
  much harder to bring an unsafe product to market I would assume that's the point of the regulation in the first place? Nothing guarantees great software, but say requiring companies to pay for independent 3rd party testing adds significant barriers.
- wglb 14y ago
  But don't you think the existence of regulation influences the culture?
  - toomuchtodo 14y ago
    Penalties influence the culture. As we all know, the first lesson in economics is that incentives matter. Sometimes, the right people aren't being incentivized to do the right thing.
jimmytucson 14y ago
I don't blame anybody who thinks it's not as regulated as it should be but you might be surprised at how extensive the current requirements are. Storing ~200 million daily transactions in order to fulfill arbitrarily complex reporting specifications from SEC, FINRA, internal audit (rogue trading detection, etc.), not to mention analytics for other trading systems... Oracle/IBM/whatever enterprise big data provider gets the contract must be sending one gigantic fruit basket around Christmas time. EDIT: Here's a little light reading for you http://finra.complinet.com/en/display/display_main.html?rbid=2403&element_id=4430 http://finra.complinet.com/en/display/display_main.html?rbid... (or you could just take sandpaper and rub it against your brain)

AncientPC 14y ago

I can't find the links at the moment, but some HFT guys have responded to relevant Slashdot articles over the years. From what I remember from those accounts, traders often like to tweak algorithms at the last second and there is little to no QA before changes gets pushed live. It's sort of a byproduct of the necessity for speed. Algorithms are quite complex, and even running a basic test suite that takes a few minutes may be deemed unnecessary.

thibaut_barrere 14y ago

I can tell first-hand that it depends on the places: some have fairly clean tests including fuzzy testing on trading automation, while other have, well, "fix it when it fails in production".

ewood 14y ago

I have experience in HFT, there are similarities to market making and I have plenty of colleagues who've worked in market making. Just like any company the culture is largely dependent on those in charge. Founders of these companies fall into three buckets - traders, techies, and mathematicians/physicists - and quality control will generally be a function of the founder mix. Mostly techies: strong software culture, unit testing, realtime monitoring. Mostly scientists: strong algos, software that works, monitoring varies. Mostly traders: risk v reward is the driver, software quality is unimportant unless it affects short-term profit, monitoring is unimportant - dollars and cents are sufficient. Obviously every firm's goals are driven by the goals of those in control. In the case of Knight they are largely a trader driven firm that has arrived late to the algo party. They were looking to get ahead by being one of the first market makers on the NYSE's new retail order matching system and probably cut some corners to get there. From a risk v reward perspective it probably looked like a good bet - with no major competitors customers would flood in and any bugs could be ironed out in live. Unfortunately the 'fat tail' (http://en.wikipedia.org/wiki/Fat_tail http://en.wikipedia.org/wiki/Fat_tail) struck and it may have sunk their company. For a closer look at what went wrong see http://www.zerohedge.com/news/what-happens-when-hft-algo-goes-totally-berserk-and-serves-knight-capital-bill http://www.zerohedge.com/news/what-happens-when-hft-algo-goe...

jriley 14y ago
Thanks for link. Given your experience, do you have any thoughts about a small circuit-breaker on every security that trips for, say any 3-standard deviation event? In a quick Google search, it seems circuit breakers do not kick in for first 15 minutes of trading: http://www.nytimes.com/2012/08/02/business/unusual-volume-roils-early-trading-in-some-stocks.html http://www.nytimes.com/2012/08/02/business/unusual-volume-ro...
- ewood 14y ago
  In derivative markets such as futures there are predefined price and volume limits (in the jargon 'limit up' and 'limit down' - ie: max up and down movements.) These limits exist primarily to prevent market manipulation or cornering a market (buying the entire supply of a commodity) but have also been triggered by trading around events such as the Japan earthquake. Doing a quick search it seems that there are plans to trial these controls on equity markets (http://blogs.law.harvard.edu/corpgov/2012/06/13/limit-up-limit-down-plan-and-circuit-breakers-approved http://blogs.law.harvard.edu/corpgov/2012/06/13/limit-up-lim...) The problem here though was that while some stocks had dramatic price movements that might have triggered a limit control, more heavily traded stocks were able to absorb the additional volume and the price did not move significantly. Knight were not doing anything outside of normal bands, they were buying normal volumes of stocks close to the current market price and selling close to the current price. What they were doing was illogical in a profit sense because they were buying at a high price and selling at a lower price and thus immediately losing money. I think it would be very difficult for an exchange to trap this kind of problem. In all the issue of determining 'normal' trading is very difficult as markets tend to be much noisier than you might expect. The majority of trading occurs near the open and closes of major markets (Hong Kong, London, New York) or data releases (eg US Unemployment) so large spikes in volume and price are a regular occurrence. In equities this is even more difficult as smaller stocks will tend to be more volatile and profit reporting season increases this volatility even further. Markets are ruled by fear and greed, and falsely triggering a limit may cause larger issues than it solves.

njharman 14y ago

> considering that the survival of the company could be at stake It's at stake either way. QA is not risk free. Time spent in QA is opportunity cost lost. Many things are very time to market sensitive. One must balance "perfect" against "shipped". This is even more true in (electronic) financial industry.