8 ms·

High-Speed Trading Firm Deleted Some Code by Accident

75 points by fahimulhaq 11y ago

keyle 11y ago
Doesn't matter how much money is involved or the quality of a team, someone somewhere will always do an oopsie. Maybe that particular 'code' should have had a little comment such as // used by ISO stuff do not delete Commenting tricky / non-obvious code is well recommended.
- deckar01 11y ago
  Some unit tests would be nice. Maybe even a special folder full of tests that know when laws have been broken.
  - azinman2 11y ago
    Probably exists now, and not before. Writing this software would be very scary for me -- I'm confident in my work for the most part but I'm only human. To then let a "for-loop" go wild making thousands of trades per second on the entire market with real money against other high frequency traders no less.. man... you gotta be really really really really sure of your code!
    - mathgenius 11y ago
      Yeah, well, fkups happen, you just gotta make sure you are making more money than you lose in those incidents.
      - smoothgrammer 11y ago
        Or you just write a test that checks to make sure you don't screw up.
        mathgenius 11y ago
        If you're designing a new airliner then yeah, you can't screw up, and you end up taking years to deploy. In the HFT world, you may have only days, or sometimes hours, to deploy. Yes you still write tests, and a bazillion safety checks when you deploy, but if you manage your risk down to zero then this is definitely not the most profitable way to go.
    - smoothgrammer 11y ago
      It is not about confidence. It is about ownership and being a good engineer. A good engineer would have cared about the code base and ensured that a test would fail, a use case would be linked to that failure, and someone would see that they have now violated a regulation. Then if the regulation changed, someone would grep through the use cases, and see what behavior is occuring/ how to modify the code base to cope with the change in regulations (this occurs a lot). Of course most finance firms don't do this and actually violate regulations all the time. There is literally a gold mine for the SEC/FINRA to go after, but the government is fairly incompetent at mining through the data and holding companies accountable. The reason being that they don't pay enough to hire the talent that would uncover the violations.
      - vitalyd 11y ago
        I'd be surprised if they didn't make postmortem changes to prevent such incidents in the future. Your posts in this thread imply that no mistake should ever happen, but let's be real - mistakes will always happen, the key is to do proper postmortem analysis and learn from them (and obviously prevent recurrences).
        smoothgrammer 11y ago
        Some mistakes, yes. But this was a case of such a basic and clear cut use case that should have been tested. My understanding is that companies such as this rely purely on manual testing. Thus it is easy for things to slip through the cracks. Usually they never get hit by penalties or get caught so the pressure to do better is not there.
      - tbrownaw 11y ago
        Real engineers are like true Scotsmen, because you can never have enough process. In this case, of course tests really do prove the absence of bugs rather than just (not) proving their presence. I really don't like the whole "unit tests lets you not worry about breaking things" view. That's not confidence, it's overconfidence. And it doesn't even mostly work for things where "correct" is more fuzzy than "eventually produces this exact output".
  - x0054 11y ago
    Or how about at least some code documentation. In this case the programer deleted the code because he didn't understand why it was there. Well a simple "# Needed to Comply with regulation X" would probably have prevented this problem.
    - osi 11y ago
      There's a specific incantation required to send the routed ISO orders via BATS as mentioned in the article. I can see how a refactoring might accidentally change that. With that said, libraries that send orders to exchanges are required by the exchanges to be certified against their expected behaviors. If you want to send routed ISO orders, you have to test that with them. When making changes to said libraries, it can often be good to re-certify to ensure things are still working as expected.
jwatte 11y ago
Acceptance testing fail! When changing code, and especially code that has consequences, doing so without tests is a bad idea. Doing so to protocol code without conformance tests is doubly bad.
- smoothgrammer 11y ago
  Sadly, this is the norm in finance. You would be shocked at the shittastic state of the companies that dominate in this industry. They are literally a house of cards waiting for a strong gust of wind.
tbrownaw 11y ago
...I'm missing something here. How do they not know how much they've asked to trade and not heard back on yet? Wouldn't they have to track that, to not try to trade the same exact shares the same exact way a bunch of times? Is their local system maybe just more synchronous than I'm assuming would be possible?
- deleted 11y ago
  [deleted]
zeeshanm 11y ago
Another reason code needs to be idiomatic.
Tinyyy 11y ago
I don't know much about stocks trading, can someone ELI5? Thanks.
- smoothgrammer 11y ago
  Basically imagine you want to buy apples from 3 different fruit stands in different parts of a city. Fruit stand 1 is the cheapest, fruit stand 2 is the 2nd cheapest, and fruit stand 3 is the third cheapest. You want to buy all the apples out there. The rule is that you have to go to the cheapest fruit stand first, and exhaust their supply before moving onto the next cheapest fruit stand. If you want to go to ALL the fruit stands at the same time you need to tell them you are an 'OK' guy and are really buying fruit everywhere. Well, someone deleted code and ignored one of the fruit stands. This violated a rule and cost the fruit buyer a lot of money in penalties.
  - bdonlan 11y ago
    Why does the SEC care that you bought shares at a price higher than the cheapest? How does that hurt anyone other than yourself?
    - deleted 11y ago
      [deleted]
    - waqf 11y ago
      And relatedly, what is the point in the existence of multiple markets, if not that you can choose which market to trade on for your own technical–logistic reasons?
      - mrchicity 11y ago
        You can, but only at the same price. It's a weird system, especially when markets are located different places, so from each trader or exchange's vantage point, things are in different states. Working around this friction led to an explosion of special order types that some people argue are unfair or overly complex. In Europe and Asia, there are competitive exchanges, but no rule to force people to route there. There are some guidelines for brokers with customer orders to have a duty of "best execution", but for proprietary traders or professionals, they can do whatever they feel is ideal. And guess what, there are very few "trade throughs" or cases where markets invert with each other despite lacking such a rule. People usually act in their economic best interest without a law mandating they do so. If it happens, it's an arbitrage that someone will eliminate very quickly.
smoothgrammer 11y ago
In finance you'll find low accountability for developers. Basically finance is where great developers go to get paid a lot to become horrible developers. Very few companies in finance are pushing the bar on quality code since they culturally cannot accept the cost of that. It doesn't make sense until it is too late and by then their code base is too entrenched. I've seen this at countless finance companies, and from interviewing people across the entire spectrum of finance. It is a rare corner of the industry where you'll find quality engineering. Usually everyone in a finance company thinks they are amazing engineers, and are just plain wrong. My advice: work smarter, not harder.
- mdpopescu 11y ago
  I can confirm this. I just wish I had known about it before I accepted my current contract...
  - smoothgrammer 11y ago
    Get out while you can.
- tbrownaw 11y ago
  But is it enough to put a company that doesn't do that at a competitive advantage? Is this really just company culture and assumptions, or are there actual market forces behind it?
  - smoothgrammer 11y ago
    It is a mixed bag. In finance technology is seen as a cost center, not a profit center. They want the minimum needed to ship. Early on this results in a lot of tech debt. Later on, fixing the system is too hard and thus they need to hire super smart people to figure out the mess they are in for any code change. Enforcement actions are INCREDIBLY rare. The penalties are shockingly low. Thus the cost of failure to comply is low, and the businesses are not prioritizing compliance.
    - 11y ago
x0054 11y ago
Every time I read stories about high speed trading, I have to wonder what is actually the point of high speed trading to the society. The point of a free and fast stock market is to provide monetary liquidity and stabilize the economy by shortening the feedback loop. But something tells me that our society and economy derives zero benefit from nanosecond resolution trading vs. trading with say 1 minute resolution. Meanwhile, the market is exposed to wild swings which might be smoothed over or even prevented if markets worked a little slower.
- Houshalter 11y ago
  I don't think high frequency trading increases the probability of dangerous swings. I don't know if they add any value, but I doubt they cause all the problems people blame on them. When they do start doing something stupid, then they will lose money and be punished and other traders will profit from them.
  - rasz_pl 11y ago
    the premise is they might add value in some circumstances, but in the mean time they extract value 24/7 >When they do start doing something stupid, then they will lose money and be punished hehe no, trades will be reversed, people that "maliciously took advantage of poor algo error" will be punished instead
- kylebrown 11y ago
  > But something tells me that our society and economy derives zero benefit from nanosecond resolution trading vs. trading with say 1 minute resolution. Not only does nanosecond resolution provide zero benefit, models indicate that it actually harms liquidity. To be specific, serial order processing in continuous-time is more efficient in "time-space", but less efficient in "volume-space". The better mechanism is batch order processing in discrete-time (i.e. process all arriving orders simultaneously in batch every 100 milliseconds, rather than one-by-one every nanosecond): http://faculty.chicagobooth.edu/eric.budish/research/HFT-FrequentBatchAuctions-Slides.pdf http://faculty.chicagobooth.edu/eric.budish/research/HFT-Fre...
herbig 11y ago
The best part of this article is the feature graphic, that of a castle with the caption: THIS IS A TOWER. PHOTOGRAPHER: DAMIEN MEYER/AFP/GETTY IMAGES The article mentions "Tower Research". Is the image/caption a joke, auto-generated by content, or just plain laziness?
- yawaramin 11y ago
  Also, 'La Tour' means 'The Tower'.
- davmre 11y ago
  It's a joke, consistent with Matt Levine's writing style. See for example the image and caption for his article on Panda Express: http://www.bloombergview.com/articles/2015-09-24/panda-express-s-life-coach-might-have-overshared-a-bit http://www.bloombergview.com/articles/2015-09-24/panda-expre...
JimDash2145 11y ago
Who doesn't notice a huge deletion in one changeset? This seems to be a convenient excuse. The SEC report. http://www.sec.gov/litigation/admin/2015/34-76029.pdf http://www.sec.gov/litigation/admin/2015/34-76029.pdf
Sander_Marechal 11y ago
Can someone explain why the SEC rules insist you must buy from the exchange with the lowest price first? It appears to me to be a rule just to keep the small exchanges alive.
- osullivj 11y ago
  To stop brokers screwing their clients by routing orders to their mates.
- kylebrown 11y ago
  Otherwise a broker would just route mom n' pop's orders to the exchange that pays the biggest bribe to the broker (and screw mom n' pop).
  - mrchicity 11y ago
    Yes but for proprietary traders using their own capital, it doesn't make a lot of sense to force them to trade in a particular way since executing sub-optimally can only impact their own performance. There are legitimate reasons not to trade the lowest price market (maybe you believe the quote is slow, want to trade bigger size all at once, don't want to risk signaling to the market): http://www.hudson-trading.com/static/files/RegNMShudsonriver081304.pdf http://www.hudson-trading.com/static/files/RegNMShudsonriver... And retail brokers already do what you suggest. All their marketable flow gets sold to off-exchange market makers, while their limit order flow gets routed to exchanges that pay the highest liquidity rebates, Reg NMS just mandates what price it can trade at: http://news.indiana.edu/releases/iu/2014/02/study-of-potential-conflict-of-interest-in-routing-limit-orders%20.shtml http://news.indiana.edu/releases/iu/2014/02/study-of-potenti...
pjc50 11y ago
Move fast. Break things. Pay millions of dollars in SEC fines.
fahimulhaq 11y ago
Given hindsight, it's always easier to ask for more tests or code comments. I guess the rule of thumb is that if the code is there, it most probably is there for a reason. When something like this happens, it's always a series of oversights. 1. Someone forgot to add comments or test cases while writing the code (or perhaps he wrote it but people are not running the test cases before commits). 2. Someone else thought that the code is dead and deleted it (and skipped the test cases if there were some covering that particular scenario). 3. Either the person deleting the code didn't wait for the code review or code reviewers missed that as well. (If code reviews are non-existent, it's a disaster waiting to happen). So in the hindsight, write comments, write test cases, run test cases and do code reviews carefully.
fahimulhaq 11y ago
On a related note, Knight Capital lost $440 million due to a software bug a few years ago. http://www.bloomberg.com/news/articles/2012-08-02/knight-has-all-hands-on-deck-after-440-million-bug http://www.bloomberg.com/news/articles/2012-08-02/knight-has...
- HolyLampshade 11y ago
  Blew up the whole firm actually, and was more closely related to change management policies and systems monitoring. The SEC order there is an interesting read for anyone involved in large scale transactional systems: https://www.sec.gov/litigation/admin/2013/34-70694.pdf https://www.sec.gov/litigation/admin/2013/34-70694.pdf
osullivj 11y ago
A change like this making it through to production is indicative of a test failure, either automated or human. I have had the luxury of substantial human test teams on trading systems before 2008, and it was great. They wouldn't let something like this through...

75 points by fahimulhaq 11y ago

keyle 11y ago

Doesn't matter how much money is involved or the quality of a team, someone somewhere will always do an oopsie. Maybe that particular 'code' should have had a little comment such as // used by ISO stuff do not delete Commenting tricky / non-obvious code is well recommended.

deckar01 11y ago
Some unit tests would be nice. Maybe even a special folder full of tests that know when laws have been broken.
- azinman2 11y ago
  Probably exists now, and not before. Writing this software would be very scary for me -- I'm confident in my work for the most part but I'm only human. To then let a "for-loop" go wild making thousands of trades per second on the entire market with real money against other high frequency traders no less.. man... you gotta be really really really really sure of your code!
  - mathgenius 11y ago
    Yeah, well, fkups happen, you just gotta make sure you are making more money than you lose in those incidents.
    - smoothgrammer 11y ago
      Or you just write a test that checks to make sure you don't screw up.
      - mathgenius 11y ago
        If you're designing a new airliner then yeah, you can't screw up, and you end up taking years to deploy. In the HFT world, you may have only days, or sometimes hours, to deploy. Yes you still write tests, and a bazillion safety checks when you deploy, but if you manage your risk down to zero then this is definitely not the most profitable way to go.
  - smoothgrammer 11y ago
    It is not about confidence. It is about ownership and being a good engineer. A good engineer would have cared about the code base and ensured that a test would fail, a use case would be linked to that failure, and someone would see that they have now violated a regulation. Then if the regulation changed, someone would grep through the use cases, and see what behavior is occuring/ how to modify the code base to cope with the change in regulations (this occurs a lot). Of course most finance firms don't do this and actually violate regulations all the time. There is literally a gold mine for the SEC/FINRA to go after, but the government is fairly incompetent at mining through the data and holding companies accountable. The reason being that they don't pay enough to hire the talent that would uncover the violations.
    - vitalyd 11y ago
      I'd be surprised if they didn't make postmortem changes to prevent such incidents in the future. Your posts in this thread imply that no mistake should ever happen, but let's be real - mistakes will always happen, the key is to do proper postmortem analysis and learn from them (and obviously prevent recurrences).
      - smoothgrammer 11y ago
        Some mistakes, yes. But this was a case of such a basic and clear cut use case that should have been tested. My understanding is that companies such as this rely purely on manual testing. Thus it is easy for things to slip through the cracks. Usually they never get hit by penalties or get caught so the pressure to do better is not there.
    - tbrownaw 11y ago
      Real engineers are like true Scotsmen, because you can never have enough process. In this case, of course tests really do prove the absence of bugs rather than just (not) proving their presence. I really don't like the whole "unit tests lets you not worry about breaking things" view. That's not confidence, it's overconfidence. And it doesn't even mostly work for things where "correct" is more fuzzy than "eventually produces this exact output".
- x0054 11y ago
  Or how about at least some code documentation. In this case the programer deleted the code because he didn't understand why it was there. Well a simple "# Needed to Comply with regulation X" would probably have prevented this problem.
  - osi 11y ago
    There's a specific incantation required to send the routed ISO orders via BATS as mentioned in the article. I can see how a refactoring might accidentally change that. With that said, libraries that send orders to exchanges are required by the exchanges to be certified against their expected behaviors. If you want to send routed ISO orders, you have to test that with them. When making changes to said libraries, it can often be good to re-certify to ensure things are still working as expected.

jwatte 11y ago

Acceptance testing fail! When changing code, and especially code that has consequences, doing so without tests is a bad idea. Doing so to protocol code without conformance tests is doubly bad.

smoothgrammer 11y ago
Sadly, this is the norm in finance. You would be shocked at the shittastic state of the companies that dominate in this industry. They are literally a house of cards waiting for a strong gust of wind.

tbrownaw 11y ago

...I'm missing something here. How do they not know how much they've asked to trade and not heard back on yet? Wouldn't they have to track that, to not try to trade the same exact shares the same exact way a bunch of times? Is their local system maybe just more synchronous than I'm assuming would be possible?

deleted 11y ago
[deleted]

zeeshanm 11y ago

Another reason code needs to be idiomatic.

Tinyyy 11y ago

I don't know much about stocks trading, can someone ELI5? Thanks.

smoothgrammer 11y ago
Basically imagine you want to buy apples from 3 different fruit stands in different parts of a city. Fruit stand 1 is the cheapest, fruit stand 2 is the 2nd cheapest, and fruit stand 3 is the third cheapest. You want to buy all the apples out there. The rule is that you have to go to the cheapest fruit stand first, and exhaust their supply before moving onto the next cheapest fruit stand. If you want to go to ALL the fruit stands at the same time you need to tell them you are an 'OK' guy and are really buying fruit everywhere. Well, someone deleted code and ignored one of the fruit stands. This violated a rule and cost the fruit buyer a lot of money in penalties.
- bdonlan 11y ago
  Why does the SEC care that you bought shares at a price higher than the cheapest? How does that hurt anyone other than yourself?
  - deleted 11y ago
    [deleted]
  - waqf 11y ago
    And relatedly, what is the point in the existence of multiple markets, if not that you can choose which market to trade on for your own technical–logistic reasons?
    - mrchicity 11y ago
      You can, but only at the same price. It's a weird system, especially when markets are located different places, so from each trader or exchange's vantage point, things are in different states. Working around this friction led to an explosion of special order types that some people argue are unfair or overly complex. In Europe and Asia, there are competitive exchanges, but no rule to force people to route there. There are some guidelines for brokers with customer orders to have a duty of "best execution", but for proprietary traders or professionals, they can do whatever they feel is ideal. And guess what, there are very few "trade throughs" or cases where markets invert with each other despite lacking such a rule. People usually act in their economic best interest without a law mandating they do so. If it happens, it's an arbitrage that someone will eliminate very quickly.

smoothgrammer 11y ago

In finance you'll find low accountability for developers. Basically finance is where great developers go to get paid a lot to become horrible developers. Very few companies in finance are pushing the bar on quality code since they culturally cannot accept the cost of that. It doesn't make sense until it is too late and by then their code base is too entrenched. I've seen this at countless finance companies, and from interviewing people across the entire spectrum of finance. It is a rare corner of the industry where you'll find quality engineering. Usually everyone in a finance company thinks they are amazing engineers, and are just plain wrong. My advice: work smarter, not harder.

mdpopescu 11y ago
I can confirm this. I just wish I had known about it before I accepted my current contract...
- smoothgrammer 11y ago
  Get out while you can.
tbrownaw 11y ago
But is it enough to put a company that doesn't do that at a competitive advantage? Is this really just company culture and assumptions, or are there actual market forces behind it?
- smoothgrammer 11y ago
  It is a mixed bag. In finance technology is seen as a cost center, not a profit center. They want the minimum needed to ship. Early on this results in a lot of tech debt. Later on, fixing the system is too hard and thus they need to hire super smart people to figure out the mess they are in for any code change. Enforcement actions are INCREDIBLY rare. The penalties are shockingly low. Thus the cost of failure to comply is low, and the businesses are not prioritizing compliance.
  - 11y ago

x0054 11y ago

Every time I read stories about high speed trading, I have to wonder what is actually the point of high speed trading to the society. The point of a free and fast stock market is to provide monetary liquidity and stabilize the economy by shortening the feedback loop. But something tells me that our society and economy derives zero benefit from nanosecond resolution trading vs. trading with say 1 minute resolution. Meanwhile, the market is exposed to wild swings which might be smoothed over or even prevented if markets worked a little slower.

Houshalter 11y ago
I don't think high frequency trading increases the probability of dangerous swings. I don't know if they add any value, but I doubt they cause all the problems people blame on them. When they do start doing something stupid, then they will lose money and be punished and other traders will profit from them.
- rasz_pl 11y ago
  the premise is they might add value in some circumstances, but in the mean time they extract value 24/7 >When they do start doing something stupid, then they will lose money and be punished hehe no, trades will be reversed, people that "maliciously took advantage of poor algo error" will be punished instead
kylebrown 11y ago
> But something tells me that our society and economy derives zero benefit from nanosecond resolution trading vs. trading with say 1 minute resolution. Not only does nanosecond resolution provide zero benefit, models indicate that it actually harms liquidity. To be specific, serial order processing in continuous-time is more efficient in "time-space", but less efficient in "volume-space". The better mechanism is batch order processing in discrete-time (i.e. process all arriving orders simultaneously in batch every 100 milliseconds, rather than one-by-one every nanosecond): http://faculty.chicagobooth.edu/eric.budish/research/HFT-FrequentBatchAuctions-Slides.pdf http://faculty.chicagobooth.edu/eric.budish/research/HFT-Fre...

herbig 11y ago

The best part of this article is the feature graphic, that of a castle with the caption: THIS IS A TOWER. PHOTOGRAPHER: DAMIEN MEYER/AFP/GETTY IMAGES The article mentions "Tower Research". Is the image/caption a joke, auto-generated by content, or just plain laziness?

yawaramin 11y ago
Also, 'La Tour' means 'The Tower'.
davmre 11y ago
It's a joke, consistent with Matt Levine's writing style. See for example the image and caption for his article on Panda Express: http://www.bloombergview.com/articles/2015-09-24/panda-express-s-life-coach-might-have-overshared-a-bit http://www.bloombergview.com/articles/2015-09-24/panda-expre...

JimDash2145 11y ago

Who doesn't notice a huge deletion in one changeset? This seems to be a convenient excuse. The SEC report. http://www.sec.gov/litigation/admin/2015/34-76029.pdf http://www.sec.gov/litigation/admin/2015/34-76029.pdf

Sander_Marechal 11y ago

Can someone explain why the SEC rules insist you must buy from the exchange with the lowest price first? It appears to me to be a rule just to keep the small exchanges alive.

osullivj 11y ago
To stop brokers screwing their clients by routing orders to their mates.
kylebrown 11y ago
Otherwise a broker would just route mom n' pop's orders to the exchange that pays the biggest bribe to the broker (and screw mom n' pop).
- mrchicity 11y ago
  Yes but for proprietary traders using their own capital, it doesn't make a lot of sense to force them to trade in a particular way since executing sub-optimally can only impact their own performance. There are legitimate reasons not to trade the lowest price market (maybe you believe the quote is slow, want to trade bigger size all at once, don't want to risk signaling to the market): http://www.hudson-trading.com/static/files/RegNMShudsonriver081304.pdf http://www.hudson-trading.com/static/files/RegNMShudsonriver... And retail brokers already do what you suggest. All their marketable flow gets sold to off-exchange market makers, while their limit order flow gets routed to exchanges that pay the highest liquidity rebates, Reg NMS just mandates what price it can trade at: http://news.indiana.edu/releases/iu/2014/02/study-of-potential-conflict-of-interest-in-routing-limit-orders%20.shtml http://news.indiana.edu/releases/iu/2014/02/study-of-potenti...

pjc50 11y ago

Move fast. Break things. Pay millions of dollars in SEC fines.

fahimulhaq 11y ago

Given hindsight, it's always easier to ask for more tests or code comments. I guess the rule of thumb is that if the code is there, it most probably is there for a reason. When something like this happens, it's always a series of oversights. 1. Someone forgot to add comments or test cases while writing the code (or perhaps he wrote it but people are not running the test cases before commits). 2. Someone else thought that the code is dead and deleted it (and skipped the test cases if there were some covering that particular scenario). 3. Either the person deleting the code didn't wait for the code review or code reviewers missed that as well. (If code reviews are non-existent, it's a disaster waiting to happen). So in the hindsight, write comments, write test cases, run test cases and do code reviews carefully.

fahimulhaq 11y ago

On a related note, Knight Capital lost $440 million due to a software bug a few years ago. http://www.bloomberg.com/news/articles/2012-08-02/knight-has-all-hands-on-deck-after-440-million-bug http://www.bloomberg.com/news/articles/2012-08-02/knight-has...

HolyLampshade 11y ago
Blew up the whole firm actually, and was more closely related to change management policies and systems monitoring. The SEC order there is an interesting read for anyone involved in large scale transactional systems: https://www.sec.gov/litigation/admin/2013/34-70694.pdf https://www.sec.gov/litigation/admin/2013/34-70694.pdf

osullivj 11y ago

A change like this making it through to production is indicative of a test failure, either automated or human. I have had the luxury of substantial human test teams on trading systems before 2008, and it was great. They wouldn't let something like this through...