12 ms·
Wall Street’s Big Banks Are Waging a Technological Arms Race
- olivermarks 9y agoI highly recommend Rana Faroohar's book 'makers and takers' to add some perspective to this insanity... https://www.amazon.com/Makers-Takers-Finance-American-Business/dp/0553447238 https://www.amazon.com/Makers-Takers-Finance-American-Busine...
- chollida1 9y ago> Bam. Bam. Bam. Dummy trade signals that were supposed to stay within the company’s electronic systems broke loose and slammed into computers at the New York Stock Exchange’s options markets. So many orders crashed through that by 8:44 a.m., safeguards within Goldman Sachs sprang into action, severing the connection between the company and the exchanges. Anyone whose built a trading system lives with this type of fear on a daily basis. Write your risk system first and your algo second. > The cause? A coder had mistakenly programmed a router to send placeholder bids as live orders. If not for the good graces of the options exchanges, the bank would have lost $500 million, according to the U.S. Securities and Exchange Commission. Cancellations and price adjustments reduced that to $38 million And this is at the top of my things that are unfair in the markets. Knight Capital wasn't bailed out by the exchanges for the same thing and it bankrupted them. Goldman with its alumni every where.... was.
- PerryCox 9y agoYeah I really don't understand why Goldman should get a pass like this. They fucked up, they should lose money. Or better yet, errors like this should be reversed or safeguarded against better.
- fokov 9y agoHonestly, this stuff just shows us how rigged the system is. This isn't the first time it has happened. Flash crashes caused by bad algorithms lost a bunch of money that was reversed as well. IMO, it is horrible to reverse this. They took the risk and got to keep the reward, but as soon as they fail, they get bailed out every single time.
- buschkowitz 9y agoIf I were an exchange and would earn tons of cash from Goldman on trading commissions, I would bail them out too, I guess.
- digi_owl 9y agoWhy do this bring to mind how casinos treat big name gamblers...
- lucozade 9y ago> I really don't understand why Goldman should get a pass like this. They didn't. They cancelled a lot of the trades before they were hit and a whole bunch were filtered out at the exchange end because they were way off market. They lost $38mm from what wasn't caught by either. In other words, they fucked up and lost money. And it's generally agreed that safeguards needed to be better. That was a big part of Mifid II.
- zweiterlinde 9y agoThat's a bit of a mischaracterization I think. Exchanges have well-defined rules for clearly erroneous executions, and these are not the same in the equities (Knight) and options (Goldman) markets. If anyone else had made the options trades that Goldman did that day, their trades would have been busted as well.
- chollida1 9y ago> Exchanges have well-defined rules for clearly erroneous executions, I'd disagree with this statement. it's true that each exchange does have its own rules and that they do set some guidelines but as far as I know they all have rule that state that they carry final veto power over what trade do and don't get broken. For instance from Nasdaq > Nasdaq is providing general guidance on when transactions may be deemed erroneous under Rule 11890. This guidance is not a mandate of the rule but is information on how Nasdaq generally applies the terms of the rule (at Nasdaq2019s discretion) to determine whether an execution is clearly erroneous.
- ransom1538 9y agoImagine coming back from lunch, you start to unwrap your sandwich at your desk. Then, your boss walks by and throws your sandwich across the room and explains while you were out your bug caused 440 million dollars of erroneous trades - and the company now no longer exists. https://dealbook.nytimes.com/2012/08/02/knight-capital-says-trading-mishap-cost-it-440-million/ https://dealbook.nytimes.com/2012/08/02/knight-capital-says-...
- alecco 9y ago> your bug Knight had a terrible software delivery process and a lot of deadline-oriented pressure on techies.
- drharby 9y agoStrongly concur
- lagadu 9y agoAfter it passes review and all stages of testing it's everyone's bug.
- taneq 9y agoExactly. You don't just do code reviews and run tests to catch bugs. You also do code reviews and run tests so that, when something does slip through, it's no one person's fault.
- mooreds 9y ago... and this everyone's responsibility. Makes total sense. It's almost as if everyone at a company should be on the same team :)
- deleted 9y ago[deleted]
- internetman55 9y ago
- mrchicity 9y agoDifferent situations so I doubt it has anything to do with GS nepotism. Knight couldn’t cancel their bad trades because most weren’t clearly erroneous, based on exchange rules. If the market was bid 20.01 offered at 20.02, Knight’s test program was buying 20.02 and selling 20.01 over and over. Those prices weren’t far from the prevailing market, they just executed stupidly and very quickly. Goldman was selling $10 options for $0.01, so they met the exchange rules for breaking erroneous trades since they were so far from fair value. If I were an evil Machiavellian exchange, I’d much rather stick Goldman with a nasty error than Knight. At the end of the day, exchanges need transaction volume to make a profit. Goldman is an 800lb Gorilla with huge capital reserves and diverse business lines, so they’d still come back to trade after licking their wounds. Knight almost went bankrupt.
- nostromo 9y ago> By the time the trades were blocked for the last time, less than an hour after they began, Goldman Sachs executed orders to sell more than 1.5 million options contracts for $1. The cause? A coder had mistakenly programmed a router to send placeholder bids as live orders. If not for the good graces of the options exchanges, the bank would have lost $500 million, according to the U.S. Securities and Exchange Commission. Cancellations and price adjustments reduced that to $38 million. It sure does seem like there are two sets of rules on Wall Street. I doubt any small trader would be able to reduce their losses 90+% after errantly submitting a bunch of live options orders.
- matte_black 9y agoSorry, but rules need to scale. What is acceptable for a small trader can not always be acceptable for a massive Corp.
- hueving 9y agoYour statement is backwards though. The massive corp is being granted much more leeway for acceptable behavior than that of a small trader.
- wastedhours 9y agoNot so much leeways as "huge company has a massive amount of context to use as evidence this was out of the ordinary". A smaller trader has proportionally less history in the market to use as a evidence something is wrong and the rollback rules should apply.
- Applejinx 9y agoYou're not contradicting the objection, you're just outlining another mechanic by which it happens. "The New England Patriots reached ten Super Bowls, therefore the foot out of bounds just before scoring a touchdown was clearly a mistake and out of the ordinary. Therefore, award the Patriots the point, on the grounds that they usually don't make mistakes like that. And don't check the pressure in the football please, there have been ten Super Bowls where that wasn't apparently a problem…" (I wonder who I'm insulting more, the Pats or Goldman Sachs :D )
- osrec 9y agoI personally feel we need to change the way exchanges operate. Too much emphasis exists on speed of execution. We must consider alternative auction formats where speed of execution is not the only criteria for matching an order, as it gives rise to masses of spam generated by a thundering herd of traders as they all try and fulfill the same opportunity.
- dsacco 9y agoWhat do you suggest? What features would you use to give priority to a trade?
- kasey_junk 9y agoThe standard response here is Budish’s auction system: https://bfi.uchicago.edu/research/working-paper/high-frequency-trading-arms-race-frequent-batch-auctions-market-design https://bfi.uchicago.edu/research/working-paper/high-frequen... Personally I think it glosses over lots of the reality of the markets but I’m ready for country to try it to see.
- osrec 9y agoWhat do you feel it glosses over?
- kasey_junk 9y agoIn no particular order, their solution doesn't seem to deal with: - It doesn't account for the arbitrage types that seem to bother people the most: cross-exchange arb, locality arb, regnms arb or payment for order flow. - If the discrete time chunks are very short, it doesn't seem to solve latency arb much. - If the discrete time chunks are very long, it defaults to pro-rata matching. We have pro-rata products already, they are not kind to the little guy. - Most of the really weird edge cases in exchanges happen due to so called 'exotic' order types, which are really just ways to get the exchange to atomically do something for you. In the discrete auction world I'd imagine you'd have more need for complex instructions not less. Finally, I think that the problem they are attempting to solve is fairly low priority to solve. We are operating in the lowest cost trading environment of all time. It could hardly be more fair when it comes to order execution. To throw all that out seems dumb, when there are much bigger fish to fry (such as our government bailing out one set of traders but not others).
- parm289 9y agoWhy aren’t these firms insuring against this risk? Seems like any other risk that can be managed - pay premiums and so that a third party would fund any erroneous losses.
- kasey_junk 9y agoBecause they don't have to? They don't suffer from the risk... Clearing frequently act like a form of insurance for smaller firms, up to and including asking for killswitches in your trading engines and audits of your risk procedures. But big firms clear themselves so don't have that. Theoretically the internal risk team is responsible for that but those teams are frequently undergunned.
- parm289 9y agoWhat about a firm like Knight?
- kasey_junk 9y agoCleared themselves. I'd be shocked if any company in the world would act as counterparty to a trade that hedged 'operational risk at Knight scale'. One of the thing many of us in the industry at the time commented on, was how little was done to bail Knight out. Other than the 'oligarchy' argument that states that the old timers hated them (they did) the argument I subscribed to was, they weren't systematically important. Their entire function in the market could be taken over quickly by someone else with little disruption and largely they were the only ones that lost money on that day (not really but to an approximate).
- AnimalMuppet 9y ago"If you just hurt yourself, nobody cares. If hurting yourself hurts others, then other people care." That's pretty cold-blooded and brutal, but it does make some sense...
- parm289 9y ago
- thomaspaine 9y ago> The cause? A coder had mistakenly programmed a router to send placeholder bids as live orders. If not for the good graces of the options exchanges, the bank would have lost $500 million, according to the U.S. Securities and Exchange Commission. Cancellations and price adjustments reduced that to $38 million It's interesting to read the full SEC report because it's a little more complicated than that: https://www.sec.gov/litigation/admin/2015/34-75331.pdf https://www.sec.gov/litigation/admin/2015/34-75331.pdf It basically outlines a cascade of failure in controls, bad configuration defaults, and poor SDLC. In particular: >In addition, the firm’s operation and management of its electronic “circuit breakers” did not effectively block the erroneous orders sent on August 20. These circuit breakers existed to prevent erroneous orders by halting all message traffic to the exchanges once that traffic had exceeded a certain rate. However, on August 20, the firm’s control personnel repeatedly lifted the circuit breakers blocks between 8:44 a.m. and 9:32 a.m., thereby permitting additional erroneous orders to be sent to the exchanges. Before lifting the circuit breaker blocks, the control personnel did not obtain authorization from the responsible technology employees, as required under written firm policies. >The firm’s policies relating to the manual “lifting” of those circuit breakers were not disseminated to or fully understood by the employees responsible for deciding when the circuit breakers should be lifted, and, prior to August 20, 2013, GSCO personnel had lifted circuit breaker blocks shortly after learning of the block and while still investigating the cause of the circuit breaker trip. From what I remember, this circuit breaker was notorious for raising so many false positives that control personnel just got used to lifting it without thinking.
- marcc 9y agoThat’s really a terrible place to put people in. From what this says, the SEC put part of the blame on the employees who lifted the circuit breakers without approval, and also state that employees didn’t know that they needed approval? Before lifting the circuit breaker blocks, the control personnel did not obtain authorization from the responsible technology employees, as required under written firm policies. The firm’s policies relating to the manual “lifting” of those circuit breakers were not disseminated...
- Zenst 9y agoOne upside is such environments help nurture future technologies as they will happily pay millions for something just a bit faster than what they already have and more so before others have it. Hence it is avenues of business like this that have help feed certain industries over the years, from networking to FPGA's. Another avenue of financing technology which we all know is the military. As for the whole stock market being driven faster and faster in precision from seconds to nanoseconds and picoseconds, I've always questioned the need to drive towards a faster processing and if the stock market only updated once every minute, it would sure help in curtailing many ineeded aspects that are creeping into stock trading.
- senatorobama 9y agoHow do I get a job in this field?
- paulie_a 9y agoStep one: sell your soul
- poster123 9y agoHigh-frequency trading reduces transaction costs and therefore reduces the cost of capital, encouraging investment.
- kasey_junk 9y agoApply.
- poster123 9y agoEfinancialcareers is a good site to peruse job ads and see what people are looking for. The banks themselves all have job postings on their sites. Once you have the main qualifications you can apply directly or contact a financial recruiter.
- notyourday 9y agos/$TITLE/Salaries of people with clue are going up, up and up/g
- goldmanguy 9y agoExcept it wasn't the machines that caused the issue, it was a bug (and lack of following the controls in place) caused by Lucas Renick, of Goldman Sachs, who was fired on the spot and remained unemployed (though still on the Goldman payroll, in order to avoid a lawsuit) for many months until he joined Bank of America and later Morgan Stanley
- lawrenceyan 9y agoPlaces like D.E. Shaw, Two Sigma, Jane Street, Renaissance, Citadel, etc... were doing this long before any of these banks. To call it innovation is a bit of a misnomer in my opinion, more of a catching up with the times really.