5 ms·

I've worked in "FinTech DevOps" for the past 10 years and this reminds me of a story from a past job: We hired the Global Head of Clearing [0] from a big bank

I've worked in "FinTech DevOps" for the past 10 years and this reminds me of a story from a past job: We hired the Global Head of Clearing [0] from a big bank and on their first day they were giving a presentation on their background and how financial clearing works. Someone asked "What is your nightmare scenario?" The response: We have a large overnight position in a security. Our Prime Broker (PB) [1] comes back and says "We disagree with you on the position so you can't trade." Even if it turns out that the PB is wrong, by the time it all gets sorted out, the market has moved so much in that security that it bankrupts us. I've seen it happen to other firms and it's not pretty. I also started at Knight right after their big outage. [2] People like to talk about the big tech outages sinking financial firms but it can just as easily be plain old issues with bookkeeping that can blow up a firm. 0 - https://www.investopedia.com/terms/c/clearing.asp https://www.investopedia.com/terms/c/clearing.asp 1 - https://www.investopedia.com/articles/professionals/110415/role-prime-broker.asp https://www.investopedia.com/articles/professionals/110415/r... 2 - https://www.thestreet.com/investing/stocks/knight-capital-shuts-down-trading-on-electrical-outage-11752932 https://www.thestreet.com/investing/stocks/knight-capital-sh...

dickjocke 7y ago
Totally talking out of my butt here, I have no finance knowledge. But just as a matter of general game theory stuff I understand that sometimes you have to push all your chips in on a position, but that doesn't seem sustainable? Like how often is it reasonable for a big bank to have a position in a single security that could bankrupt them?
- alexpotato 7y ago
  Great question! Realized I was mixing different company sizes in the original post. This would most likely be hitting smaller to medium sizes firms (e.g. Hedge Funds) that are holding a larger position in a security. The small/med firms don't "self clear" so they tend to depend on a larger firm to do that for them. I should also point out: I used the example of one big position but it could also be a bunch of medium sized positions and then the PB says "Hmmm, we don't agree with 1 or more of those positions so we are stopping trading with you on all of them."
- lowdose 7y ago
  The London whale, Long Term Capital Management, etc etc.
- btilly 7y ago
  For some companies, every single night! Money chases the sun. At the end of trading in New York you make a swap with a company in Japan that at the end of trading in Japan does the same with London then back to New York. Any money not swapped doesn't make money overnight. This is called the overnight repo market. But if you don't get your money back in the morning, you've got a problem.
  - 7373737373 7y ago
    Is it possible to quantify how much "active" vs "passive" money/capital there is?
    - btilly 7y ago
      I doubt that it is possible to properly quantify how much capital there is, let alone to classify it. http://www.fintools.com/docs/Warren%20Buffet%20on%20Derivatives.pdf http://www.fintools.com/docs/Warren%20Buffet%20on%20Derivati... is nearly 20 years old, but Warren Buffet's criticism of how we account for the value of derivatives is as true today as it was then.
- seanhunter 7y ago
  It's never reasonable although it doesn't stop people from doing it. History has shown us the folly of this (eg read "When Genius Failed" for a great history of the LTCM collapse). Due to multiple failures, large banks are to a greater or lesser extent prevented from doing this, but smaller financial players do get themselves into trouble.
czinck 7y ago
Why wouldn't the Prime Broker just force you to sell it, instead of blocking all trades? I understand that they could dislike the position because it's moved enough to bankrupt you already, or unloading it could move the market even more against you, but that's gotta be better than just holding on to it longer.
- JumpCrisscross 7y ago
  > Why wouldn't the Prime Broker just force you to sell it One, that could result in a fire sale. Freezing is safer than forcing liquidation. If your liabilities outweigh your assets, the latter are the PB's; getting pennies on the dollar instead of dimes is against their interest. Two, the client may be able to post collateral. You don't want to lose an institutional relationship because you were trigger happy. Three, you might have made a mistake. Freezing the position until you get your facts straight is prudent in most markets.
- seanhunter 7y ago
  They can't force you to close the position - they "Don't Know" (DK) the position. ie there is a dispute with you for whatever reason that you actually have that position and nothing can be done either way until this is clarified. If they are right, making you put on a trade to close the position they don't think you have will actually put you into a position of equal size in the other direction. eg: You are my pb and I think I'm long 20 S&P e-mini futures. You don't know that trade so think I'm flat. If you make me "close" my position and you're right then I'm actually now short 20 futures. This type of situation can happen a lot more easily than you might think especially in markets where there are multiple trading channels (eg voice and electronic) and settlement instructions etc can go wrong. One place I worked we had a trade be DKd because the instructions were sent by fax and needed to be confirmed by voice. The pos Solaris printer driver used to leak memory so one day it leaked so much memory that it just fell over and died and one of the trade confirms wasn't printed out, therefore the human who had the shitty job of picking them up off the printer and calling the other side to confirm the trade didn't confirm. 5m dollars right there.
whitepoplar 7y ago
Given your knowledge of the industry, which retail broker(s) do you think is the absolute safest place to trade + hold equities? Interactive Brokers? Fidelity/Schwab? One of the big banks?
- atomicnumber3 7y ago
  Recently-ex-sell-side here: retail investors generally don't need to worry about this at all. There's a lot of protection. Your assets are held separately from the firm's assets, you have SIPC insurance, and there are enormous amounts of regulation in place to try to prevent you from even needing to rely on those two mechanics. Really the worst that could happen is it might be a headache to get to your stuff for a small period of time while something is figured out as to where your assets are going if your brokerage just up and poofs. That aside, I generally trust the discount brokerages a bit more than people like BofA/Merril and Wells Fargo and JP Morgan, just because I feel like the discount brokerages are a bit more content to just sit on deposits and earn interest (interest accounted for > 50% of schwab's revenue last year) rather than get up to highly leveraged hi-jinks or open fraudulent advisor accounts.
- rajacombinator 7y ago
  IB for sure, it’s not even a question.
  - whitepoplar 7y ago
    Any chance you can explain some of the reasoning, even if it's a bit esoteric? One thing that's keeping me from going 100% IB is the lack of a transparently-stated fraud guarantee, like Schwab/Fidelity have. Other than that, IB seems like the best.
jmb12686 7y ago
I was doing "devops" for Scottrade's order management system during the exact same time as the Knight implosion. If I remember correctly, the Facebook IPO was a larger mess we had to mop up for our client base...ahh the memories...glad to be out of consumer / retail finance these days.
- bryan_w 7y ago
  Remember when we thought the Twitter IPO was going to wreck us and it turned out to be a big nothingburger
- cik 7y ago
  This is exactly what I liked about consumer finance. The problems were absolutely fascinating, and when dealing with transactional data it was at a near real-time scale that you don't see anywhere else.

dickjocke 7y ago

Totally talking out of my butt here, I have no finance knowledge. But just as a matter of general game theory stuff I understand that sometimes you have to push all your chips in on a position, but that doesn't seem sustainable? Like how often is it reasonable for a big bank to have a position in a single security that could bankrupt them?

alexpotato 7y ago
Great question! Realized I was mixing different company sizes in the original post. This would most likely be hitting smaller to medium sizes firms (e.g. Hedge Funds) that are holding a larger position in a security. The small/med firms don't "self clear" so they tend to depend on a larger firm to do that for them. I should also point out: I used the example of one big position but it could also be a bunch of medium sized positions and then the PB says "Hmmm, we don't agree with 1 or more of those positions so we are stopping trading with you on all of them."
lowdose 7y ago
The London whale, Long Term Capital Management, etc etc.
btilly 7y ago
For some companies, every single night! Money chases the sun. At the end of trading in New York you make a swap with a company in Japan that at the end of trading in Japan does the same with London then back to New York. Any money not swapped doesn't make money overnight. This is called the overnight repo market. But if you don't get your money back in the morning, you've got a problem.
- 7373737373 7y ago
  Is it possible to quantify how much "active" vs "passive" money/capital there is?
  - btilly 7y ago
    I doubt that it is possible to properly quantify how much capital there is, let alone to classify it. http://www.fintools.com/docs/Warren%20Buffet%20on%20Derivatives.pdf http://www.fintools.com/docs/Warren%20Buffet%20on%20Derivati... is nearly 20 years old, but Warren Buffet's criticism of how we account for the value of derivatives is as true today as it was then.
seanhunter 7y ago
It's never reasonable although it doesn't stop people from doing it. History has shown us the folly of this (eg read "When Genius Failed" for a great history of the LTCM collapse). Due to multiple failures, large banks are to a greater or lesser extent prevented from doing this, but smaller financial players do get themselves into trouble.

czinck 7y ago

Why wouldn't the Prime Broker just force you to sell it, instead of blocking all trades? I understand that they could dislike the position because it's moved enough to bankrupt you already, or unloading it could move the market even more against you, but that's gotta be better than just holding on to it longer.

JumpCrisscross 7y ago
> Why wouldn't the Prime Broker just force you to sell it One, that could result in a fire sale. Freezing is safer than forcing liquidation. If your liabilities outweigh your assets, the latter are the PB's; getting pennies on the dollar instead of dimes is against their interest. Two, the client may be able to post collateral. You don't want to lose an institutional relationship because you were trigger happy. Three, you might have made a mistake. Freezing the position until you get your facts straight is prudent in most markets.
seanhunter 7y ago
They can't force you to close the position - they "Don't Know" (DK) the position. ie there is a dispute with you for whatever reason that you actually have that position and nothing can be done either way until this is clarified. If they are right, making you put on a trade to close the position they don't think you have will actually put you into a position of equal size in the other direction. eg: You are my pb and I think I'm long 20 S&P e-mini futures. You don't know that trade so think I'm flat. If you make me "close" my position and you're right then I'm actually now short 20 futures. This type of situation can happen a lot more easily than you might think especially in markets where there are multiple trading channels (eg voice and electronic) and settlement instructions etc can go wrong. One place I worked we had a trade be DKd because the instructions were sent by fax and needed to be confirmed by voice. The pos Solaris printer driver used to leak memory so one day it leaked so much memory that it just fell over and died and one of the trade confirms wasn't printed out, therefore the human who had the shitty job of picking them up off the printer and calling the other side to confirm the trade didn't confirm. 5m dollars right there.

whitepoplar 7y ago

Given your knowledge of the industry, which retail broker(s) do you think is the absolute safest place to trade + hold equities? Interactive Brokers? Fidelity/Schwab? One of the big banks?

atomicnumber3 7y ago
Recently-ex-sell-side here: retail investors generally don't need to worry about this at all. There's a lot of protection. Your assets are held separately from the firm's assets, you have SIPC insurance, and there are enormous amounts of regulation in place to try to prevent you from even needing to rely on those two mechanics. Really the worst that could happen is it might be a headache to get to your stuff for a small period of time while something is figured out as to where your assets are going if your brokerage just up and poofs. That aside, I generally trust the discount brokerages a bit more than people like BofA/Merril and Wells Fargo and JP Morgan, just because I feel like the discount brokerages are a bit more content to just sit on deposits and earn interest (interest accounted for > 50% of schwab's revenue last year) rather than get up to highly leveraged hi-jinks or open fraudulent advisor accounts.
rajacombinator 7y ago
IB for sure, it’s not even a question.
- whitepoplar 7y ago
  Any chance you can explain some of the reasoning, even if it's a bit esoteric? One thing that's keeping me from going 100% IB is the lack of a transparently-stated fraud guarantee, like Schwab/Fidelity have. Other than that, IB seems like the best.

jmb12686 7y ago

I was doing "devops" for Scottrade's order management system during the exact same time as the Knight implosion. If I remember correctly, the Facebook IPO was a larger mess we had to mop up for our client base...ahh the memories...glad to be out of consumer / retail finance these days.

bryan_w 7y ago
Remember when we thought the Twitter IPO was going to wreck us and it turned out to be a big nothingburger
cik 7y ago
This is exactly what I liked about consumer finance. The problems were absolutely fascinating, and when dealing with transactional data it was at a near real-time scale that you don't see anywhere else.