11 ms·

Oil crash busted broker’s computers and inflicted big losses

204 points by wclax04 6y ago

AndrewBissell 6y ago
Incorrectly assuming values can never be negative is an all-too-common occurrence in trading and financial software. In 2012 Swedish stock futures trading was suspended for a time because their matching engine used an unsigned type for order quantities and someone submitted an order with a negative value which wrapped around to 4 billion: https://www.reuters.com/article/markets-sweden-bug/swedish-stock-futures-market-problem-caused-by-mega-order-glitch-idUSL5E8MTB4I20121129 https://www.reuters.com/article/markets-sweden-bug/swedish-s... Interactive Brokers' software is usually very solid. I'm surprised they weren't ready to handle this, the possibility of oil going negative had been discussed for some time before it happened.
- H8crilA 6y ago
  I particularly dislike the attitude of the CEO which shows he either doesn't know or pretends that he doesn't know how such contracts work. Please read the contract specs and educate yourself a little, Mr. Peterffy, they're public and free. > Peterffy said there’s a problem with how exchanges design their contracts because the trading dries up as they near expiration. The May oil futures contract -- the one that went negative -- expired the day after the historic plunge, so most of the market had moved to trading the June contract, which expires May 19 and currently trades around $24 a barrel. > “That’s how it’s possible for these contracts to go absolutely crazy and close at a price that has no economic justification,” Peterffy said. “The issue is whose responsibility is this?” Nobody ever promised neither liquidity nor positivity of prices, it is the fault of the brokerage, plain and simple. Thankfully $100M is something that IBKR can take on their books (they have $3B of cash according to the latest filling).
  - AndrewBissell 6y ago
    Yes this is a very CYA statement. IB has been in this business for a long time, Peterffy knows very well that expiring futures contracts can experience all kinds of liquidity problems and large swings in price.
    - azinman2 6y ago
      But if it’s never happened before, and you’re some random programmer making this stuff, I’m not surprised there are all kinds of assumptions being made when choosing data types and validating inputs...
  - topynate 6y ago
    "No economic justification"? That has to be faux-naïf. Yes, I was shocked at first to hear of prices turning negative. For about thirty seconds. Once I learned that the oil had to be delivered somewhere without enough storage, and under conditions of extreme low demand, it was perfectly obvious that you'd end up paying someone else to take the oil off your hands. Either Peterffy is too stupid to understand mass-media reports on his own field of business – not likely – or he's disingenuous, which says something about how he sees his clients.
  - scurvy 6y ago
    Peterffy is a long time industry vet. He brought a lot of tech to trading in the 80's. He knows very well how things work.
- bobcostas55 6y ago
  >Interactive Brokers' software is usually very solid. That is certainly not my experience. Endless bugs in TWS over the years, and the support people are unbelievably rude. Every time I try to report a bug they start out by blaming me, it usually takes 2-3 back and forth rounds until they admit it's actually broken and tell me they'll forward the issue to the tech people. After that it's radio silence and you never know if they'll actually fix it or not. And let's not get into the disaster that is their API...
  - AndrewBissell 6y ago
    I'll admit I haven't worked with their API or done anything too fancy with it. My experience is all relatively infrequent click trading, chart generation, and such.
  - brobinson 6y ago
    Completely agree. Their mobile app, TWS, API, and websites are all a nightmare. I left them as soon as I stopped trading on margin (the only really compelling reason to use them for a retail trader/investor).
- isoos 6y ago
  > Interactive Brokers' software is usually very solid. Except their web portal. 40-50% of the time it is unable to load my portfolio data (even without the current market value, just the number of stocks and cash balances). At the same time I log into the mobile app, it forces a logout on the web app (why?) and it is able to load the balances and portfolio. No explanation. Same network, no adblocker or other browser plugin.
  - scurvy 6y ago
    The single session thing is annoying. I'm guessing it is related to their various data provider regulations. The large brokers can get around this because, well, you are not really seeing pool depth and the like.
- adrr 6y ago
  I'm going to guess there's two teams involved on the IB side of things. There's financial engine team which processes and handles trades. And there is the UI team that displays the data and allows people to create trades. I bet the engine handled these fine and issue was that the UI team had data validation checks to prevent negative values.
  - analyticstime 6y ago
    Nope. It was not possible to trade at negative.
BubRoss 6y ago
As usual, this nonsense headline is clickbait for the truth - some traders couldn't trade when oil went below zero.
muyth 6y ago
Would any of the traders mentioned that owe all that money to Interactive Broker's have to pay the total amount? Or is that part of Interactive Brokers' loss claim.
- whatok 6y ago
  From the story: > Customers will be made whole, Peterffy said. “We will rebate from our own funds to our customers who were locked in with a long position during the time the price was negative any losses they suffered below zero.” I'd imagine this is so they don't get sued and have some shoddy code pop up during trial.
  - vmception 6y ago
    or they're raising capital and the end justifies the means same result of avoiding litigation, but even fewer consequences
changoplatanero 6y ago
non paywall link https://business.financialpost.com/pmn/business-pmn/oil-crash-busted-a-brokers-computers-and-inflicted-huge-losses https://business.financialpost.com/pmn/business-pmn/oil-cras...
elliekelly 6y ago
> Peterffy said there’s a problem with how exchanges design their contracts because the trading dries up as they near expiration. The May oil futures contract -- the one that went negative -- expired the day after the historic plunge, so most of the market had moved to trading the June contract, which expires May 19 and currently trades around $24 a barrel. > “That’s how it’s possible for these contracts to go absolutely crazy and close at a price that has no economic justification,” Peterffy said. “The issue is whose responsibility is this?” It’s pretty well known that commodity futures contracts are a game of hot potato for most investors as the expiration date approaches. But the Interactive Brokers CEO doesn’t offer an alternative solution. How would the contracts be structured instead that would avoid this? I don’t see how it would be possible.
- wbl 6y ago
  Financial settlement against a spot market in a large port.
  - curiousllama 6y ago
    Cash settled contracts exist. These people just chose not to buy them. The question is how to structure physically settled contracts. After all, oil needs to get delivered to someone at some point.
    - AnimalMuppet 6y ago
      If you get into the market for physically settled contracts with no intention of taking delivery, then you're almost certainly a speculator. I'm not sure that it's the market's job to make that safer for you. I am not justifying inaccurate pricing. Burning speculators is fine, but give everyone accurate information.
      - vardump 6y ago
        Or possibly someone who needs large amounts of oil and want to hedge against fluctuations? Freight industry, airliner, etc.
SrslyJosh 6y ago
I am playing my second tiniest violin for them right now.
- joncrane 6y ago
  Are you saving the tiniest one for when <insert noted eschewer of wearing masks in public> comes down with COVID-19? (I'm curious who the tiniest one is for)
  - mring33621 6y ago
    tiny violins fit well in tiny hands
0x8BADF00D 6y ago
It's really odd that this bug occured, as IB has no issue pricing credit spreads with negative values. Must be an issue specific to commodities futures contracts. I wonder what data types they were using.
- derriz 6y ago
  I’ve written code (a year or so ago) against the IB API and it seemed clear to me that the API was a thinish skin over multiple backend systems. The feed you get for products from different markets (even different futures markets) was different - the population of fields in price and trade feed was wildly inconsistent. I’m guessing each market is accessed in by a different IB system. Negative prices are a feature of some massively traded futures - interest rate futures for example - but can effectively never occur for index futures or the like.
  - centimeter 6y ago
    From what I've seen, almost all financial companies developed a bunch of systems for different security types (which all have different rules and edge cases) independently, and only tried to tie them together as time went on. It's a recipe for a lot of confusion and inconsistency. There are tons of opportunities in finance to make short-sighted proclamations like "the number of futures in this kind of contract is always 100" or "this type of security can't go negative", and have it be true at the time, but false 5 years later when they add a new type of contract.
    - 0x8BADF00D 6y ago
      Kind of orthogonal, but I’ve often faced pushback from Product Managers when trying to future proof a feature that I’m going to ship. The pushback is usually “oh, there’s no way our customers will ever need that!”. Only until many years later will it blow up in your face, with costly consequences.
      - centimeter 6y ago
ralph84 6y ago
Would be nice to have more details on exactly what happened with the trades. Did the trades clear? Did IB liquidate the contracts before expiration? Was someone on the hook for taking physical delivery?
- ashtonkem 6y ago
  I feel like any sane clearing house should’ve rejected these trades.
ninetyfurr 6y ago
Oh a guy with $77,000 in his day trading account didn't know futures can go negative. I'm shocked. Ironically this is the same guy who will sell you a gym contract without a cancellation option and blame you for not doing your research when you owe him $1100.
socrates1998 6y ago
If you don't know what is going on, then why on earth would you risk so much money? IB fucked up, no doubt, but these idiots are trading shit they know nothing about. Don't trade on margin.
- SilasX 6y ago
  Well, yeah, it's reasonable to expect investors to appropriately researching something before buying, and be cognizant of the risk that it could crash. It's not reasonable for them to expect to deal with a platform that misrepresents the state of the market and executes trades at a non-market price, as was happening here. (It was telling them the oil futures still had a positive price when it was negative, and making them pay on that basis.)
  - james-mcelwain 6y ago
    > Well, yeah, it's reasonable to expect investors to appropriately researching something before buying, and be cognizant of the risk that it could crash. Is it? Isn't this the entire reason risk management departments exist?
- codenesium 6y ago
  I don't think they knew in this case they were potentially trading on margin. The broker didn't know either which is crazy.
  - whatok 6y ago
    If anyone is trading futures and did not know they were trading on margin or trading highly levered instruments then they either are trading on a platform that has zero compliance or they misrepresented themselves as an investor.
    - CrazyStat 6y ago
      The broker calculated the margin requirements based on the assumption that the price could not go negative. So the investors thought they were risking $30 per contract when it was actually a couple orders of magnitude larger than that. Effectively the broker lied to them about how leveraged they were.
codenesium 6y ago
Wasn't the original purpose of futures to let farmers and others lock in prices early so they can mitigate risk? Speculation on futures seems dumb if you have no intention of taking delivery.
- ashtonkem 6y ago
  Taking physical delivery and hedging are not one and the same. It’s entirely possible to use a cash settled future to hedge against market movements; the farmer sells at a steep loss, but their cash settled wheat futures offset a large percentage of the loss on a cash basis. The distinction you’re looking for here is those who are speculating on market prices, vs. those who are hedging against market prices. If you use or sell oil in large amounts, it makes sense to use futures to stabilize your downside risk, even if those futures are cash settled. That being said, I think that cash settled futures make purely speculative trading much easier, so you’d have a good point if that’s what you were heading towards.
  - codenesium 6y ago
    That is the point I was trying to make. Basically if you're in the business of producing or buying and selling the commodity the futures are for you. If you're just speculating how does that help anybody? I guess you could make the argument that having more eyes on the market means there is more information so the price is a better reflection of the true value.
    - PeterisP 6y ago
      You may hedge your risk of prices of a different but related product. E.g. a bread manufacturer wants stability on their cost of goods, so they may buy wheat futures even though they don't ever want to receive direct shipment of wheat, they're dealing with specific regional flour suppliers but the wheat futures at a major location are a good proxy for that price.
    - whatok 6y ago
      If you only had farmers and individuals purchasing their goods allowed to trade futures, there is direct incentive for either party to manipulate the physical market through their actions, in ways that they only could, that wouldn't necessarily make any economic sense and could have spillover effects into the real economy. By having a more open market for futures, you theoretically have a more efficient market for all participants which you mention in your last sentence.
flint 6y ago
I'm surprised IB let speculators trade in a contract going to delivery. I worked as a risk manager in a commodity trading firm and only hedgers qualified to take delivery were permitted to hold contracts going to delivery.
- everlost 6y ago
  Wasn't this a cash settled contract?
  - whatok 6y ago
    No.
    - jdadj 6y ago
      It's not a cash settled contract. "Delivery shall be made free-on-board ("F.O.B.") at any pipeline or storage facility in Cushing, Oklahoma with pipeline access to Enterprise, Cushing storage or Enbridge, Cushing storage." Source: https://www.cmegroup.com/trading/energy/crude-oil/light-sweet-crude_contract_specifications.html https://www.cmegroup.com/trading/energy/crude-oil/light-swee...
    - mathgenius 6y ago
      It may have been ICE's crude contract, which is cash settled: https://www.theice.com/products/213/WTI-Crude-Futures https://www.theice.com/products/213/WTI-Crude-Futures "The West Texas Intermediate Light Sweet Crude Oil futures contract is cash settled against the prevailing market price for US light sweet crude."
      - whatok 6y ago
        Sure, but a majority of trading was done on CME futures. In either case, you still ran into the issue of negative prices. USO only recently started explicitly stating that they would also potentially invest in ICE futures alongside CME futures.
  - 6y ago
dilandau 6y ago
More to the point is that a bunch of dumbasses were speculating on oil futures without understanding the nature of the market, and no ability to take delivery on the oil in any event. Nobody should be holding those when they're so close to expiry unless they know what they're doing.
thanatropism 6y ago
About two months ago I delivered to a private client a (monthly-basis) prediction model for ethanol prices that included Brent -- in logarithm -- as a predictor. I know the monthly spot average of that statistic (fetched from FRED, the Fed of St. Louis system) won't turn negative, but still...
supernova87a 6y ago
For futures on physical deliverable objects (well, I guess most futures are such, but anyway) -- would volatility and speculation be dampened/improved if the clearinghouse forced everyone (or the seller) participating in a trade to certify that they had rights to the specific thing being traded? Could actually produce the contract -- like the oil producer is certified to have <xyz> barrels allowed to be sold? My notion is that if much of the trading (and it can be shown by futures volumes) cannot possibly be on actual physically deliverable quantities, then most must be "speculation" by people who cannot actually produce the asset. Would this be a help to stabilize the market? I know it all has to get settled in the end by the expiration date, but just an idea.
- kasey_junk 6y ago
  It likely will settle the markets by driving spreads through the roof and costing buyers/sellers of the physical items. The underlying theme here is that speculators are not providing utility to the market, which is wrong. They are bridging time and risk.
- lazyjones 6y ago
  > then most must be "speculation" by people who cannot actually produce the asset. Let's just call it what it is: gambling.
- Rexxar 6y ago
  Some companies can use this as a proxy to protect themself against variation of a product they need. For example an airline can use this to protect itself against price variation of kerosene once they have sold a ticket. As kerosene is not directly available on commodity market they use this future because their price are strongly correlated to kerosene price.
mathgenius 6y ago
If you look at the CME website using the wayback machine, you can clearly see a hi and low limit for the price. There is no high limit, but the low limit is set at "0.01". [1] This is kind of weird, because the snapshot is pulling current market prices, and still showing an incorrect hi/lo limit. It looks like they have since removed the hi/lo display: [2]. Someone should take a screenshot of this. [1] https://web.archive.org/web/20200117115242/https://www.cmegroup.com/trading/energy/crude-oil/light-sweet-crude.html https://web.archive.org/web/20200117115242/https://www.cmegr... [2] https://www.cmegroup.com/trading/energy/crude-oil/light-sweet-crude.html https://www.cmegroup.com/trading/energy/crude-oil/light-swee...
LatteLazy 6y ago
Fuck IB and this “trader”, idiots should lose their shirts, that’s one of the intended outcomes of an efficient market. That said, there is an issue here with futures contracts: you can get very very large leverage when the price is near zero. This is the real issue with instruments that can negative price and just like their are “circuit breakers” in markets for big price swings, there should be breakers for entering the “near zero” range.
- mostlymeme 6y ago
  An efficient market that don't let traders operate? Did you even read the article? Futures contracts that CAN BECOME NEGATIVE don't let large leverage when price is near zero, that's NOT TRUE. Future contracts margin is calculated with SPAN, and if it's done correctly, it considers the scenarios where price can go below 0.
  - LatteLazy 6y ago
    >Futures contracts that CAN BECOME NEGATIVE don't let large leverage when price is near zero, that's NOT TRUE It clearly is. If I can buy a contract for 1c, I can get 100,000 contracts for 1000usd. Then if the price rises of falls by 1usd, I'm up/down 100,000 dollars. Can you think of any retail product with that sort of leverage? That's the danger of putting zero in a denominator.
    - analyticstime 6y ago
      You can't buy a contract for 1c. You need to pay the margin, and the margin for futures is not calculated based on current price, it's calculated with SPAN, that considers different scenarios in which you can lose money. For example, for this contract Bloomber says IB asked for $30 margin. But the margin is usually $7000 for this contract, that it was IB should have requested as collateral at least for each contract. in a day with that volatility should be much higher in IB, as they take that also in consideration, probably around $20000 per contract. The problem was that IB didn't consider scenarios in which the price can go below 0. The software was designed in that way. But it shouldn't.
haltingproblem 6y ago
“Five days, including the weekend, with the coronavirus going on and a complex system where we have to make many changes, was not a sufficient amount of time,” he said. “The idea we could have bugs is not, in my mind, a surprise.” He also acknowledged the error in the margin model Interactive Brokers used that day.....We have called the CFTC and complained bitterly,” Peterffy said. “It appears the exchanges are going scot-free.” Thomas Peterffy must think we are idiots. Anyone who trades commodity contracts for any period of time knows that the real cost of the contract is the actual cost of the commodity - storage costs. When storage costs spike and the actually commodity spot costs go down, the future will become negative! One way to get a handle on storage costs is think of them being inversely proportional to the value density. The higher the value density, e.g. gold the less the storage costs. Oil is not so dense so storage costs matter. Financial instruments like the Treasury Bonds and the S&P futures contract have zero storage costs. Storage cost is of-course different than carry cost (the cost of funding your long position). On another aside, I have known folks who have worked at IB in the past, and their systems absolutely suck dead goats. Huge masses of legacy C++ code with poor testing. Most of these brokerage firms have legacy code base from the 90s that is poorly understood. They also have nonexistent organizational quotient around code validation, correctness and testing their risk models. A futures margin model is not something one can whip up over a weekend but a good CS undergraduate can program one over a couple months. Sorry for the IB customers but I have zero sympathy for IB or should I say negative ;)
- hellofunk 6y ago
  Agreed. IB's data is also not so accurate much of the time; the way it reports volume in particular is misleading. I've seen traders abandon IB after big losses because their strategies required actual live volume data that didn't suck, and nearly any other broker provides this.
  - ajflores1604 6y ago
    What are some recommended alternative platforms that provide api access?
nromiun 6y ago
> Its software couldn’t cope with that pesky minus sign, even though it was always technically possible -- though this was an outlandish idea before the pandemic -- for the crude market to go upside down. Wow, just wow. They are handling millions (billions?) of dollars every day and couldn't find the time to test that they can just DISPLAY a minus sign. That's insane. And it's not even that outlandish. People were saying it could go into the negative weeks before it happened. This just seems like pure laziness. Just pretend everything is business as usual.
- EdwardDiego 6y ago
  Makes me wonder about their testing - any decent tester is going to throw a negative number in as a matter of course. I have a gut feeling that this bug was flagged and closed as WONTDO because "that'll never happen"
  - dmurray 6y ago
    Just as likely it was a feature. If you really think "that'll never happen", then the right thing to do is explicitly ignore negative prices or refuse to send orders at negative prices, and that's exactly the kind of failsafe I'd want to have in my trading software, since you can get spurious prices for all kinds of reasons. It turned out the assumption was wrong and yeah, you should remove the logic handling that once it's evident futures may go negative, and you should have a process capable of making that change with only a day or two notice. But being robust the rest of the time at the expense of mishandling a once-in-fifty-years event is not in itself a bug. If you disagree, should a trading system also allow negative prices for precious metals futures? Stocks? Currencies? Options? Bonds? Futures on stocks or bonds? I can contemplate all of those trading negative in extraordinary, contrived scenarios but I would design systems today not to trade them at negative prices.
    - EdwardDiego 6y ago
      But the fact that they didn't display negative prices to customers indicates that they were explicitly ignoring them in a very very bad way. I'm suspecting an abs() function was involved.
danans 6y ago
Hopefully, one of the things that we price in going forward is the volatility in the price of oil vs alternatives, especially those that can be produced domestically in a way decoupled from international events. That includes natural gas and especially renewables. Remember that just a 6 years ago, oil was north of $100/barrel [1], and recently it's close to zero. Wind on the other hand has had a steadily reducing LCOE[2][3]. Solar's LCOE is also steadily reducing [3]. Volatility has a cost. With oil, it's one that the US and other countries hae historically tried to dampen with various industrial and political methods (the national strategic oil reserve, military/political "influence" on foreign oil producers, subsidies for domestic production), but seems like the current situation is beyond those methods' ability to control. 1. https://www.macrotrends.net/1369/crude-oil-price-history-chart https://www.macrotrends.net/1369/crude-oil-price-history-cha... 2. https://www.energy.gov/sites/prod/files/2015/08/f25/LCOE.pdf https://www.energy.gov/sites/prod/files/2015/08/f25/LCOE.pdf 3.https://en.wikipedia.org/wiki/Cost_of_electricity_by_source#/media/File:US_projected_cost_of_wind_power.png https://en.wikipedia.org/wiki/Cost_of_electricity_by_source#... 4. https://en.wikipedia.org/wiki/Cost_of_electricity_by_source#/media/File:EU-PV-LCOE-Projection.png https://en.wikipedia.org/wiki/Cost_of_electricity_by_source#...
neonate 6y ago
https://archive.md/EGIGB https://archive.md/EGIGB
raviolo 6y ago
Comments about storage costs are nice theory but don’t tell me those storage costs went up 50 dollars a barrel to cause CLK0 to go from +10 to -40 within something like 30 minutes. It was all forced liquidation by brokers like IB and people absolutely bamboozled by negative prices puking their positions. Nothing to do with costs for actual storage.
unixhero 6y ago
I would recommend http://www.saxobank.com http://www.saxobank.com , it's a kick ass broker.
tester89 6y ago
I don’t really understand why the traders would end up owing money. If they thought they were paying 0,01 $/bbl but we’re actually “paying” -37 $/bbl wouldn’t IB owe them?
- Stierlitz 6y ago
  @tester89 > I don’t really understand why the traders would end up owing money. If they thought they were paying 0,01 $/bbl but we’re actually “paying” -37 $/bbl wouldn’t IB owe them? My thoughts precicely, is there anyone on here that could explain the intricacies of these kind of trades?
- em-bee 6y ago
  i understood that he bought at -3.7, and then it dropped further to -37.
- tedunangst 6y ago
  He sold it (or IB did so on his behalf at end of day) at -$37.
Stierlitz 6y ago
> Crude was actually around negative $3.70 a barrel when Shah’s screen had it at 1 cent A bit of a design flaw, I wonder did they spend as much money on the software as Equifax did? if (-3.70 != 1) { send an alert; }
huy-nguyen 6y ago
Another entry should be added to the “things programmers believe about commodity prices.”
kragen 6y ago
I've written real-time trading systems trading on IB. Their client library is a real pile of shit, so it's not surprising to hear that their internal systems are too. Reassuring to hear that they're going to eat the US$100M loss themselves instead of letting their customers have it. (Not that they were going to get US$9M out of this Shah guy anyway.) They never did us any wrong when we were their customers. Does this pose a risk of IB going insolvent? If they do, is there a risk of their customers being just another creditor of a bankrupt corporation, with respect to the stocks and futures that IB holds on behalf of those customers? Or are those instruments held in bailment, or actually by some other company, rather than as IB assets?

204 points by wclax04 6y ago

Incorrectly assuming values can never be negative is an all-too-common occurrence in trading and financial software. In 2012 Swedish stock futures trading was suspended for a time because their matching engine used an unsigned type for order quantities and someone submitted an order with a negative value which wrapped around to 4 billion: https://www.reuters.com/article/markets-sweden-bug/swedish-stock-futures-market-problem-caused-by-mega-order-glitch-idUSL5E8MTB4I20121129 https://www.reuters.com/article/markets-sweden-bug/swedish-s... Interactive Brokers' software is usually very solid. I'm surprised they weren't ready to handle this, the possibility of oil going negative had been discussed for some time before it happened.

H8crilA 6y ago
I particularly dislike the attitude of the CEO which shows he either doesn't know or pretends that he doesn't know how such contracts work. Please read the contract specs and educate yourself a little, Mr. Peterffy, they're public and free. > Peterffy said there’s a problem with how exchanges design their contracts because the trading dries up as they near expiration. The May oil futures contract -- the one that went negative -- expired the day after the historic plunge, so most of the market had moved to trading the June contract, which expires May 19 and currently trades around $24 a barrel. > “That’s how it’s possible for these contracts to go absolutely crazy and close at a price that has no economic justification,” Peterffy said. “The issue is whose responsibility is this?” Nobody ever promised neither liquidity nor positivity of prices, it is the fault of the brokerage, plain and simple. Thankfully $100M is something that IBKR can take on their books (they have $3B of cash according to the latest filling).
- AndrewBissell 6y ago
  Yes this is a very CYA statement. IB has been in this business for a long time, Peterffy knows very well that expiring futures contracts can experience all kinds of liquidity problems and large swings in price.
  - azinman2 6y ago
    But if it’s never happened before, and you’re some random programmer making this stuff, I’m not surprised there are all kinds of assumptions being made when choosing data types and validating inputs...
- topynate 6y ago
  "No economic justification"? That has to be faux-naïf. Yes, I was shocked at first to hear of prices turning negative. For about thirty seconds. Once I learned that the oil had to be delivered somewhere without enough storage, and under conditions of extreme low demand, it was perfectly obvious that you'd end up paying someone else to take the oil off your hands. Either Peterffy is too stupid to understand mass-media reports on his own field of business – not likely – or he's disingenuous, which says something about how he sees his clients.
- scurvy 6y ago
  Peterffy is a long time industry vet. He brought a lot of tech to trading in the 80's. He knows very well how things work.
bobcostas55 6y ago
>Interactive Brokers' software is usually very solid. That is certainly not my experience. Endless bugs in TWS over the years, and the support people are unbelievably rude. Every time I try to report a bug they start out by blaming me, it usually takes 2-3 back and forth rounds until they admit it's actually broken and tell me they'll forward the issue to the tech people. After that it's radio silence and you never know if they'll actually fix it or not. And let's not get into the disaster that is their API...
- AndrewBissell 6y ago
  I'll admit I haven't worked with their API or done anything too fancy with it. My experience is all relatively infrequent click trading, chart generation, and such.
- brobinson 6y ago
  Completely agree. Their mobile app, TWS, API, and websites are all a nightmare. I left them as soon as I stopped trading on margin (the only really compelling reason to use them for a retail trader/investor).
isoos 6y ago
> Interactive Brokers' software is usually very solid. Except their web portal. 40-50% of the time it is unable to load my portfolio data (even without the current market value, just the number of stocks and cash balances). At the same time I log into the mobile app, it forces a logout on the web app (why?) and it is able to load the balances and portfolio. No explanation. Same network, no adblocker or other browser plugin.
- scurvy 6y ago
  The single session thing is annoying. I'm guessing it is related to their various data provider regulations. The large brokers can get around this because, well, you are not really seeing pool depth and the like.
adrr 6y ago
I'm going to guess there's two teams involved on the IB side of things. There's financial engine team which processes and handles trades. And there is the UI team that displays the data and allows people to create trades. I bet the engine handled these fine and issue was that the UI team had data validation checks to prevent negative values.
- analyticstime 6y ago
  Nope. It was not possible to trade at negative.

BubRoss 6y ago

As usual, this nonsense headline is clickbait for the truth - some traders couldn't trade when oil went below zero.

muyth 6y ago

Would any of the traders mentioned that owe all that money to Interactive Broker's have to pay the total amount? Or is that part of Interactive Brokers' loss claim.

whatok 6y ago
From the story: > Customers will be made whole, Peterffy said. “We will rebate from our own funds to our customers who were locked in with a long position during the time the price was negative any losses they suffered below zero.” I'd imagine this is so they don't get sued and have some shoddy code pop up during trial.
- vmception 6y ago
  or they're raising capital and the end justifies the means same result of avoiding litigation, but even fewer consequences

changoplatanero 6y ago

non paywall link https://business.financialpost.com/pmn/business-pmn/oil-crash-busted-a-brokers-computers-and-inflicted-huge-losses https://business.financialpost.com/pmn/business-pmn/oil-cras...

elliekelly 6y ago

> Peterffy said there’s a problem with how exchanges design their contracts because the trading dries up as they near expiration. The May oil futures contract -- the one that went negative -- expired the day after the historic plunge, so most of the market had moved to trading the June contract, which expires May 19 and currently trades around $24 a barrel. > “That’s how it’s possible for these contracts to go absolutely crazy and close at a price that has no economic justification,” Peterffy said. “The issue is whose responsibility is this?” It’s pretty well known that commodity futures contracts are a game of hot potato for most investors as the expiration date approaches. But the Interactive Brokers CEO doesn’t offer an alternative solution. How would the contracts be structured instead that would avoid this? I don’t see how it would be possible.

wbl 6y ago
Financial settlement against a spot market in a large port.
- curiousllama 6y ago
  Cash settled contracts exist. These people just chose not to buy them. The question is how to structure physically settled contracts. After all, oil needs to get delivered to someone at some point.
  - AnimalMuppet 6y ago
    If you get into the market for physically settled contracts with no intention of taking delivery, then you're almost certainly a speculator. I'm not sure that it's the market's job to make that safer for you. I am not justifying inaccurate pricing. Burning speculators is fine, but give everyone accurate information.
    - vardump 6y ago
      Or possibly someone who needs large amounts of oil and want to hedge against fluctuations? Freight industry, airliner, etc.

SrslyJosh 6y ago

I am playing my second tiniest violin for them right now.

joncrane 6y ago
Are you saving the tiniest one for when <insert noted eschewer of wearing masks in public> comes down with COVID-19? (I'm curious who the tiniest one is for)
- mring33621 6y ago
  tiny violins fit well in tiny hands

0x8BADF00D 6y ago

It's really odd that this bug occured, as IB has no issue pricing credit spreads with negative values. Must be an issue specific to commodities futures contracts. I wonder what data types they were using.

derriz 6y ago
I’ve written code (a year or so ago) against the IB API and it seemed clear to me that the API was a thinish skin over multiple backend systems. The feed you get for products from different markets (even different futures markets) was different - the population of fields in price and trade feed was wildly inconsistent. I’m guessing each market is accessed in by a different IB system. Negative prices are a feature of some massively traded futures - interest rate futures for example - but can effectively never occur for index futures or the like.
- centimeter 6y ago
  From what I've seen, almost all financial companies developed a bunch of systems for different security types (which all have different rules and edge cases) independently, and only tried to tie them together as time went on. It's a recipe for a lot of confusion and inconsistency. There are tons of opportunities in finance to make short-sighted proclamations like "the number of futures in this kind of contract is always 100" or "this type of security can't go negative", and have it be true at the time, but false 5 years later when they add a new type of contract.
  - 0x8BADF00D 6y ago
    Kind of orthogonal, but I’ve often faced pushback from Product Managers when trying to future proof a feature that I’m going to ship. The pushback is usually “oh, there’s no way our customers will ever need that!”. Only until many years later will it blow up in your face, with costly consequences.
    - centimeter 6y ago

ralph84 6y ago

Would be nice to have more details on exactly what happened with the trades. Did the trades clear? Did IB liquidate the contracts before expiration? Was someone on the hook for taking physical delivery?

ashtonkem 6y ago
I feel like any sane clearing house should’ve rejected these trades.

ninetyfurr 6y ago

Oh a guy with $77,000 in his day trading account didn't know futures can go negative. I'm shocked. Ironically this is the same guy who will sell you a gym contract without a cancellation option and blame you for not doing your research when you owe him $1100.

socrates1998 6y ago

If you don't know what is going on, then why on earth would you risk so much money? IB fucked up, no doubt, but these idiots are trading shit they know nothing about. Don't trade on margin.

SilasX 6y ago
Well, yeah, it's reasonable to expect investors to appropriately researching something before buying, and be cognizant of the risk that it could crash. It's not reasonable for them to expect to deal with a platform that misrepresents the state of the market and executes trades at a non-market price, as was happening here. (It was telling them the oil futures still had a positive price when it was negative, and making them pay on that basis.)
- james-mcelwain 6y ago
  > Well, yeah, it's reasonable to expect investors to appropriately researching something before buying, and be cognizant of the risk that it could crash. Is it? Isn't this the entire reason risk management departments exist?
codenesium 6y ago
I don't think they knew in this case they were potentially trading on margin. The broker didn't know either which is crazy.
- whatok 6y ago
  If anyone is trading futures and did not know they were trading on margin or trading highly levered instruments then they either are trading on a platform that has zero compliance or they misrepresented themselves as an investor.
  - CrazyStat 6y ago
    The broker calculated the margin requirements based on the assumption that the price could not go negative. So the investors thought they were risking $30 per contract when it was actually a couple orders of magnitude larger than that. Effectively the broker lied to them about how leveraged they were.

codenesium 6y ago

Wasn't the original purpose of futures to let farmers and others lock in prices early so they can mitigate risk? Speculation on futures seems dumb if you have no intention of taking delivery.

ashtonkem 6y ago
Taking physical delivery and hedging are not one and the same. It’s entirely possible to use a cash settled future to hedge against market movements; the farmer sells at a steep loss, but their cash settled wheat futures offset a large percentage of the loss on a cash basis. The distinction you’re looking for here is those who are speculating on market prices, vs. those who are hedging against market prices. If you use or sell oil in large amounts, it makes sense to use futures to stabilize your downside risk, even if those futures are cash settled. That being said, I think that cash settled futures make purely speculative trading much easier, so you’d have a good point if that’s what you were heading towards.
- codenesium 6y ago
  That is the point I was trying to make. Basically if you're in the business of producing or buying and selling the commodity the futures are for you. If you're just speculating how does that help anybody? I guess you could make the argument that having more eyes on the market means there is more information so the price is a better reflection of the true value.
  - PeterisP 6y ago
    You may hedge your risk of prices of a different but related product. E.g. a bread manufacturer wants stability on their cost of goods, so they may buy wheat futures even though they don't ever want to receive direct shipment of wheat, they're dealing with specific regional flour suppliers but the wheat futures at a major location are a good proxy for that price.
  - whatok 6y ago
    If you only had farmers and individuals purchasing their goods allowed to trade futures, there is direct incentive for either party to manipulate the physical market through their actions, in ways that they only could, that wouldn't necessarily make any economic sense and could have spillover effects into the real economy. By having a more open market for futures, you theoretically have a more efficient market for all participants which you mention in your last sentence.

flint 6y ago

I'm surprised IB let speculators trade in a contract going to delivery. I worked as a risk manager in a commodity trading firm and only hedgers qualified to take delivery were permitted to hold contracts going to delivery.

everlost 6y ago
Wasn't this a cash settled contract?
- whatok 6y ago
  No.
  - jdadj 6y ago
    It's not a cash settled contract. "Delivery shall be made free-on-board ("F.O.B.") at any pipeline or storage facility in Cushing, Oklahoma with pipeline access to Enterprise, Cushing storage or Enbridge, Cushing storage." Source: https://www.cmegroup.com/trading/energy/crude-oil/light-sweet-crude_contract_specifications.html https://www.cmegroup.com/trading/energy/crude-oil/light-swee...
  - mathgenius 6y ago
    It may have been ICE's crude contract, which is cash settled: https://www.theice.com/products/213/WTI-Crude-Futures https://www.theice.com/products/213/WTI-Crude-Futures "The West Texas Intermediate Light Sweet Crude Oil futures contract is cash settled against the prevailing market price for US light sweet crude."
    - whatok 6y ago
      Sure, but a majority of trading was done on CME futures. In either case, you still ran into the issue of negative prices. USO only recently started explicitly stating that they would also potentially invest in ICE futures alongside CME futures.
- 6y ago

dilandau 6y ago

More to the point is that a bunch of dumbasses were speculating on oil futures without understanding the nature of the market, and no ability to take delivery on the oil in any event. Nobody should be holding those when they're so close to expiry unless they know what they're doing.

thanatropism 6y ago

About two months ago I delivered to a private client a (monthly-basis) prediction model for ethanol prices that included Brent -- in logarithm -- as a predictor. I know the monthly spot average of that statistic (fetched from FRED, the Fed of St. Louis system) won't turn negative, but still...

supernova87a 6y ago

For futures on physical deliverable objects (well, I guess most futures are such, but anyway) -- would volatility and speculation be dampened/improved if the clearinghouse forced everyone (or the seller) participating in a trade to certify that they had rights to the specific thing being traded? Could actually produce the contract -- like the oil producer is certified to have <xyz> barrels allowed to be sold? My notion is that if much of the trading (and it can be shown by futures volumes) cannot possibly be on actual physically deliverable quantities, then most must be "speculation" by people who cannot actually produce the asset. Would this be a help to stabilize the market? I know it all has to get settled in the end by the expiration date, but just an idea.

kasey_junk 6y ago
It likely will settle the markets by driving spreads through the roof and costing buyers/sellers of the physical items. The underlying theme here is that speculators are not providing utility to the market, which is wrong. They are bridging time and risk.
lazyjones 6y ago
> then most must be "speculation" by people who cannot actually produce the asset. Let's just call it what it is: gambling.
Rexxar 6y ago
Some companies can use this as a proxy to protect themself against variation of a product they need. For example an airline can use this to protect itself against price variation of kerosene once they have sold a ticket. As kerosene is not directly available on commodity market they use this future because their price are strongly correlated to kerosene price.

mathgenius 6y ago

If you look at the CME website using the wayback machine, you can clearly see a hi and low limit for the price. There is no high limit, but the low limit is set at "0.01". [1] This is kind of weird, because the snapshot is pulling current market prices, and still showing an incorrect hi/lo limit. It looks like they have since removed the hi/lo display: [2]. Someone should take a screenshot of this. [1] https://web.archive.org/web/20200117115242/https://www.cmegroup.com/trading/energy/crude-oil/light-sweet-crude.html https://web.archive.org/web/20200117115242/https://www.cmegr... [2] https://www.cmegroup.com/trading/energy/crude-oil/light-sweet-crude.html https://www.cmegroup.com/trading/energy/crude-oil/light-swee...

LatteLazy 6y ago

Fuck IB and this “trader”, idiots should lose their shirts, that’s one of the intended outcomes of an efficient market. That said, there is an issue here with futures contracts: you can get very very large leverage when the price is near zero. This is the real issue with instruments that can negative price and just like their are “circuit breakers” in markets for big price swings, there should be breakers for entering the “near zero” range.

mostlymeme 6y ago
An efficient market that don't let traders operate? Did you even read the article? Futures contracts that CAN BECOME NEGATIVE don't let large leverage when price is near zero, that's NOT TRUE. Future contracts margin is calculated with SPAN, and if it's done correctly, it considers the scenarios where price can go below 0.
- LatteLazy 6y ago
  >Futures contracts that CAN BECOME NEGATIVE don't let large leverage when price is near zero, that's NOT TRUE It clearly is. If I can buy a contract for 1c, I can get 100,000 contracts for 1000usd. Then if the price rises of falls by 1usd, I'm up/down 100,000 dollars. Can you think of any retail product with that sort of leverage? That's the danger of putting zero in a denominator.
  - analyticstime 6y ago
    You can't buy a contract for 1c. You need to pay the margin, and the margin for futures is not calculated based on current price, it's calculated with SPAN, that considers different scenarios in which you can lose money. For example, for this contract Bloomber says IB asked for $30 margin. But the margin is usually $7000 for this contract, that it was IB should have requested as collateral at least for each contract. in a day with that volatility should be much higher in IB, as they take that also in consideration, probably around $20000 per contract. The problem was that IB didn't consider scenarios in which the price can go below 0. The software was designed in that way. But it shouldn't.

haltingproblem 6y ago

“Five days, including the weekend, with the coronavirus going on and a complex system where we have to make many changes, was not a sufficient amount of time,” he said. “The idea we could have bugs is not, in my mind, a surprise.” He also acknowledged the error in the margin model Interactive Brokers used that day.....We have called the CFTC and complained bitterly,” Peterffy said. “It appears the exchanges are going scot-free.” Thomas Peterffy must think we are idiots. Anyone who trades commodity contracts for any period of time knows that the real cost of the contract is the actual cost of the commodity - storage costs. When storage costs spike and the actually commodity spot costs go down, the future will become negative! One way to get a handle on storage costs is think of them being inversely proportional to the value density. The higher the value density, e.g. gold the less the storage costs. Oil is not so dense so storage costs matter. Financial instruments like the Treasury Bonds and the S&P futures contract have zero storage costs. Storage cost is of-course different than carry cost (the cost of funding your long position). On another aside, I have known folks who have worked at IB in the past, and their systems absolutely suck dead goats. Huge masses of legacy C++ code with poor testing. Most of these brokerage firms have legacy code base from the 90s that is poorly understood. They also have nonexistent organizational quotient around code validation, correctness and testing their risk models. A futures margin model is not something one can whip up over a weekend but a good CS undergraduate can program one over a couple months. Sorry for the IB customers but I have zero sympathy for IB or should I say negative ;)

hellofunk 6y ago
Agreed. IB's data is also not so accurate much of the time; the way it reports volume in particular is misleading. I've seen traders abandon IB after big losses because their strategies required actual live volume data that didn't suck, and nearly any other broker provides this.
- ajflores1604 6y ago
  What are some recommended alternative platforms that provide api access?

nromiun 6y ago

> Its software couldn’t cope with that pesky minus sign, even though it was always technically possible -- though this was an outlandish idea before the pandemic -- for the crude market to go upside down. Wow, just wow. They are handling millions (billions?) of dollars every day and couldn't find the time to test that they can just DISPLAY a minus sign. That's insane. And it's not even that outlandish. People were saying it could go into the negative weeks before it happened. This just seems like pure laziness. Just pretend everything is business as usual.

EdwardDiego 6y ago
Makes me wonder about their testing - any decent tester is going to throw a negative number in as a matter of course. I have a gut feeling that this bug was flagged and closed as WONTDO because "that'll never happen"
- dmurray 6y ago
  Just as likely it was a feature. If you really think "that'll never happen", then the right thing to do is explicitly ignore negative prices or refuse to send orders at negative prices, and that's exactly the kind of failsafe I'd want to have in my trading software, since you can get spurious prices for all kinds of reasons. It turned out the assumption was wrong and yeah, you should remove the logic handling that once it's evident futures may go negative, and you should have a process capable of making that change with only a day or two notice. But being robust the rest of the time at the expense of mishandling a once-in-fifty-years event is not in itself a bug. If you disagree, should a trading system also allow negative prices for precious metals futures? Stocks? Currencies? Options? Bonds? Futures on stocks or bonds? I can contemplate all of those trading negative in extraordinary, contrived scenarios but I would design systems today not to trade them at negative prices.
  - EdwardDiego 6y ago
    But the fact that they didn't display negative prices to customers indicates that they were explicitly ignoring them in a very very bad way. I'm suspecting an abs() function was involved.

danans 6y ago

Hopefully, one of the things that we price in going forward is the volatility in the price of oil vs alternatives, especially those that can be produced domestically in a way decoupled from international events. That includes natural gas and especially renewables. Remember that just a 6 years ago, oil was north of $100/barrel [1], and recently it's close to zero. Wind on the other hand has had a steadily reducing LCOE[2][3]. Solar's LCOE is also steadily reducing [3]. Volatility has a cost. With oil, it's one that the US and other countries hae historically tried to dampen with various industrial and political methods (the national strategic oil reserve, military/political "influence" on foreign oil producers, subsidies for domestic production), but seems like the current situation is beyond those methods' ability to control. 1. https://www.macrotrends.net/1369/crude-oil-price-history-chart https://www.macrotrends.net/1369/crude-oil-price-history-cha... 2. https://www.energy.gov/sites/prod/files/2015/08/f25/LCOE.pdf https://www.energy.gov/sites/prod/files/2015/08/f25/LCOE.pdf 3.https://en.wikipedia.org/wiki/Cost_of_electricity_by_source#/media/File:US_projected_cost_of_wind_power.png https://en.wikipedia.org/wiki/Cost_of_electricity_by_source#... 4. https://en.wikipedia.org/wiki/Cost_of_electricity_by_source#/media/File:EU-PV-LCOE-Projection.png https://en.wikipedia.org/wiki/Cost_of_electricity_by_source#...

neonate 6y ago

https://archive.md/EGIGB https://archive.md/EGIGB

raviolo 6y ago

Comments about storage costs are nice theory but don’t tell me those storage costs went up 50 dollars a barrel to cause CLK0 to go from +10 to -40 within something like 30 minutes. It was all forced liquidation by brokers like IB and people absolutely bamboozled by negative prices puking their positions. Nothing to do with costs for actual storage.

unixhero 6y ago

I would recommend http://www.saxobank.com http://www.saxobank.com , it's a kick ass broker.

tester89 6y ago

I don’t really understand why the traders would end up owing money. If they thought they were paying 0,01 $/bbl but we’re actually “paying” -37 $/bbl wouldn’t IB owe them?

Stierlitz 6y ago
@tester89 > I don’t really understand why the traders would end up owing money. If they thought they were paying 0,01 $/bbl but we’re actually “paying” -37 $/bbl wouldn’t IB owe them? My thoughts precicely, is there anyone on here that could explain the intricacies of these kind of trades?
em-bee 6y ago
i understood that he bought at -3.7, and then it dropped further to -37.
tedunangst 6y ago
He sold it (or IB did so on his behalf at end of day) at -$37.

Stierlitz 6y ago

> Crude was actually around negative $3.70 a barrel when Shah’s screen had it at 1 cent A bit of a design flaw, I wonder did they spend as much money on the software as Equifax did? if (-3.70 != 1) { send an alert; }

huy-nguyen 6y ago

Another entry should be added to the “things programmers believe about commodity prices.”

kragen 6y ago

I've written real-time trading systems trading on IB. Their client library is a real pile of shit, so it's not surprising to hear that their internal systems are too. Reassuring to hear that they're going to eat the US$100M loss themselves instead of letting their customers have it. (Not that they were going to get US$9M out of this Shah guy anyway.) They never did us any wrong when we were their customers. Does this pose a risk of IB going insolvent? If they do, is there a risk of their customers being just another creditor of a bankrupt corporation, with respect to the stocks and futures that IB holds on behalf of those customers? Or are those instruments held in bailment, or actually by some other company, rather than as IB assets?