5 ms·

"A data device critical to the Tokyo Stock Exchange’s trading system had malfunctioned, and the automatic backup had failed to kick in. It was less than an hour

"A data device critical to the Tokyo Stock Exchange’s trading system had malfunctioned, and the automatic backup had failed to kick in. It was less than an hour before the system, called Arrowhead, was due to start processing orders in the $6 trillion equity market. Exchange officials could see no solution." You know, just from a human perspective, talk about a bad day. It's like seeing the cruise liner heading for the port too fast, knowing it is going to crash and cause immense damage, and realizing there is absolutely nothing that can now be done to prevent the damage. Except, in this case, it's just way worse.

tuatoru 6y ago
How many people died? How many were hospitalized?
- gruez 6y ago
  Are you talking about the cruise ship crashing? It's a hypothetical. Nobody was hurt.
- stanrivers 6y ago
  Fair point! I was talking about monetary damages - excluding any kind of loss of life or personal injury. I could have picked a better metaphor!
  - Scoundreller 6y ago
    I question the monetary losses too. How much was lost by being unable to sell something today that you weren’t willing to sell yesterday?
    - outworlder 6y ago
      > How much was lost by being unable to sell something today that you weren’t willing to sell yesterday? Could be a lot. Markets are not disconnected. If only Japan existed, sure. Some orders would no longer exist, but nothing major, because noone else would be trading anyway. But suppose this was during an economic downturn. You are trying to SELL SELL SELL because all countries worldwide are feeling some economic pressure and you can't because the exchange is down. The next day, whatever assets you had are now worth a fraction of what they were one day before. Oh, and there are some financial instruments that expire.
      - Scoundreller 6y ago
        > You are trying to SELL SELL SELL because all countries worldwide are feeling some economic pressure and you can't because the exchange is down. Some exchanges deliberately close in those circumstances. Or when big news is about to be released. But as much as people were unable to sell, there are nearly as many situations where being unable to sell is a good thing. I guess we’re eventually going to argue for 24/7 stock exchanges and for some reason very few are. My own experience in Canada is that a lot of big Canadian stocks also trade in NY, and when one market is closed and the other isn’t because of different holidays, not a lot happens. I guess an unexpected crash is different, but my guess is that everyone takes the day off, avoids releasing any big news out of respect for the situation and gets back to it tomorrow.
        MetalGuru 6y ago
        I’ve always wondered why markets haven’t switched to being open 24/7. Seems like having strict open hours is a relic of their pre-tech days.
        andylynch 6y ago
        Except for FX, there is no need for 24/7 and plenty of drawbacks; both in terms of overhaed and execution quality. In fact the trend is in the opposite direction, towards shorter trading hours as well as more turnover in the opening and closing auctions. Shorter hours especially would be a huge win for work-life balance and diversity in the industry, and just as importly for reducing costs. Having the markets open longer only spreads liquidity thinner across the day, and moreover having announcements made, corporate actions processed, etc etc is better done when the market is closed so everyone can be on the same page when the happen.
        ycombobreaker 6y ago
        No need to pull 24/7 uptime into the conversation, I would say. The impact is that the exchange was not open when they were expected to be open. An outage like that is more disruptive than scheduled "downtime" because of being unplanned.
    - stanrivers 6y ago
      A large industry is setup, with how many [thousands] of people employed and expected to execute on their jobs related to this market, and they just lost a whole day of doing their job. So you have the opportunity costs for those people being paid to do nothing for a day and generating nothing in the form of revenue. Rightly or wrongly, a significant amount of profit generated in the stock market is from short term moves / games, not from buying and holding for long periods. Second, and I don't know if this is the case, but what about companies that were issuing equities to raise cash on that day? They had geared up entire investment banking teams, with customers orders lined up, and had a whole game plan for executing the offerings. That all went up in smoke and they will have to re-prepare everything for another day. That's true to IPOs, general follow-on offerings, to some extent ATM programs, etc. Those are a couple ideas, maybe there are more?
      - Scoundreller 6y ago
        Oh, my biggest worry was futures expiring, but I hope nobody thought to switch over to a new system on the same day, and same with an IPO/2ndary offering.
      - stainforth 6y ago
        Still seems extremely minor. Just wait. Every other player had to as well.
    - hnracer 6y ago
      The exchange lost a lot of revenue. Trader's and investor's time was wasted, you can't just change the open time without screwing up a lot of systems. Risk management would've been an issue too, a lot of people crossing the spread to derisk and transferring wealth to market makers.
      - throwaway0a5e 6y ago
        Opening late won't break much. Most systems will be like "well shucks there's no trades to process, better just sit here and wait". Opening early will break everything.
    - juiyout 6y ago
      Stock exchanges earn major share of their profit from processing the orders. They just lost at least a whole day of revenue. After compiling some number for their annual report[1], I would say the loss from TSE revenue alone is roughly 2 million USD. But considering the effect trickling down the revenue stream where other security partners whose revenue depends on earning transaction fees, I would say the real damage would be many folds of that 2 million loss from TSE. [1]: https://www.jpx.co.jp/english/corporate/investor-relations/ir-library/annual-reports/tvdivq0000008t9q-att/Financial_and_Corporate_Data.pdf https://www.jpx.co.jp/english/corporate/investor-relations/i...
      - MauranKilom 6y ago
        More or less by definition, this won't impact their yearly bottom line by more than 0.3% (at least to within first order). Which is quite a bit, but it still seems super minor. Just like 2 million. Isn't that peanuts for a stock exchange?
        juiyout 6y ago
        Agreed that 2 million USD is not much compared to total revenue. However, you see, TSE YoY growth from 2017 to 2018 is roughly 4 million USD. Growth is hard as it is for TSE, leaving money on the table like this time definitely hurts.
- centimeter 6y ago
  A large number of people suffered a small amount of efficiency loss. The number of utils or QALY or whatever metric you want lost was probably fairly substantial. The line of reasoning “well, no one died” is a classic example of humans’ unfortunate tendency to round small numbers to zero in utilitarian calculations, which leads to all sorts of bad decisions when you’re dealing with things that effect lots of people in small ways.
  - oconnor663 6y ago
    The opposite end of the spectrum causes problems too: "It's ok for some people to die, if it results in a small amount of good for a large number of people." One way to look at the problem with that in practice is that small amounts of good and large numbers of people both tend to involve a lot of uncertainty. When we're allowed to do motivated reasoning with enough uncertainty, we can usually prove whatever we want, and that sort of simple utilitarianism ends up being too easy to abuse. I think we can look at cultural taboos as a practical solution to this problem: "Ok, clearly there's no reliable way for us to say what killing is good and what killing is bad, and we all agree that most of it is bad, so let's make the simplifying assumption that all killing is bad."
  - Scoundreller 6y ago
    > A large number of people suffered a small amount of efficiency loss. The number of utils or QALY or whatever metric you want lost was probably fairly substantial. If the market closing is so bad for its participants, why aren’t they lobbying to keep it open past 4PM? Or on Saturdays?
    - centimeter 6y ago
      We can use depository receipts and similar strategies to keep trading 24/7. But the markets would probably be a lot more efficient if they were open all the time. That’s not to say that we (traders) would necessarily make much more money off them that way. We’d probably make less, actually, because it would simplify/remove the kind of arbitrage market opens/closes let us do.
    - 6y ago
- _alex_ 6y ago
  Loss of life or bodily harm aren’t required for someone to have a bad day.

tuatoru 6y ago

How many people died? How many were hospitalized?

gruez 6y ago
Are you talking about the cruise ship crashing? It's a hypothetical. Nobody was hurt.
stanrivers 6y ago
Fair point! I was talking about monetary damages - excluding any kind of loss of life or personal injury. I could have picked a better metaphor!
- Scoundreller 6y ago
  I question the monetary losses too. How much was lost by being unable to sell something today that you weren’t willing to sell yesterday?
  - outworlder 6y ago
    > How much was lost by being unable to sell something today that you weren’t willing to sell yesterday? Could be a lot. Markets are not disconnected. If only Japan existed, sure. Some orders would no longer exist, but nothing major, because noone else would be trading anyway. But suppose this was during an economic downturn. You are trying to SELL SELL SELL because all countries worldwide are feeling some economic pressure and you can't because the exchange is down. The next day, whatever assets you had are now worth a fraction of what they were one day before. Oh, and there are some financial instruments that expire.
    - Scoundreller 6y ago
      > You are trying to SELL SELL SELL because all countries worldwide are feeling some economic pressure and you can't because the exchange is down. Some exchanges deliberately close in those circumstances. Or when big news is about to be released. But as much as people were unable to sell, there are nearly as many situations where being unable to sell is a good thing. I guess we’re eventually going to argue for 24/7 stock exchanges and for some reason very few are. My own experience in Canada is that a lot of big Canadian stocks also trade in NY, and when one market is closed and the other isn’t because of different holidays, not a lot happens. I guess an unexpected crash is different, but my guess is that everyone takes the day off, avoids releasing any big news out of respect for the situation and gets back to it tomorrow.
      - MetalGuru 6y ago
        I’ve always wondered why markets haven’t switched to being open 24/7. Seems like having strict open hours is a relic of their pre-tech days.
        andylynch 6y ago
        Except for FX, there is no need for 24/7 and plenty of drawbacks; both in terms of overhaed and execution quality. In fact the trend is in the opposite direction, towards shorter trading hours as well as more turnover in the opening and closing auctions. Shorter hours especially would be a huge win for work-life balance and diversity in the industry, and just as importly for reducing costs. Having the markets open longer only spreads liquidity thinner across the day, and moreover having announcements made, corporate actions processed, etc etc is better done when the market is closed so everyone can be on the same page when the happen.
      - ycombobreaker 6y ago
        No need to pull 24/7 uptime into the conversation, I would say. The impact is that the exchange was not open when they were expected to be open. An outage like that is more disruptive than scheduled "downtime" because of being unplanned.
  - stanrivers 6y ago
    A large industry is setup, with how many [thousands] of people employed and expected to execute on their jobs related to this market, and they just lost a whole day of doing their job. So you have the opportunity costs for those people being paid to do nothing for a day and generating nothing in the form of revenue. Rightly or wrongly, a significant amount of profit generated in the stock market is from short term moves / games, not from buying and holding for long periods. Second, and I don't know if this is the case, but what about companies that were issuing equities to raise cash on that day? They had geared up entire investment banking teams, with customers orders lined up, and had a whole game plan for executing the offerings. That all went up in smoke and they will have to re-prepare everything for another day. That's true to IPOs, general follow-on offerings, to some extent ATM programs, etc. Those are a couple ideas, maybe there are more?
    - Scoundreller 6y ago
      Oh, my biggest worry was futures expiring, but I hope nobody thought to switch over to a new system on the same day, and same with an IPO/2ndary offering.
    - stainforth 6y ago
      Still seems extremely minor. Just wait. Every other player had to as well.
  - hnracer 6y ago
    The exchange lost a lot of revenue. Trader's and investor's time was wasted, you can't just change the open time without screwing up a lot of systems. Risk management would've been an issue too, a lot of people crossing the spread to derisk and transferring wealth to market makers.
    - throwaway0a5e 6y ago
      Opening late won't break much. Most systems will be like "well shucks there's no trades to process, better just sit here and wait". Opening early will break everything.
  - juiyout 6y ago
    Stock exchanges earn major share of their profit from processing the orders. They just lost at least a whole day of revenue. After compiling some number for their annual report[1], I would say the loss from TSE revenue alone is roughly 2 million USD. But considering the effect trickling down the revenue stream where other security partners whose revenue depends on earning transaction fees, I would say the real damage would be many folds of that 2 million loss from TSE. [1]: https://www.jpx.co.jp/english/corporate/investor-relations/ir-library/annual-reports/tvdivq0000008t9q-att/Financial_and_Corporate_Data.pdf https://www.jpx.co.jp/english/corporate/investor-relations/i...
    - MauranKilom 6y ago
      More or less by definition, this won't impact their yearly bottom line by more than 0.3% (at least to within first order). Which is quite a bit, but it still seems super minor. Just like 2 million. Isn't that peanuts for a stock exchange?
      - juiyout 6y ago
        Agreed that 2 million USD is not much compared to total revenue. However, you see, TSE YoY growth from 2017 to 2018 is roughly 4 million USD. Growth is hard as it is for TSE, leaving money on the table like this time definitely hurts.
centimeter 6y ago
A large number of people suffered a small amount of efficiency loss. The number of utils or QALY or whatever metric you want lost was probably fairly substantial. The line of reasoning “well, no one died” is a classic example of humans’ unfortunate tendency to round small numbers to zero in utilitarian calculations, which leads to all sorts of bad decisions when you’re dealing with things that effect lots of people in small ways.
- oconnor663 6y ago
  The opposite end of the spectrum causes problems too: "It's ok for some people to die, if it results in a small amount of good for a large number of people." One way to look at the problem with that in practice is that small amounts of good and large numbers of people both tend to involve a lot of uncertainty. When we're allowed to do motivated reasoning with enough uncertainty, we can usually prove whatever we want, and that sort of simple utilitarianism ends up being too easy to abuse. I think we can look at cultural taboos as a practical solution to this problem: "Ok, clearly there's no reliable way for us to say what killing is good and what killing is bad, and we all agree that most of it is bad, so let's make the simplifying assumption that all killing is bad."
- Scoundreller 6y ago
  > A large number of people suffered a small amount of efficiency loss. The number of utils or QALY or whatever metric you want lost was probably fairly substantial. If the market closing is so bad for its participants, why aren’t they lobbying to keep it open past 4PM? Or on Saturdays?
  - centimeter 6y ago
    We can use depository receipts and similar strategies to keep trading 24/7. But the markets would probably be a lot more efficient if they were open all the time. That’s not to say that we (traders) would necessarily make much more money off them that way. We’d probably make less, actually, because it would simplify/remove the kind of arbitrage market opens/closes let us do.
  - 6y ago
_alex_ 6y ago
Loss of life or bodily harm aren’t required for someone to have a bad day.