4 ms·

there’s a move now towards 24/7 trading. I guess we’ll see how the rigors of the trading environment mesh with zero down time. I’m sure the rollout will be

there’s a move now towards 24/7 trading. I guess we’ll see how the rigors of the trading environment mesh with zero down time. I’m sure the rollout will be slow and steady.

jedberg 5mo ago
I've seen that. I suspect the exchanges will never go for it for this exact reason -- they need downtime for maintenance. But if does go through, it will be a fun challenge to get 100% uptime! I've always said that with infinite money we could get 100% uptime, but no one has infinite money. Trading firms are about as close as I can imagine to infinite money though.
- gigatexal 5mo ago
  I hated my time as an SRE. But … can’t it be done with some combination of canaries and blue green deployments and extensive testing? Where when things look good you just swap all the traffic to the good stuff keeping the rollback hot etc etc?
  - jedberg 5mo ago
    That's how we got 99.99% at Netflix. And it cost a lot of money. But a canary implies that something may go wrong and you have to roll back. The canary is still production traffic, so some transactions would fail, which isn't allowed for this kind of workload. I image you'd have to use shadow execution, where you roll out a full second copy, run every transaction through both, and compare the results. And then, only after a certain time, switch traffic to the new infra and tear down the old. But you would need a ton of extra hardware (more than double) and a lot of ways to keep data in sync. And of course if you put an LLM or other non-deterministic system in there, that's a whole other can of worms. Like I said, a fun problem to solve. :)
    - gigatexal 5mo ago
      Folks that keep the lights on 24/7 aka SREs are super heroes that wear capes. Thank you for your service. I couldn’t do it. I like infra and all but it’s just not my cup of tea. Def true that in a trading pov the trade must be executed. It must settle. It must work. Or capital flight will be huge.
  - cgio 5mo ago
    There are different kinds of updates that influence options and feasibility. Keeping in mind that deep in the heart of an exchange is a single threaded process, the sequencer. Therefore, you have three layers, external facing protocols, sequencer/matching engines, and internal interfaces. Internal interfaces are the easiest for b/g. External protocols, any change worth its weight changes the protocol and therefore requires participants to change their codebases too. Versioning protocols is an option, but still the integration with consumers is much more transparent and usually you have them test on pre-prod environments, occasionally also requiring attestation and conformance testing (regulated markets). Sequencer and matching engine are at the core. You could do parallel runs but not b/g. Theoretically you could abstract the matching engine and keep a barebones sequencer immutable, but this will have performance implications. So yes, you can do things, but not in a completely transparent way, unless if you introduce an “upgrade jitter” to give you a window for transparent upgrades. It’s an interesting domain, I think people will just accept occasional downtimes as a better option than constant jitter cost.
- amluto 5mo ago
  An amusing, moderately expensive solution that might actually work would be to have a weekday system and a weekend system. Think of it as a spare D/R system that you intentionally swap twice a week :) If done right, it would be a complete separate system. Separate IP addresses and all.
  - nippoo 5mo ago
    That's effectively time-based request sharding which seems sensible but you'd still have to reconcile trades and any open positions (etc) across the time boundary where one system stops accepting requests and the other one starts. And keep the databases synchronous (ie have some system to make sure they're in sync at the changeover time) - or have a few minutes/hours of downtime between weekends and weekdays while you copy the whole production database from one system to another. The devil is in the details!
    - amluto 5mo ago
      For what it’s worth, in some financial markets, there is a sort of natural daily cutover time [0] across which you are often not trading quite the same instrument. For example, the settlement date may roll over, etc. And a lot of Very Serious Finance is already built on the idea that most parties do not instantaneously reconcile anything and don’t depend on real-time trade lists. I really can imagine a system in which the Monday trading system runs all day and then turns off at a predetermined time. Then it has 15 minutes to produce and disseminate a final list of all transactions, after which it becomes completely unavailable and is ready for maintenance. Any subsequent amendment to Monday’s trading would be done out of band. Open orders at the end of Monday do not carry over immediately to Tuesday, although front ends are welcome to recreate them. Everyone would understand that liquidity would be thin for the first few seconds after the system rolls over. For added fun, Monday and Tuesday could actually be allowed to overlap in a hypothetical trading system, although the market participants might not love this. [0] which is not the same for all instruments, and holidays mean that not every instrument rolls over meaningfully every day.
- skippyboxedhero 5mo ago
  How do you think on-chain exchanges do it? Hyperliquid has 16 employees, not engineers...total employees. It is possible, it isn't going to be possible for many of the legacy exchanges. I work with a major one and, being honest, from day one it was obvious they were incompetent. They employ a huge number of engineers and are unable to deliver basic features at any reasonable pace. Not even remotely close to it either (as in: you ask them to do something, they say yes, execs say yes, you get a deadline, date comes...deployment difficulties, environment not working, run around goes on and on forever). I remember the CEO got on a call with us at the start and was slapping himself on the back saying they had no downtime...because they were able to do maintenance when markets shut (and have heard very bad things about how that goes). But it is 24/7 world now, our service is up 24/7 and, of course, this led to massive issues in time due to the very different expectations around delivery/quality. Our execs were impressed, our engineers said this was a bad sign. And, ofc, it transpired that they were total amateurs (to be clear, this is one of the biggest exchanges in the world) and were unable to deliver. To come back to my original statement: there is a company of 16 people total who is, from the point of view of customers, delivering features faster. It is difficult to understate how insane that is.
  - jedberg 5mo ago
    From what I've seen, on-chain transaction times are measured in seconds and minutes, not milliseconds. It's a lot easier when you have time to wait to process a queue.
    - skippyboxedhero 5mo ago
      Fastest ones are processing a block every 10ms. It depends what you mean by easy. Even if you are using a slow chain, you still have to compete for finite block space, you still have to work out how to risk/matching fast, etc. With chains built for exchange use, operating them easier, that is why they don't require thousands of engineers. But the actual technical capability of the system is significantly in excess of tradfi exchanges. For example, risk function is real-time on-chain as opposed to EoD settlement. This significantly changes the possible feature set. Once you have built it, it is very easy...the question is why big exchanges rely so heavily on eod processes? The answer is: they are bad at engineering.
- justinclift 5mo ago
  Heh, maybe they'll develop a sudden interest in the old Vax VMS clustering approach? ;)
TacticalCoder 5mo ago
> there’s a move now towards 24/7 trading. Isn't the plan more like 23/5 like is already the case for several markets? I can't see the standard sessions moving more 9:30am/4pm weekdays to 24/7. I take it they'd still let, at least, one hour off for technical reasons. If I'm not mistaken it's the reason several markets are 23/5 and not 24/5: that one hour of downtime is basically for servers/maintenance right? (maybe someone can chime in) P.S: I take it technically there's 24/7 trading already seen that cryptocurrencies exchanges are opened 24/7 (I'm not sure: but I think that's the case) but I don't think those do anywhere near the volume of, say, options trading on equities during standard sessions (40 Gbit/s with peak over 70 Gbit/s for the full options feed).
- dmurray 5mo ago
  The 23/7 is not so much for maintenance as to have a defined window for changes to the market to happen. Every so often a new stock is listed or a stock ticker is changed or a stock is split, etc. There are smaller changes every single day, like to the settlement date of your trade. It's very convenient to be able to restart all your systems at 5pm, have them all load the updated reference data, and start them again in time for 6pm (or 7pm, or 4am tomorrow...). Even if you trade stocks and options and currencies and futures all over the world, a quirk of the calendar means they're basically all closed between 4 and 5pm Chicago time. Of course it's possible in principle to build systems where all this is dynamic and you can seamlessly trade with the old configuration at 4:59:59.999 and start trading the new one a millisecond later. But literally everyone has built systems that don't work on this, that rely on being able to chunk the continuous passage of time into discrete days. It would be painful to rearchitect them all now.
cgio 5mo ago
Only US. Other markets barely have liquidity during daytime and get most liquidity in opening and closing auctions. Maintenance periods are actually a complication. A few more state transitions for the system, but barely used for maintenance. The only value is for upgrades, which would still be scheduled with the market down and systems up, as participants also need to transition codebases for breaking changes, a test weekend or more is required etc. These systems are extremely resilient. You most often get an incident not because the system is down but because the latency profile has changed by a few ms.
- willtemperley 5mo ago
  Not really US only. LMAX is 24/7 and is a UK company, famous on HN for open sourcing their ring buffer. Crypto trading has been 24/7 since it began.
  - cgio 5mo ago
    Ok, I will make it mostly US. Maybe a couple more markets, London and Tokyo? Futures may be a bit broader adoption too. Vast majority won’t move to 24/7. Crypto is a different game for the time being at least. It has its own challenges but also escapes quite a few of traditional exchange complexities.
bostik 5mo ago
Sports betting exchanges have been doing that for a very long time. There is never a good time to take the system down for maintenance - event settlements happen every few minutes, and live games with in-play betting are going on somewhere in the world at any given time. Makes things damn hard indeed, because you have to truly learn asynchronicity, CQRS and complex live migrations. (Incidentally, engineers who have worked on such systems tend to be over-represented in extreme HA businesses.)
antonvs 5mo ago
Crypto exchanges have been doing 24/7 trading for well over a decade. Of course they’ve had their own, uh, issues - but generally, reliability of transactions hasn’t been a major one, for the big exchanges.

there’s a move now towards 24/7 trading. I guess we’ll see how the rigors of the trading environment mesh with zero down time. I’m sure the rollout will be slow and steady.

jedberg 5mo ago

I've seen that. I suspect the exchanges will never go for it for this exact reason -- they need downtime for maintenance. But if does go through, it will be a fun challenge to get 100% uptime! I've always said that with infinite money we could get 100% uptime, but no one has infinite money. Trading firms are about as close as I can imagine to infinite money though.

gigatexal 5mo ago
I hated my time as an SRE. But … can’t it be done with some combination of canaries and blue green deployments and extensive testing? Where when things look good you just swap all the traffic to the good stuff keeping the rollback hot etc etc?
- jedberg 5mo ago
  That's how we got 99.99% at Netflix. And it cost a lot of money. But a canary implies that something may go wrong and you have to roll back. The canary is still production traffic, so some transactions would fail, which isn't allowed for this kind of workload. I image you'd have to use shadow execution, where you roll out a full second copy, run every transaction through both, and compare the results. And then, only after a certain time, switch traffic to the new infra and tear down the old. But you would need a ton of extra hardware (more than double) and a lot of ways to keep data in sync. And of course if you put an LLM or other non-deterministic system in there, that's a whole other can of worms. Like I said, a fun problem to solve. :)
  - gigatexal 5mo ago
    Folks that keep the lights on 24/7 aka SREs are super heroes that wear capes. Thank you for your service. I couldn’t do it. I like infra and all but it’s just not my cup of tea. Def true that in a trading pov the trade must be executed. It must settle. It must work. Or capital flight will be huge.
- cgio 5mo ago
  There are different kinds of updates that influence options and feasibility. Keeping in mind that deep in the heart of an exchange is a single threaded process, the sequencer. Therefore, you have three layers, external facing protocols, sequencer/matching engines, and internal interfaces. Internal interfaces are the easiest for b/g. External protocols, any change worth its weight changes the protocol and therefore requires participants to change their codebases too. Versioning protocols is an option, but still the integration with consumers is much more transparent and usually you have them test on pre-prod environments, occasionally also requiring attestation and conformance testing (regulated markets). Sequencer and matching engine are at the core. You could do parallel runs but not b/g. Theoretically you could abstract the matching engine and keep a barebones sequencer immutable, but this will have performance implications. So yes, you can do things, but not in a completely transparent way, unless if you introduce an “upgrade jitter” to give you a window for transparent upgrades. It’s an interesting domain, I think people will just accept occasional downtimes as a better option than constant jitter cost.
amluto 5mo ago
An amusing, moderately expensive solution that might actually work would be to have a weekday system and a weekend system. Think of it as a spare D/R system that you intentionally swap twice a week :) If done right, it would be a complete separate system. Separate IP addresses and all.
- nippoo 5mo ago
  That's effectively time-based request sharding which seems sensible but you'd still have to reconcile trades and any open positions (etc) across the time boundary where one system stops accepting requests and the other one starts. And keep the databases synchronous (ie have some system to make sure they're in sync at the changeover time) - or have a few minutes/hours of downtime between weekends and weekdays while you copy the whole production database from one system to another. The devil is in the details!
  - amluto 5mo ago
    For what it’s worth, in some financial markets, there is a sort of natural daily cutover time [0] across which you are often not trading quite the same instrument. For example, the settlement date may roll over, etc. And a lot of Very Serious Finance is already built on the idea that most parties do not instantaneously reconcile anything and don’t depend on real-time trade lists. I really can imagine a system in which the Monday trading system runs all day and then turns off at a predetermined time. Then it has 15 minutes to produce and disseminate a final list of all transactions, after which it becomes completely unavailable and is ready for maintenance. Any subsequent amendment to Monday’s trading would be done out of band. Open orders at the end of Monday do not carry over immediately to Tuesday, although front ends are welcome to recreate them. Everyone would understand that liquidity would be thin for the first few seconds after the system rolls over. For added fun, Monday and Tuesday could actually be allowed to overlap in a hypothetical trading system, although the market participants might not love this. [0] which is not the same for all instruments, and holidays mean that not every instrument rolls over meaningfully every day.
skippyboxedhero 5mo ago
How do you think on-chain exchanges do it? Hyperliquid has 16 employees, not engineers...total employees. It is possible, it isn't going to be possible for many of the legacy exchanges. I work with a major one and, being honest, from day one it was obvious they were incompetent. They employ a huge number of engineers and are unable to deliver basic features at any reasonable pace. Not even remotely close to it either (as in: you ask them to do something, they say yes, execs say yes, you get a deadline, date comes...deployment difficulties, environment not working, run around goes on and on forever). I remember the CEO got on a call with us at the start and was slapping himself on the back saying they had no downtime...because they were able to do maintenance when markets shut (and have heard very bad things about how that goes). But it is 24/7 world now, our service is up 24/7 and, of course, this led to massive issues in time due to the very different expectations around delivery/quality. Our execs were impressed, our engineers said this was a bad sign. And, ofc, it transpired that they were total amateurs (to be clear, this is one of the biggest exchanges in the world) and were unable to deliver. To come back to my original statement: there is a company of 16 people total who is, from the point of view of customers, delivering features faster. It is difficult to understate how insane that is.
- jedberg 5mo ago
  From what I've seen, on-chain transaction times are measured in seconds and minutes, not milliseconds. It's a lot easier when you have time to wait to process a queue.
  - skippyboxedhero 5mo ago
    Fastest ones are processing a block every 10ms. It depends what you mean by easy. Even if you are using a slow chain, you still have to compete for finite block space, you still have to work out how to risk/matching fast, etc. With chains built for exchange use, operating them easier, that is why they don't require thousands of engineers. But the actual technical capability of the system is significantly in excess of tradfi exchanges. For example, risk function is real-time on-chain as opposed to EoD settlement. This significantly changes the possible feature set. Once you have built it, it is very easy...the question is why big exchanges rely so heavily on eod processes? The answer is: they are bad at engineering.
justinclift 5mo ago
Heh, maybe they'll develop a sudden interest in the old Vax VMS clustering approach? ;)

TacticalCoder 5mo ago

> there’s a move now towards 24/7 trading. Isn't the plan more like 23/5 like is already the case for several markets? I can't see the standard sessions moving more 9:30am/4pm weekdays to 24/7. I take it they'd still let, at least, one hour off for technical reasons. If I'm not mistaken it's the reason several markets are 23/5 and not 24/5: that one hour of downtime is basically for servers/maintenance right? (maybe someone can chime in) P.S: I take it technically there's 24/7 trading already seen that cryptocurrencies exchanges are opened 24/7 (I'm not sure: but I think that's the case) but I don't think those do anywhere near the volume of, say, options trading on equities during standard sessions (40 Gbit/s with peak over 70 Gbit/s for the full options feed).

dmurray 5mo ago
The 23/7 is not so much for maintenance as to have a defined window for changes to the market to happen. Every so often a new stock is listed or a stock ticker is changed or a stock is split, etc. There are smaller changes every single day, like to the settlement date of your trade. It's very convenient to be able to restart all your systems at 5pm, have them all load the updated reference data, and start them again in time for 6pm (or 7pm, or 4am tomorrow...). Even if you trade stocks and options and currencies and futures all over the world, a quirk of the calendar means they're basically all closed between 4 and 5pm Chicago time. Of course it's possible in principle to build systems where all this is dynamic and you can seamlessly trade with the old configuration at 4:59:59.999 and start trading the new one a millisecond later. But literally everyone has built systems that don't work on this, that rely on being able to chunk the continuous passage of time into discrete days. It would be painful to rearchitect them all now.

cgio 5mo ago

Only US. Other markets barely have liquidity during daytime and get most liquidity in opening and closing auctions. Maintenance periods are actually a complication. A few more state transitions for the system, but barely used for maintenance. The only value is for upgrades, which would still be scheduled with the market down and systems up, as participants also need to transition codebases for breaking changes, a test weekend or more is required etc. These systems are extremely resilient. You most often get an incident not because the system is down but because the latency profile has changed by a few ms.

willtemperley 5mo ago
Not really US only. LMAX is 24/7 and is a UK company, famous on HN for open sourcing their ring buffer. Crypto trading has been 24/7 since it began.
- cgio 5mo ago
  Ok, I will make it mostly US. Maybe a couple more markets, London and Tokyo? Futures may be a bit broader adoption too. Vast majority won’t move to 24/7. Crypto is a different game for the time being at least. It has its own challenges but also escapes quite a few of traditional exchange complexities.

bostik 5mo ago

Sports betting exchanges have been doing that for a very long time. There is never a good time to take the system down for maintenance - event settlements happen every few minutes, and live games with in-play betting are going on somewhere in the world at any given time. Makes things damn hard indeed, because you have to truly learn asynchronicity, CQRS and complex live migrations. (Incidentally, engineers who have worked on such systems tend to be over-represented in extreme HA businesses.)

antonvs 5mo ago

Crypto exchanges have been doing 24/7 trading for well over a decade. Of course they’ve had their own, uh, issues - but generally, reliability of transactions hasn’t been a major one, for the big exchanges.