4 ms·

At a past job (hedge fund), my role was to co-ordinate investigations into why latency may have changed when sending orders. A couple of quants had built a ran

At a past job (hedge fund), my role was to co-ordinate investigations into why latency may have changed when sending orders. A couple of quants had built a random forest regression model that could take inputs like time of day, exchange, order volume etc and spit out an interval of what latency had historically been in that range. If the latency moved outside that range, an alert would fire and then I would co-ordinate a response with the a variety of teams e.g. trading, networking, Linux etc If we excluded changes on our side as the culprit, we would reach out to the exchange and talk to our sales rep there would might also pull in networking etc. Some exchanges, EUREX comes to mind, were phenomenal at helping us identify issues. e.g. they once swapped out a cable that was a few feet longer than the older cable and that's why the latency increased. One day, it's IEX, of Flash Boys fame, that triggers an alert. Nothing changed on our side so we call them. We are going back and forth with the networking engineer and then the sales rep says, in almost hushed tones: "Look, I've worked at other exchange so I get where you are coming from in asking these questions. Problem is, b/c of our founding ethos, we are actually not allowed to track our own internal latency so we really can't help you identify the root cause. I REALLY wish it was different." I love this story b/c HN, as a technology focused site, often thinks all problems have technical solutions but sometimes it's actually a people or process solution. Also, incentives and "philosophy of the founders" matter a lot too.

noitpmeder 11mo ago
Curious what your actual role was -- sounds very interesting! Project manager? Dev? Operations specialist? E.g. were you hired into this role, and what were the requisites?
- alexpotato 11mo ago
  I was what was called "Trade Desk". Many firms have them and they are a hybrid of: - DevOps (e.g. we help, or own, deployments to production) - SRE (e.g. we own the dashboards that monitored trading and would manage outages etc) - Trading Operations (e.g. we would work with exchanges to set up connections, cancel orders etc) My background is: - CompSci/Economics BA - MBA - ~20 years of basically doing the above roles. I started supporting an in house Order Management System at a large bank and then went from there. For more detail, here is my LinkedIn: https://www.linkedin.com/in/alex-elliott-3210352/ https://www.linkedin.com/in/alex-elliott-3210352/ I also have a thread about the types of outages you see in this line of work here: https://x.com/alexpotato/status/1215876962809339904?s=20 https://x.com/alexpotato/status/1215876962809339904?s=20 (I have a lot of other trading/SRE related threads here: https://x.com/alexpotato/status/1212223167944478720?s=20 https://x.com/alexpotato/status/1212223167944478720?s=20)
  - noitpmeder 11mo ago
    Thanks for all the info! I'm a front office engineer at a prop firm -- always interesting to get insight into how others do it. We have fairly similar parallels, maybe with the exception of throwing new exchange connections to the dedicated networking group. Always love watching their incident responses from afar (usually while getting impacted desks to put away the pitchforks). Great examples of crisis management, effectiveness and prioritization under pressure, ... All while being extremely pragmatic about actual vs perceived risk. (I'm sure joining KCG in August of 2012 was a wild time...)
    - alexpotato 11mo ago
      You are very welcome! It's definitely a job that you don't hear much about but has a lot of interesting positives for people who like technology and trading. Especially if you prefer shorter term, high intensity work vs long term projects (e.g. like developers). > Always love watching their incident responses from afar I actually have a thread on that too: https://x.com/alexpotato/status/1227335960788160513?s=20 https://x.com/alexpotato/status/1227335960788160513?s=20 > (I'm sure joining KCG in August of 2012 was a wild time...) And not surprisingly, a thread on that as well: https://x.com/alexpotato/status/1501174282969305093?s=20 https://x.com/alexpotato/status/1501174282969305093?s=20
  - 8cvor6j844qw_d6 11mo ago
    May I know if someone with no trading knowledge can get into this field? Or do new hires that you've seen generally have some background knowledge on related to trading, etc.? I did consider applying for a role in a very similar field, but figured I'll be fighting an uphill battle with no knowledge in trading/stock market/etc.
    - TheJoeMan 11mo ago
      You know how coders are expected to grind out leetcode interviews? For the finance fields, a common interview topic is what you read in “The Journal” (WSJ). So just stay on top of it for a few weeks, see some trends, etc.
reactordev 11mo ago
All technical problems are people problems
philipov 11mo ago
What kind of founding ethos doesn't allow tracking internal latency? Is their founding ethos "Never Admit Responsibility?"; "Never Leave A Paper Trail?" This company's official ethical foundation is "Don't Get Caught."
- WJW 11mo ago
  From the wiki about IEX: "It was founded in 2012 in order to mitigate the effects of high-frequency trading." I can see how they don't want to track internal latency as part of that, or at least not share those numbers with outsiders. That just encourages high frequency traders again.
  - pants2 11mo ago
    One would hope for a more technical solution to HFT than willful ignorance lol. For example, they could batch up orders every second and randomize them.
    - Workaccount2 11mo ago
      This is what should happen, because what the game actually being played is to profit off those who cannot react fast enough to news event, rather than profit off those who mispriced their order. Or leave things in place, but put a 1 minute transaction freeze during binary events, and fill the order book during that time with no regard for when an order was placed, just random allocation of order fills coming out of the 1 minute pause. These funds would lose their shit if they had to go back to knowledge being the only edge rather than speed and knowledge.
      - neonbrain 11mo ago
        This isn't a good approach because it assumes there are no market makers on trading venues, and that they (as well as exchanges) do not compete for order flow. Also, maybe you haven't noticed, but stocks are often frozen during news announcements by regulatory request, so such pauses are already in place and are designed to maintain market integrity, not disrupt it with arbitrary fills.
dboreham 11mo ago
> they once swapped out a cable that was a few feet longer than the older cable and that's why the latency increased That was not why. Possibly the cable made a difference (had an open circuit that made the NICs back down to a lower speed; noisy leading to retransmissions) but it wasn't the length per se.
- thijson 11mo ago
  It can be the length, there's about 1ns latency per foot.
- thadt 11mo ago
  Well, it depends on the granularity of the time scale right? When you're measuring milliseconds, then the cable length probably isn't a thing factoring into your latency calculation. When we're measuring time on the scale of nanoseconds then, yes, cable length is definitely something we care about and will reliably show up in measurements. In some situations, we not only care about the cable length, but also its temperature.
lopatin 11mo ago
Can you talk a bit more about the incentives to trade latency sensitive strategies on IEX in the first place? Is it still lucrative for its liquidity despite them artificially slowing down orders? Does a meta game evolve with HFTs all working around their system, essentially making it still a HFT playground but with extra steps? Do you think their unexpected latency increase for you guys was intentional, to free the water from sharks?

noitpmeder 11mo ago

Curious what your actual role was -- sounds very interesting! Project manager? Dev? Operations specialist? E.g. were you hired into this role, and what were the requisites?

alexpotato 11mo ago
I was what was called "Trade Desk". Many firms have them and they are a hybrid of: - DevOps (e.g. we help, or own, deployments to production) - SRE (e.g. we own the dashboards that monitored trading and would manage outages etc) - Trading Operations (e.g. we would work with exchanges to set up connections, cancel orders etc) My background is: - CompSci/Economics BA - MBA - ~20 years of basically doing the above roles. I started supporting an in house Order Management System at a large bank and then went from there. For more detail, here is my LinkedIn: https://www.linkedin.com/in/alex-elliott-3210352/ https://www.linkedin.com/in/alex-elliott-3210352/ I also have a thread about the types of outages you see in this line of work here: https://x.com/alexpotato/status/1215876962809339904?s=20 https://x.com/alexpotato/status/1215876962809339904?s=20 (I have a lot of other trading/SRE related threads here: https://x.com/alexpotato/status/1212223167944478720?s=20 https://x.com/alexpotato/status/1212223167944478720?s=20)
- noitpmeder 11mo ago
  Thanks for all the info! I'm a front office engineer at a prop firm -- always interesting to get insight into how others do it. We have fairly similar parallels, maybe with the exception of throwing new exchange connections to the dedicated networking group. Always love watching their incident responses from afar (usually while getting impacted desks to put away the pitchforks). Great examples of crisis management, effectiveness and prioritization under pressure, ... All while being extremely pragmatic about actual vs perceived risk. (I'm sure joining KCG in August of 2012 was a wild time...)
  - alexpotato 11mo ago
    You are very welcome! It's definitely a job that you don't hear much about but has a lot of interesting positives for people who like technology and trading. Especially if you prefer shorter term, high intensity work vs long term projects (e.g. like developers). > Always love watching their incident responses from afar I actually have a thread on that too: https://x.com/alexpotato/status/1227335960788160513?s=20 https://x.com/alexpotato/status/1227335960788160513?s=20 > (I'm sure joining KCG in August of 2012 was a wild time...) And not surprisingly, a thread on that as well: https://x.com/alexpotato/status/1501174282969305093?s=20 https://x.com/alexpotato/status/1501174282969305093?s=20
- 8cvor6j844qw_d6 11mo ago
  May I know if someone with no trading knowledge can get into this field? Or do new hires that you've seen generally have some background knowledge on related to trading, etc.? I did consider applying for a role in a very similar field, but figured I'll be fighting an uphill battle with no knowledge in trading/stock market/etc.
  - TheJoeMan 11mo ago
    You know how coders are expected to grind out leetcode interviews? For the finance fields, a common interview topic is what you read in “The Journal” (WSJ). So just stay on top of it for a few weeks, see some trends, etc.

reactordev 11mo ago

All technical problems are people problems

philipov 11mo ago

What kind of founding ethos doesn't allow tracking internal latency? Is their founding ethos "Never Admit Responsibility?"; "Never Leave A Paper Trail?" This company's official ethical foundation is "Don't Get Caught."

WJW 11mo ago
From the wiki about IEX: "It was founded in 2012 in order to mitigate the effects of high-frequency trading." I can see how they don't want to track internal latency as part of that, or at least not share those numbers with outsiders. That just encourages high frequency traders again.
- pants2 11mo ago
  One would hope for a more technical solution to HFT than willful ignorance lol. For example, they could batch up orders every second and randomize them.
  - Workaccount2 11mo ago
    This is what should happen, because what the game actually being played is to profit off those who cannot react fast enough to news event, rather than profit off those who mispriced their order. Or leave things in place, but put a 1 minute transaction freeze during binary events, and fill the order book during that time with no regard for when an order was placed, just random allocation of order fills coming out of the 1 minute pause. These funds would lose their shit if they had to go back to knowledge being the only edge rather than speed and knowledge.
    - neonbrain 11mo ago
      This isn't a good approach because it assumes there are no market makers on trading venues, and that they (as well as exchanges) do not compete for order flow. Also, maybe you haven't noticed, but stocks are often frozen during news announcements by regulatory request, so such pauses are already in place and are designed to maintain market integrity, not disrupt it with arbitrary fills.

dboreham 11mo ago

> they once swapped out a cable that was a few feet longer than the older cable and that's why the latency increased That was not why. Possibly the cable made a difference (had an open circuit that made the NICs back down to a lower speed; noisy leading to retransmissions) but it wasn't the length per se.

thijson 11mo ago
It can be the length, there's about 1ns latency per foot.
thadt 11mo ago
Well, it depends on the granularity of the time scale right? When you're measuring milliseconds, then the cable length probably isn't a thing factoring into your latency calculation. When we're measuring time on the scale of nanoseconds then, yes, cable length is definitely something we care about and will reliably show up in measurements. In some situations, we not only care about the cable length, but also its temperature.

lopatin 11mo ago

Can you talk a bit more about the incentives to trade latency sensitive strategies on IEX in the first place? Is it still lucrative for its liquidity despite them artificially slowing down orders? Does a meta game evolve with HFTs all working around their system, essentially making it still a HFT playground but with extra steps? Do you think their unexpected latency increase for you guys was intentional, to free the water from sharks?