13 ms·
Shortwave Trading: The West Chicago Tower Mystery
- dx034 8y agoI guess low bandwidth is no problem. You only need to send a stream of price changes of the most common instruments (currencies, stocks). And only if there are meaningful movements. I guess a few kbit/s can already provide a lot of value. How fast would such a connection be? Is there an easy way to calculate the distance the signal takes compared to great circle line?
- coob 8y agoWouldn't you just send whatever is needed to place the order? Do all of the processing at the source and just send the final instruction? (honest question I have no idea how this works)
- dmeeker 8y agoGenerally speaking, folks use the CME futures as a leading indicator, so you may have several different trading strategies that trade against the European markets and just use the CME pricing as one of their inputs. In that case, the most bandwidth-efficient play is to send your market data over once from Chicago, replicate it / make it available to all your trading strategies, and let them do their work locally.
- tomalpha 8y agoThat might well be true, but if you've only got a few kbits/second of bandwidth even that might be too much.
- dmeeker 8y agoSure, you're certainly not sending a full feed -- presumably you've got some algo sitting in Chicago that's looking for interesting / potentially profitable price / vol changes and sending them via the fast path, while the remainder of the feed is going via higher-bandwidth but slower paths.
- pbhjpbhj 8y agoYou might as well have your decision logic before the communication path, then you can send like buy/sell orders in code using an agreed decryption pad/dictionary. (Akin to Morse's original conception).
- CamTin 8y agoI'm not an algo trader, but you need the data from both ends of the link to determine if there is a trade you want to make, right? One side or the other is going to need prices from both ends in order to make a trading decision, and preferably both, since each side can only make decisions that are relevant to their side of the link, because needing a round trip negates all of the speed advantage you incurred by developing a top secret HF trading link. Edit: I meant the HF here to refer to the "high frequency" wavelengths in use in the link (in Ham parlance, HF and shortwave are roughly synonymous), but just realized it also could be "high frequency" as in the frequency of trades.
- 21 8y agoThe price on the other side also moves, so it's more complicated.
- dx034 8y agoYou can send samples. E.g. the USD/EUR rate (or the price of an equities future) if it changes by more than x within the last 50ms. The difference of that rate to the "local" rate in London is enough to generate the trade. Same with equity indices. You'll know that if there's no movement, there's no opportunity. So it's enough to send a signal if there has been a significant move that's "young" enough so that the other side can't know it yet. Your trades only make money on the difference of the public information in London and what you get faster from Chicago. Therefore sending trade instructions isn't helpful as Chicago doesn't know London prices. But just giving the London algorithm that bit of inside can help a lot. I assume trade strategies are mostly very simple as well. Basically just anticipating a change in price and quickly buying/selling to make some money off it. N.B.: I only talk about Chicago->London here but given that Chicago hosts one of the largest Futures markets, I'd assume a lot is actually done in reverse. Using local prices from London of an underlying instrument (e.g. London stock market) to "predict" the value of the future traded in Chicago. You can make a lot of money if you know the price of the underlying instrument of a future before others do.
- edf13 8y agoIt may well be signals going over the stream... or significant movements only.
- chrisseaton 8y ago> Mystery #6: What could be interesting across an ocean? Err.... the rest of the world?
- ndespres 8y agoThe question is asked in the context of the tower being owned by U.S. Cellular, who would not need shortwave antennae pointed across the pond to service local cell customers.
- tomglynch 8y agoSo intriguing. Bringing out the old tech for new use cases
- madengr 8y agoI wouldn’t call it old tech. It’s just another portion of the spectrum. There is some pretty advanced stuff going on with HF.
- badosu 8y agoGiven the limitations of such a medium I wonder if the TCP/IP stack is still viable or another transport protocol is used.
- tomalpha 8y agoIt's been a few years since I was involved with any of this, but even for higher-bandwidth microwave networks you want to squeeze every last bit out of the protocol. I'd guess in this instance that they'll skip both the TCP and IP layers completely and essentially be sending minimally encapsulated payloads out almost raw. Minimally encapsulated at this level might actually mean no encapsulation. Maybe some error-detection/CRC type stuff as these links tend to be super lossy.
- gaius 8y agoWhy IP over HF Radio should be Avoided https://www.isode.com/whitepapers/ip-over-stanag-5066.html https://www.isode.com/whitepapers/ip-over-stanag-5066.html
- 21 8y agoYou probably want to combine in a single raw packet the data, compressed with a prediction codec with the error detection code. Since you also have a lower-latency high-speed link (the transatlantic cable), you can continuously update the probabilistic model used by the compression to squeeze the last bit.
- JustSomeNobody 8y agoI can't imagine they'd need the TCP/IP stack. You might do better with something like: SOH<header data>EOH STX<data>ETX CRC EOT Then the receiver would send an ACK or NACK. Repeat as necessary.
- dx034 8y agoWould you even acknowledge? The data is worthless after one round-trip, so you wouldn't send it twice. The receiver just needs to make sure that they read information correctly and can deal with losses.
- lallysingh 8y ago"Garden Leave" is a paid non-compete time (often a year) used in the financial industry when someone switched jobs. The prior employer invokes it at their option.
- deleted 8y ago[deleted]
- dx034 8y agoThat's not always correct. It can be shorter (3 months are common in London) and is often a fixed part of your contract (notice period). If both agree you can shorten it but it's not just invoked at the discretion of your employer.
- golergka 8y agoI wasn't aware that "paid non-compete" is a thing, thanks for clarifying it. Next time when I see a non-compete in a contract (which is usual for developers), instead of negotiating it out, I'll try to negotiate something like this. What do these agreements usually look like?
- deleted 8y ago[deleted]
- lallysingh 8y agoThe employer has the option to pay you your salary (not bonus, which is a big part of your income!) to not work at all instead of you going to a competitor. I think that the comp part is necessary to make it enforceable at all.
- madengr 8y agoThey are probably also monitoring HF propagation paths, as there can be ground wave and sky wave, with varying dispersion. The X300 has some latency too. I’ll be they are doing the majority of processing directly on the FPGA to avoid SDR latency. The X310 has a larger FPGA, and would be tell-tale. Bring on the comms with neutrinos or entangled particles.
- andai 8y agoUp next: Tachyon trading
- cameldrv 8y agoMaybe not, but almost as exotic: https://www.forbes.com/sites/brucedorminey/2012/04/30/neutrinos-to-give-high-frequency-traders-the-millisecond-edge/#6daa4994590c https://www.forbes.com/sites/brucedorminey/2012/04/30/neutri... While all of the other chumps are trying to bounce their signals around the surface in fibers or various layers of the atmosphere, you just blast straight through the earth with (anti)neutrinos! We live in a weird, weird world, and there's real money behind this idea. It might even affect the strategic nuclear balance, because submarines are supposed to be silent and invisible. Their nuclear reactors do generate extremely large numbers of antineutrinos though...
- tomfanning 8y agoWould love to know what licensing regime / category they are using for this. It definitely isn't ham, because that's non-commercial only. The fact that log period antennas are in use is interesting too - low gain but directional and useful across a broad frequency range - maybe some spread spectrum technology?
- madengr 8y agoProbably licensed on several bands, as propagation conditions change quickly. They’d need to hop around, if not simultaneously transmitting on multiple bands.
- Jommi 8y agohttps://apps.fcc.gov/els/GetAtt.html?id=205708&x= https://apps.fcc.gov/els/GetAtt.html?id=205708&x=
- jhpankow 8y agoWow, that would do it.
- Animats 8y agoThat one, WI2XER, has been heard, and seen on a spectrum analyzer.[1] They send their call sign in Morse code, send various test tones, and send what look like modem-type signals. 100 KW, so they will have no problem reaching Europe. Not much bandwidth, though, because they don't have enough spectrum space. Maybe a few hundred kilobits/second. As a way to get info from markets faster, it's limited. You avoid some switching and path delays, but the data rate is slow, so it's only faster for short messages. [1] http://i56578-swl.blogspot.com/2018/03/wi2xer-skycast-experimental-radio.html http://i56578-swl.blogspot.com/2018/03/wi2xer-skycast-experi...
- sailfast 8y agoThis is a wonderful article for so many reasons. Outlines resources for doing this yourself, uses industry knowledge to outline hypotheses and then tests them with data, and reveals a really interesting possibility for market communication. Also, a bit of local Chicago geography :) An enjoyable read! Edit: the “is this a troll” comment makes this even more enjoyable for some reason. Now it’s an even bigger treasure hunt.
- Jommi 8y agoWeirdest thing I've read in a long while. The Author's comments refer to an internet forum thread[0] in German which discusses a German news article from this April.[1]However, it turns out this article is an April Fools text. Just digging deeper into the whole blog you can find something that can only be described as a fanfiction representation of "Flashboys", but in the context of this "Shortwave trading".[2] Is this whole thing a troll? Is this real? What on earth is this? [0]http://funkbasis.de/viewtopic.php?f=39&t=45900 http://funkbasis.de/viewtopic.php?f=39&t=45900 [1]https://www.heise.de/newsticker/meldung/Boersenhandel-beschert-Kurzwellenfunk-ein-Comeback-4008891.html https://www.heise.de/newsticker/meldung/Boersenhandel-besche... [2]https://sniperinmahwah.wordpress.com/2017/06/26/network-effects-part-ii/ https://sniperinmahwah.wordpress.com/2017/06/26/network-effe...
- Jommi 8y agoSeems like it is real: https://www.six-group.com/en/home/media/releases/2018/20180503-12H-start.html https://www.six-group.com/en/home/media/releases/2018/201805...
- dx034 8y agoThis is microwave based. Those lines are well known by now and there are known routes within Europe (most notable Frankfurt<->London) and the US (NYC<->Chicago). However, microwave only works over short distances (FRA<->LON has multiple towers to cover the distance). Crossing the atlantic isn't possible. Shortwave could deliver a similar speedup over long distances. I'm a bit sceptical though. Don't the waves have to bounce several times to get to the target? I'm not sure how much of the ~30% potential speed up vs. fibre you still get.
- jhpankow 8y agoI'd suspect the latency improvement is control over the entire path and not specifically about the propagation speed/delay. Getting your data across a long haul undersea fiber requires aggregating it with everyone else's data at a central point before it gets sent out optically. This adds latency. Even with the propagation variability of the wireless path you might be saving hundreds of nanoseconds.
- social_quotient 8y agoIf this article was an intro to a fiction novel, I’d be interested in reading the entire thing. Some sort of HFT heist with a good mix of factual engineering and finance.
- yangcanvas 8y agoRougue Code by Mark Russinovich was a novel I enjoyed about a HFT heist.
- a-dub 8y agoAll along everyone thought that numbers stations were how governments send messages to their spooks, in reality they were just poor bastards holed up in tiny shacks manually reading the tape for the S&P e-mini...
- lettergram 8y agoInteresting... I actually know someone working on a project similar to this (who works at an HFT firm) - if not actually this. His team was simulating and testing how various weather conditions could affect trade placement times. For instance, does a high-pressure system, a heat wave, a blizzard, etc. impact the trade placement times, if so how does that impact the algorithms. He works out of Chicago and I've heard of similar things before, so this isn't unknown to people in the area.
- Ftuuky 8y agoFrom what I understand (and I'm no expert in HFT or electric engineering), this kind of sub-second trading is done in very specific corridors: Chicago-NY, NY-London, London-Frankfurt, London-Paris, Paris-Frankfurt. I wonder if the same applies in Asia, with Singapore, HK, Tokyo, Sidney...
- sniperinmahwah 8y agoSome fun comments here, thanks! For all those who doubt, here are some public documents (to be detailed on the blog soon) about antennas in UK, filed by a well-known "HFT" firm: https://publicaccess.southbucks.gov.uk/online-applications/applicationDetails.do?activeTab=summary&keyVal=P5UGGCOHFL500 https://publicaccess.southbucks.gov.uk/online-applications/a...
- discussedbefore 8y agoThe secret world of microwave networks https://news.ycombinator.com/item?id=12862789 https://news.ycombinator.com/item?id=12862789 (2016) HFT in My Backyard https://news.ycombinator.com/item?id=8354278 https://news.ycombinator.com/item?id=8354278 (2014)
- deleted 8y ago[deleted]
- cjsawyer 8y agoA microwave radio setup to the nearest populated area gave my rural-living parents modern internet a few years ago. It’s not the same scale but the same idea! It’s far and away better than satellite.
- aoki 8y agoi am not sure i would agree that this has been discussed before. this is not simply about use of RF (microwave, mmwave) for HFT. as i understand it, that kind of thing typically involves COTS radios and conventional dish antennas. this instance appears to involve longer-range links and SDRs (in this case, the ettus USRPs).
- bobvan13 8y agoYes, well said. I think you're making an important distinction. Shortwave is old technology, but combined with SDR, this is a novel application of it AFAIK.
- ropeadopepope 8y agoDoes anybody know what happened to Windy Apple Technologies? They promised the lowest latency microwave connection from NY to Chicago, then promptly disappeared...
- sniperinmahwah 8y agoI know
- ropeadopepope 8y agoAre you willing to say here, or is there another way I can contact you?
- sniperinmahwah 8y agosniperinmahwah@protonmail.com
- alexpil 8y agoI'm still around and in the industry. Never promised the lowest latency, but I was the first one by two years to build NY<>Chicago.
- sniperinmahwah 8y agoTrue.
- ropeadopepope 8y agoThank you for replying! I'm glad to hear you're still around. I was always impressed with your work.
- nimbius 8y agoCould it possibly be bulk data from sensors? I worked on a project in several states for IBM that handled traffic shaping and detection on highways and onramps. This system was used to identify gridlock and adjust lights at offramps or give people an ETA until theyre downtown. Most adjacent cities were staunchly NIMBY and didnt care to have the state government trenching cable to and fro for traffic control, so the data ended up hopping from town to town on microwave subcontracted from the cellular company that handled the large electric billboards on the highway. the benefit of this was also the ability to route signal control data to large intersections without having to shut them down to dig cable trenches from the IBM network.
- jlgaddis 8y agoNot at the frequencies that those LP antennas are tuned for.
- vasilipupkin 8y agoi have one question: why expend some much energy on unearthing obscure stuff that has relevance only to a very small subset of a very small industry and literally nobody else? seems like a waste of effort.
- jlgaddis 8y agoBecause it's interesting to someone. I don't understand why my girlfriend likes to watch Youtube videos of other girls doing their make-up and hair but, then again, she doesn't understand why I need all the computers that I have.
- vasilipupkin 8y agosure, that's a trivial response that doesn't really address my point. I am suggesting that the authors could expend the same energy and find or create something that's much more interesting to many more people.
- joezydeco 8y agoPerhaps because that very small subset of a very small industry can have a major impact on the entire global economy when something goes off plan? It also hints at how much money is actually in play with these "very small" companies. If they're spending hundreds of millions to get an edge (sometimes in picoseconds) on the competition, what does that say about the magnitude of the money they are working with and willing to risk?
- vasilipupkin 8y ago"Perhaps because that very small subset of a very small industry can have a major impact on the entire global economy when something goes off plan?" this is a red herring. this specific article in no way discusses or addresses how this would happen, nor does it tie in relevance of shortwave to this specific point
- FearNotDaniel 8y agoShort answer: because this is Hacker News, not Middle Manager In A Large Organization Serving A Well Established Market News. The author went to the effort simply because he himself found it interesting. He published it on a hunch that at least a few other people would find it as interesting as he did, even if they are not working in that industry, just because technology is interesting in its own right. He was right.
- wglb 8y agoSo why not listen in the vicinity of the towers and deduce what frequency this is on?
- sniperinmahwah 8y agoSome are doing it, obviously.
- bobvan13 8y agoListening is the easy part. But the sender would be foolish not to strongly encrypt what they send. Are you really "listening" if all you get after decoding is pseudo-random bits?
- wglb 8y agoListening will at least you know the frequency and type of modulation so you can at least understand who it is. And if the signal is always on or at certain times of the day only can help identify purpose
- jlgaddis 8y agoI'm not even remotely involved with the HFT world, but I can't think of any good reason why they'd want to receive "shortwave" signals from Europe in Chicago. Short-haul microwave links for low latency and high bandwidth, yes, but at shortwave frequencies you are literally talking about BITS per second. TFA says "Think dialup speeds" but this is lower than even dialup! They certainly aren't receiving any meaningful amount of data (which I presume you need for HFT) with that shortwave receiver.
- IndexicalDemon 8y agoThe crucial information is up and down ticks of the ES-mini futures at the Chicago Mercantile Exchange. These lead the S&P500 (SPY) index funding in NYC (by 4 milliseconds), and the ES-mini ticks up and down about once per second. This few bits of information carries huge value since much of the world's equity market responds to it. See https://arxiv.org/abs/1302.5966 https://arxiv.org/abs/1302.5966
- mturmon 8y agoGreat link, thank you. Abstract in case it piques people's interest: "High frequency trading has led to widespread efforts to reduce information propagation delays between physically distant exchanges. Using relativistically correct millisecond-resolution tick data, we document a 3-millisecond decrease in one-way communication time between the Chicago and New York areas that has occurred from April 27th, 2010 to August 17th, 2012. We attribute the first segment of this decline to the introduction of a latency-optimized fiber optic connection in late 2010. A second phase of latency decrease can be attributed to line-of-sight microwave networks, operating primarily in the 6-11 GHz region of the spectrum, licensed during 2011 and 2012. Using publicly available information, we estimate these networks’ latencies and bandwidths. We estimate the total infrastructure and 5-year operations costs associated with these latency improvements to exceed $500 million."
- mathgenius 8y agoIt's not amount of data but latency. A signal to buy is just one bit, but you need to get it before everyone else does.
- reaperducer 8y agoThis was covered pretty extensively by Crain's Chicago Business. Some companies are so obsessed with this sort of thing that they'll pay half a million dollars to move a receive site across the street. They measure results in picoseconds. Too much pressure for me.
- sandworm101 8y agoOne has to wonder if this cutthroat competition will ever turn aggressive. Someone with a SDR and a bit of knowhow could shutdown/jam these systems very easily. The apparatus is in public view and they don't seem to be guarding their trash. A well-timed attack on a couple of these towers would create a few seconds of instability on the markets, enough time for another HFT platform to take advantage. A blast from a 1-watt transmitter parked across the street would overwhelm any signal coming from across an ocean. Try calling the cops about to complain that your antenna reception is being interfered with. The days of cops hunting rogue radio stations are long gone.
- jandrese 8y agoThey wouldn't call the cops, they would call the FCC. If you keep doing it they will catch you.
- johnohara 8y ago> It showed a direct link between the West Chicago tower and another tower right by the Chicago Mercantile Exchange! It's pointing at the top of 10 South Canal, one of downtown Chicago's main tombstones for data communications. The building has 3-6 foot concrete walls, no windows for the first 23 stories and was reportedly designed to withstand an overhead nuclear blast. The standing joke used to be that if Chicago were nuked you'd still get a phone bill because of that place. There's a lot of underground fiber in that area too. Including 24-count bundles mounted near the ceiling of the old underground tunnel delivery system -- the one that accidentally flooded during the 90's. They were installed by Metropolitan Cable back in the late eighties/early nineties during the "politically connected" fiber gold rush to replace dedicated copper with light. It would not be speculation to think there's fiber running directly from the MERC to 10 South Canal or that the MERC's location near 10 South is a coincidence. The line-of-sight out to Aurora is interesting too. There are high spots out there with tower clusters reminiscent of East Texas in the 1930's. I believe there are backup trading centers out there as well built after 9/11.
- johnohara 8y agoCorrection: Metropolitan Fiber not Metropolitan Cable. Ultimately purchased by WorldCom.
- retzkek 8y agoThe map shows he meant the CME Group's data center in Aurora: http://www.cmegroup.com/trading/colocation/co-location-services.html http://www.cmegroup.com/trading/colocation/co-location-servi...
- johnohara 8y agoThe other thing to note is the proximity of the BNSF rail line. There are two 4-inch continuous PVC pipes next to the track, each with a gel-sealing 24-count fiber bundle inside. They run from downtown Chicago to Hinsdale to Aurora and beyond. They were installed about a year before Bershire-Hathaway purchased them.
- dghughes 8y agoI recall seeing I think on 60 Minutes HFT that wanted worse latency but I can't recall why. They have massive spools of fiber optics using them as a "resistor" (my term) to slow down the transmission. It seems to be the opposite of this but for the life of me I can't remember the advantage of doing it. Found it: https://www.theatlantic.com/business/archive/2016/06/iex-approved/487898/ https://www.theatlantic.com/business/archive/2016/06/iex-app...
- Ftuuky 8y agoI think you're referring to IEX, an exchange that adds a delay with those spools of fiber optics. According to wiki, "350 microsecond delay adds a round-trip delay of 0.0007 seconds and is designed to negate the certain speed advantages utilized by some high-frequency traders". https://en.wikipedia.org/wiki/IEX https://en.wikipedia.org/wiki/IEX Edit: didn't saw your edit.
- neomantra 8y agoUnrelated to the 60 Minutes piece, but fiber is also spooled at network aggregation points within an exchange colo environment (at least NYSE Mahwah and NASDAQ Carteret) so that access is fair (equal latency) for all the customer cabinets.
- jandrese 8y agoThe reason is that "the market" is made up multiple exchanges and to make a trade you need to do it on a specific exchange, but stocks are spread across all of the exchanges so if you need to make a big trade you have to send the trade order to several exchanges at once. The problem is that HFT firms will see the trade hit the closest exchange (in terms of network latency), and then rush out to buy out all of the remaining shares of that stock at that price on the other exchanges (beating you because they HFT better) and relisting them at a higher price. HFT firms will tell you that this is perfectly acceptable behavior because if you then buy the stock from them that the original person simply wasn't charging enough for it. They will also tell you that it improves liquidity, but that doesn't make sense either. It's not like they're sitting on the shares keeping the markets moving. All it does is add a tax to trades. IEX delays the transactions just enough that the nanosecond advantages that HFT enjoys can't be used to front run trades.
- slr555 8y agoAs a Ham I am certainly intrigued by this posting. There are a number of obstacles I can see to effectively using shortwave for financial services. Propagation conditions vary by time of day, solar activity and other exogenous factors. Another post hypothesized a 100 Kw transmitter power but even without running the numbers it seems like the FCCs Maximum Permissible Exposure Limits (MPE) would dictate a larger exclusion zone than the author encountered, however proper antenna modeling would be needed to be completely accurate. Another confounding factor to potential efficacy of the site is the surrounding terrain. The photos show nearby structures and trees which are less conducive to effective take off angles and propagation associated with the Yagi type antennas depicted. Most hardcore Ham sites with big towers have well groomed earth (or even better, water) around them. Additionally, the data would obviously have be encrypted and transmitted with extremely robust error correction. It would be fun to grab a couple of boxes from AOR and see what is coming off those antennas.
- cameldrv 8y agoFrom the McKay brothers' website: "It's better to be fast 99% of the time than slow 99.999% of the time" - Bob Meade and Stéphane Tyč, Co-Founders If your trading strategy is simply arbitraging Chicago/London, it's probably ok if your link is even 50% reliable, if for that 50% of the time, it's significantly faster than your competition's data. London is going to get the information 10ms later anyhow, so if a packet drops, you can still execute the offsetting transaction, just at the same time as everyone else.
- aoki 8y agoa meta-comment: it's weird - i submitted this a couple of days ago (https://news.ycombinator.com/item?id=17018883 https://news.ycombinator.com/item?id=17018883), but it still doesn't show up in the default HN search view. (we even edited the blog headline the same way!) but it does show up in the "by date" view.
- mhurd 8y agosome radio stuff here that may or may not be of interest FWIW: https://meanderful.blogspot.com/2017/05/lines-radios-and-cables-oh-my.html https://meanderful.blogspot.com/2017/05/lines-radios-and-cab...
- sniperinmahwah 8y ago