4 ms·

Over 500PB of data, wow. Would love to know how and why "statistical models that produce price forecasts for over 50,000 financial instruments worldwide" requir

Over 500PB of data, wow. Would love to know how and why "statistical models that produce price forecasts for over 50,000 financial instruments worldwide" require that much storage.

Beijinger 1y ago
Me too. Is is really hard for me to understand, what XTX is actually doing. Trading? VC? AI/ML? Have you seen their portfolio? PS: Company seems legit. Impressive growth. But I still don't understand what they are doing. Provide "electronic liquidity". Well....
- orbifold 1y ago
  computing correlations between 50.000 financial instruments (X^T X) and doing linear regression ;).
- adastra22 1y ago
  High frequency trading.
guerby 1y ago
If you keep all order book changes for a large number of financial instruments volume adds up quickly.
- rickette 1y ago
  Would that kind of data not compress like crazy? Or would they need to keep all that data hot and fast.
  - guerby 1y ago
    From just a single exchange you can reach up to 1 million messages of order book change per second https://www.nasdaqtrader.com/snippets/inet2.html https://www.nasdaqtrader.com/snippets/inet2.html Message Volume 1,684,103,265 Messages per Second 1,134,640 Order Volume 871,875,595 Orders per Second 581,696 Share Volume 12,814,454,760 Executions per Second 193,350 Also if you look at equity derivative products which have parameters like type call/put, strike, maturity can be hundreds of financial products for one underlying stock. I worked in this sector and volume of data is a real challenge, no wonder you often get custom software to handle that :)
    - rickette 1y ago
      Thanks for the insight!
  - pcthrowaway 1y ago
    How do you propose lossless compression for all orderbook data? Of course if you are willing to lose granularity/information, it can be compressed a lot
taneliv 1y ago
I would imagine to lesser extent government policy changes and news articles, and to larger extent online discussions on topics relevant to these instruments. Models then attempt to extract signals with predictive value from all the noise. Probably contains non-trivial amount of history to correlate words to market performance in the past, say 20 years or more. But it's really just a guess, I haven't worked in this domain.

Over 500PB of data, wow. Would love to know how and why "statistical models that produce price forecasts for over 50,000 financial instruments worldwide" require that much storage.

Beijinger 1y ago

Me too. Is is really hard for me to understand, what XTX is actually doing. Trading? VC? AI/ML? Have you seen their portfolio? PS: Company seems legit. Impressive growth. But I still don't understand what they are doing. Provide "electronic liquidity". Well....

orbifold 1y ago
computing correlations between 50.000 financial instruments (X^T X) and doing linear regression ;).
adastra22 1y ago
High frequency trading.

guerby 1y ago

If you keep all order book changes for a large number of financial instruments volume adds up quickly.

rickette 1y ago
Would that kind of data not compress like crazy? Or would they need to keep all that data hot and fast.
- guerby 1y ago
  From just a single exchange you can reach up to 1 million messages of order book change per second https://www.nasdaqtrader.com/snippets/inet2.html https://www.nasdaqtrader.com/snippets/inet2.html Message Volume 1,684,103,265 Messages per Second 1,134,640 Order Volume 871,875,595 Orders per Second 581,696 Share Volume 12,814,454,760 Executions per Second 193,350 Also if you look at equity derivative products which have parameters like type call/put, strike, maturity can be hundreds of financial products for one underlying stock. I worked in this sector and volume of data is a real challenge, no wonder you often get custom software to handle that :)
  - rickette 1y ago
    Thanks for the insight!
- pcthrowaway 1y ago
  How do you propose lossless compression for all orderbook data? Of course if you are willing to lose granularity/information, it can be compressed a lot

taneliv 1y ago

I would imagine to lesser extent government policy changes and news articles, and to larger extent online discussions on topics relevant to these instruments. Models then attempt to extract signals with predictive value from all the noise. Probably contains non-trivial amount of history to correlate words to market performance in the past, say 20 years or more. But it's really just a guess, I haven't worked in this domain.