4 ms·

i fail to see the point in using hadoop if your universe is limited to foreign exchange... you can probably fit all ticks of all pairs in existence since the 70

i fail to see the point in using hadoop if your universe is limited to foreign exchange... you can probably fit all ticks of all pairs in existence since the 70s in less than a TB

dodders 12y ago
Assuming 40 pairs, 2 ticks per second and 1kb to store a tick (including timestamps, indexes etc), I reckon that's about 70TB for 40 years worth of data, not counting weekends. I've assumed only outright ticks, hence the 40 pairs - in reality most banks store the crosses as well, which can be up to 1,600 pairs (40 x 40), which will get you into the PB range for the 40 years.
- btn 12y ago
  One kilobyte per tick seems quite generous. All of the EURUSD tick data for 2013 from histdata.com (the source mentioned in the article) is only 515MB (~20GB for 40 years, ~824GB for 40 pairs).
  - throwaway283719 12y ago
    I would say a "tick" comprises the timestamp (stored as a long int) and ten levels of the order book (bid price, ask price, bid size, ask size) each stored as a double-precision float or a long int, so that's (1 + 4 * 10) * 8 = 328 bytes per tick, so 1KB isn't far off. Obviously not every level changes on every tick, so there are opportunities for compression, that can be significant. Note that the "tick data" from histdata.com gives you prices sampled every 1 second (so not every tick) for the top level of the order book, and doesn't give you any size information at all.
- xxs 12y ago
  Not the surprise: you need less than 3bytes on average to store a tick...
jjHey 12y ago
That is a very good point. An initial idea of Wolf was to create a platform that could trade virtually anything (Craigslist, Amazon, Ebay, Stock market, Forex, etc) as long as there's a trading API, and could make use of data from just about anywhere. Not just ticks. Imagine you can write a trading rule like this: "if Mr. X invested $ in company A and Ms. Y recommends buying stocks of company B or an epidemy spreads in country C or product D received great consumer reviews then trade E for F". That would require a lot of data: news articles, financial data, customer reviews, geopolitical events, etc. One of the Insight data engineer found some cool correlations between geopolitical events from GDELT and Forex. Wolf was created end-to-end in 6 weeks so... I focused on Forex. P.S. One year of compressed Forex ticks from HistData.com is about 100GB.

i fail to see the point in using hadoop if your universe is limited to foreign exchange... you can probably fit all ticks of all pairs in existence since the 70s in less than a TB

dodders 12y ago

Assuming 40 pairs, 2 ticks per second and 1kb to store a tick (including timestamps, indexes etc), I reckon that's about 70TB for 40 years worth of data, not counting weekends. I've assumed only outright ticks, hence the 40 pairs - in reality most banks store the crosses as well, which can be up to 1,600 pairs (40 x 40), which will get you into the PB range for the 40 years.

btn 12y ago
One kilobyte per tick seems quite generous. All of the EURUSD tick data for 2013 from histdata.com (the source mentioned in the article) is only 515MB (~20GB for 40 years, ~824GB for 40 pairs).
- throwaway283719 12y ago
  I would say a "tick" comprises the timestamp (stored as a long int) and ten levels of the order book (bid price, ask price, bid size, ask size) each stored as a double-precision float or a long int, so that's (1 + 4 * 10) * 8 = 328 bytes per tick, so 1KB isn't far off. Obviously not every level changes on every tick, so there are opportunities for compression, that can be significant. Note that the "tick data" from histdata.com gives you prices sampled every 1 second (so not every tick) for the top level of the order book, and doesn't give you any size information at all.
xxs 12y ago
Not the surprise: you need less than 3bytes on average to store a tick...

jjHey 12y ago

That is a very good point. An initial idea of Wolf was to create a platform that could trade virtually anything (Craigslist, Amazon, Ebay, Stock market, Forex, etc) as long as there's a trading API, and could make use of data from just about anywhere. Not just ticks. Imagine you can write a trading rule like this: "if Mr. X invested $ in company A and Ms. Y recommends buying stocks of company B or an epidemy spreads in country C or product D received great consumer reviews then trade E for F". That would require a lot of data: news articles, financial data, customer reviews, geopolitical events, etc. One of the Insight data engineer found some cool correlations between geopolitical events from GDELT and Forex. Wolf was created end-to-end in 6 weeks so... I focused on Forex. P.S. One year of compressed Forex ticks from HistData.com is about 100GB.