5 ms·

Financial data (tick to EOD), network traffic data (TCP packet level sends / receives) and farm data (sensor + farm ERP data) All of them are basically time se

Financial data (tick to EOD), network traffic data (TCP packet level sends / receives) and farm data (sensor + farm ERP data) All of them are basically time series with some master data, none of them is more than a few dozen GB So in any case, I think time series data is worth a look.

noname123 12y ago
I run backtesting for options trading as a hobby and storing EOD tick data, querying it and extracting it is a pain. I dl my source data currently from a retail historical data provider, then store it in MongoDB in AWS. What would you recommend tech stackwise to do backtesting on time seriea data?
- remyp 12y ago
  They don't have options data, but you might check out QuantConnect or Quantopian if you're interested in stocks and FX.
- leoplct 12y ago
  Have you tried quantopian.com ?
- jaredbroad 12y ago
  QuantConnect.com has tick data for every stock in the US universe back to 1998 -- its roughly 4TB of data.
- dmichulke 12y ago
  I also first used MongoDB but I switched to cassandra and I find it much better. MongoDB was bad for this case because it devours memory and it lacks a primary key (so saving a tick that's already creates a second tick). cassandra is better (it has a PK, although it's a bit weird because of its distributed-first attitude) but in the long run I think postgres would be even better (because it's space efficient). Apart from that I use some Java libraries and clojure/incanter and program the rest around it myself.
wglb 12y ago
You can get vast amounts of stock option data from http://www.orats.com/ http://www.orats.com/. I don't think of the options stream as big data. It is more of a "fast" data problem, where at the open you need to handle 100,000 ticks per second, doing a implied volatility calculation on each tick. It is pretty demanding, when, for example, you consider that when the price of IBM moves by one penny, you get 400 options quote right then. You end up with a billion quotes in a typical day, if memory serves correctly. I didn't think of this as Big Data, somehow. It seems to me that big data is more complex than that. You most often just process the data, say for backtesting, in a strict series, from the start to the end, or for an interval. I think of Big Data having more structure.

noname123 12y ago

I run backtesting for options trading as a hobby and storing EOD tick data, querying it and extracting it is a pain. I dl my source data currently from a retail historical data provider, then store it in MongoDB in AWS. What would you recommend tech stackwise to do backtesting on time seriea data?

remyp 12y ago
They don't have options data, but you might check out QuantConnect or Quantopian if you're interested in stocks and FX.
leoplct 12y ago
Have you tried quantopian.com ?
jaredbroad 12y ago
QuantConnect.com has tick data for every stock in the US universe back to 1998 -- its roughly 4TB of data.
dmichulke 12y ago
I also first used MongoDB but I switched to cassandra and I find it much better. MongoDB was bad for this case because it devours memory and it lacks a primary key (so saving a tick that's already creates a second tick). cassandra is better (it has a PK, although it's a bit weird because of its distributed-first attitude) but in the long run I think postgres would be even better (because it's space efficient). Apart from that I use some Java libraries and clojure/incanter and program the rest around it myself.

wglb 12y ago

You can get vast amounts of stock option data from http://www.orats.com/ http://www.orats.com/. I don't think of the options stream as big data. It is more of a "fast" data problem, where at the open you need to handle 100,000 ticks per second, doing a implied volatility calculation on each tick. It is pretty demanding, when, for example, you consider that when the price of IBM moves by one penny, you get 400 options quote right then. You end up with a billion quotes in a typical day, if memory serves correctly. I didn't think of this as Big Data, somehow. It seems to me that big data is more complex than that. You most often just process the data, say for backtesting, in a strict series, from the start to the end, or for an interval. I think of Big Data having more structure.