4 ms·

I do want to roll my own for learning sake, I do fallback to iex when I need it for historical data and backtesting. Did you write the medium article? I’m curi

I do want to roll my own for learning sake, I do fallback to iex when I need it for historical data and backtesting. Did you write the medium article? I’m curious on the first leg of the architecture - cloud scheduler to cloud functions would only make sense for longer periods between injestion right? Something 24/7 would be most efficient on an instance running 24/7 via compute/kube/app engine since it’s not request based right?

srazzaque 7y ago
If you are interested in learning how to roll a timeseries database, and if you're not already familiar with it, I'd second the suggestions to try KDB/Q. Even if only to be inspired and understand what "good" looks like in this space. I say this as someone who is generally averse to proprietary enterprise software. More often than not, it's a horrible pile of crap that's sold on the golf course, and there's almost always a path through open source and custom build that does an infinitely better job. But, in the case of KDB, whilst it was pressed upon me in a former role of mine, I found it to be an _extremely_ impressive piece of tech once I got over the initial hurdles (my biases, and basics of Q). It's not without its warts, but the amount of power and expressivity it packs into a small number if chars is quite mind blowing. I watched other teams on flavour of the month tech struggle with queries over "millions" of rows, whilst our stack routinely served complex queries over billions of rows.
- ArtWomb 7y ago
  kdb/q looks interesting! My main concern in suggesting cloud functions is COST and portability. It will always save versus running something like AresDB on a GPU cluster. Performance is within typical cloud latencies so for generating trading statistics for humans its acceptable. Cache data in memory (using multiple buffers). Cloud functions can be called synchronously. And all eventing is handled via calling goroutine
- dfischer 7y ago
  What’s the insertion capability on the freeware version? Does it handle event time ordering well for distributed injestion? Appreciate the push. It does look interesting. I like the idea of working with array native type construct as the mental model is very close to analyzing time series data for a strategy.
  - srazzaque 7y ago
    Ingest - the free version as I'm sure you're aware is 32bit only. So it is limited in how much data it can address. But the /rate/ of ingest I'd imagine is close to that of the 64bit, with the caveat that you'll hit the ceiling sooner (untested). The system I was working with was ingesting in the order of 10s of millions of rows per day from multiple sources. This was obviously the 64bit version. Also if you do plan on using it for anything beyond tinkering and personal research I'd review the licensing. I'm not too sure what is and isn't allowed. Time ordering - yes. The typical architecture of a production system in KDB will have something called a Tickerplant (TP), a process through which all updates must traverse. This will stamp its own timestamp on all the updates it receives. So for time ordering - you can (1) use the TP timestamps, and/or (2) have your source nodes also provide a timestamp in their own column (eg sourceNodeTimestamp), and then you've got some means of detecting skew across your systems with comparisons to the TP timestamps. You could then order your data by whatever timestamp you want to use in your queries.

srazzaque 7y ago

If you are interested in learning how to roll a timeseries database, and if you're not already familiar with it, I'd second the suggestions to try KDB/Q. Even if only to be inspired and understand what "good" looks like in this space. I say this as someone who is generally averse to proprietary enterprise software. More often than not, it's a horrible pile of crap that's sold on the golf course, and there's almost always a path through open source and custom build that does an infinitely better job. But, in the case of KDB, whilst it was pressed upon me in a former role of mine, I found it to be an _extremely_ impressive piece of tech once I got over the initial hurdles (my biases, and basics of Q). It's not without its warts, but the amount of power and expressivity it packs into a small number if chars is quite mind blowing. I watched other teams on flavour of the month tech struggle with queries over "millions" of rows, whilst our stack routinely served complex queries over billions of rows.

ArtWomb 7y ago
kdb/q looks interesting! My main concern in suggesting cloud functions is COST and portability. It will always save versus running something like AresDB on a GPU cluster. Performance is within typical cloud latencies so for generating trading statistics for humans its acceptable. Cache data in memory (using multiple buffers). Cloud functions can be called synchronously. And all eventing is handled via calling goroutine
dfischer 7y ago
What’s the insertion capability on the freeware version? Does it handle event time ordering well for distributed injestion? Appreciate the push. It does look interesting. I like the idea of working with array native type construct as the mental model is very close to analyzing time series data for a strategy.
- srazzaque 7y ago
  Ingest - the free version as I'm sure you're aware is 32bit only. So it is limited in how much data it can address. But the /rate/ of ingest I'd imagine is close to that of the 64bit, with the caveat that you'll hit the ceiling sooner (untested). The system I was working with was ingesting in the order of 10s of millions of rows per day from multiple sources. This was obviously the 64bit version. Also if you do plan on using it for anything beyond tinkering and personal research I'd review the licensing. I'm not too sure what is and isn't allowed. Time ordering - yes. The typical architecture of a production system in KDB will have something called a Tickerplant (TP), a process through which all updates must traverse. This will stamp its own timestamp on all the updates it receives. So for time ordering - you can (1) use the TP timestamps, and/or (2) have your source nodes also provide a timestamp in their own column (eg sourceNodeTimestamp), and then you've got some means of detecting skew across your systems with comparisons to the TP timestamps. You could then order your data by whatever timestamp you want to use in your queries.