3 ms·

Thanks for the feedback, really appreciate it. 1. Yes every message has also local timestamp (100 ns precision). 2. Currently it's around 4-5TB of compressed

Thanks for the feedback, really appreciate it. 1. Yes every message has also local timestamp (100 ns precision). 2. Currently it's around 4-5TB of compressed data (so 25-35TB uncompressed - need to check to be sure). Indeed pricing is supposed to be very affordable as it's targeted at independent algo traders so without spending huge amounts of $$$ they can have good data to backtest on professional level (arguably if you can call crypto trading professional as some argue). Happy to provide you with the discount, please get in touch with me via email if interested. Edit: some obsolete info redacted

1996 7y ago
Your timestamping approach is interesting. It is well known that exchanges like bitmex suffer from special latency issues. People pay very well to study that and provide mitigation. Many approaches can be done. But running on a single computer, let alone a single VM, is very dangerous. I worked for a company that offered multiple timestamps + hardware timestamping for accuracy. A VM will cause too much latency. Even very well configured, your jitter will be above 100ns by several orders of magnitude. So be very careful about considering your extra timestamp as authoritative. You are spot on the other challenge: database. The person who focused on that achieved impressive results using a special hardware + software mix.
lopatin 7y ago
Thanks for the details. Flat files in the cloud, and heavy caching on the client sounds like the most effective way to go about it to me too. I work at a company that collects data like this for the traditional derivatives markets and fwiw about half of our data is in S3 too. The other half, Dynamo. I appreciate you making this accessible to indie algo traders, it's def what we need. Will keep an eye on this.

3 ms·

Thanks for the feedback, really appreciate it. 1. Yes every message has also local timestamp (100 ns precision). 2. Currently it's around 4-5TB of compressed

by tardis_thad 7y ago

1996 7y ago
Your timestamping approach is interesting. It is well known that exchanges like bitmex suffer from special latency issues. People pay very well to study that and provide mitigation. Many approaches can be done. But running on a single computer, let alone a single VM, is very dangerous. I worked for a company that offered multiple timestamps + hardware timestamping for accuracy. A VM will cause too much latency. Even very well configured, your jitter will be above 100ns by several orders of magnitude. So be very careful about considering your extra timestamp as authoritative. You are spot on the other challenge: database. The person who focused on that achieved impressive results using a special hardware + software mix.
lopatin 7y ago
Thanks for the details. Flat files in the cloud, and heavy caching on the client sounds like the most effective way to go about it to me too. I work at a company that collects data like this for the traditional derivatives markets and fwiw about half of our data is in S3 too. The other half, Dynamo. I appreciate you making this accessible to indie algo traders, it's def what we need. Will keep an eye on this.