4 ms·

The first section of this which relates HFT to one-pass algorithms is kind of useless. Online estimation algorithms are much more broadly applicable than that,

by abeppu 12y ago

The first section of this which relates HFT to one-pass algorithms is kind of useless. Online estimation algorithms are much more broadly applicable than that, and I'd expect that their relevance to HFT would be immediately obvious. I've never worked in HFT, but I've probably written the online variance algorithm at 3 of 4 companies I've worked. If you care about metrics such as "how quickly are each of my customers sending me requests?", then this stuff is relevant to you. But more to the point, is there anyone in these fields who doesn't know this stuff? At a basic level, the things that you can do a really good job estimating in an online way are closely related to sufficient statistics, which every undergrad stats course should cover. And hopefully anyone actually involved in writing HFT systems has way more stats background than that -- right?

kasey_junk 12y ago
a) there is a pernicious and wide spread myth about the elite status of people working in HFT. My experience is that it is largely the same as any other technical profession. That is to say, it is no more or less likely for people in HFT to know about online estimation than any where else. b) Given the reactionary view that many HFT participants have to information sharing, widely known/documented algorithms are often thought of as secret sauce and are frequently not talked about. c) I agree with your sentiment specifically to one-pass algorithms being not terribly novel but am very happy that people with HFT backgrounds are publishing anything at all as it will hopefully help to remove the backwards thinking around secrecy so prevalent in the industry.
- vasilipupkin 12y ago
  regarding a), not all technical professions require deep knowledge of massive data processing and time series modeling, so this statement is misleading regarding b), the details of how exactly to use the algorithms are the secret sauce and those details matter.
  - dllthomas 12y ago
    I think people are reading your comment as "they need to know these things, therefore they are elite" - but I think you rather meant "it's not about being elite, just that they have to know XYZ so they are more likely to know W".
    - vasilipupkin 12y ago
      yes, exactly. But, the reality is that top HFT firms at least try to hire 10x programmers. So call them elite or not, but the calibre is high
      - dllthomas 12y ago
        Eh, I don't think that follows. Top firms in many sectors try to hire elite programmers. Do they succeed? Further, how much of the industry is made up of hires of top firms? I have no particular position on the elite status of those working in HFT. Certainly there are some smart people there, but I don't know how to generalize. I don't have anything like a representative sample of either HFT or programming more broadly to compare, despite having worked in HFT and out of it.
        vasilipupkin 12y ago
        Top firms in HFT try to hire elite programmers and they tend to succeed. Think of it this way, if you are hiring hundreds or thousands of people a year, then it would be extremely difficult to make sure most of them are elite. Plus, chances are your projects don't require it anyway. Many projects need good talent, but not elite. But if you are hiring 1-20 people per year, then making sure that at least some of them are elite is not terribly difficult
        dllthomas 12y ago
        I can see the reasoning, I've just little confidence other dynamics don't dominate. I'm not saying I'm confident they do; and even if they do it's certainly possible for them to push the matter in either direction. But I'm saying that I don't know, that nothing you've said is anything I haven't considered, and that it's likely you also know less than you seem to think. And again, I'm saying that as someone who has worked in HFT.
tptacek 12y ago
I'm a systems programmer with a pretty primitive intuitive sense for maths. I "agree" with you, in the sense that the algorithms this article mentioned were intuitively obvious to me and thus seemed like they couldn't be sophisticated. What's the state of the art for online estimators? I've been playing with simulations of market maker algorithms from the literature, and simple Bayesian stuff and MLEs are all I've really seen, which makes me think that the interesting stuff isn't in the literature.
- cdavid 12y ago
  There is a lot of literature, but it may not be easy to find out, and different domains use different names for it. One keyword to look for is 'stochastic approximation', with a long (> 50 years) history in engineering and stats. Another one is 'stochastic gradient', which got a resurgence following the whole deep learning craze. There is also 'particle filtering' in signal processing. One generic article I may recommend is http://leon.bottou.org/publications/pdf/online-1998.pdf http://leon.bottou.org/publications/pdf/online-1998.pdf, it does not require much math knowledge, but that may be less practical than what you want. The 'onepass' keyword mentioned in the original article is another keyword to look for (see e.g. interesting work to do Monte-Carlo based linear algebra operations for very large matrices, this used to be a reference, not sure what's the state of the art is: http://cs-www.cs.yale.edu/homes/mmahoney/pubs/matrix1_SICOMP.pdf http://cs-www.cs.yale.edu/homes/mmahoney/pubs/matrix1_SICOMP...)
  - hcrisp 12y ago
    > ... different domains use different names ... I agree. In engineering, many of these algorithms can be coded using digital filters. Here's the exponential weighted variance: import scipy.signal as dsp import numpy as np b = np.array([alpha]) a = np.array([1., -(1 - alpha)]) # Compute exponential weighted average signal_ = dsp.lfilter(b, a, signal) # Compute exponential weighted variance ewv = dsp.lfilter(b, a, (signal - signal_)**2)
- minimax 12y ago
  There is another article in this same edition of the ACM Queue. http://queue.acm.org/detail.cfm?id=2536492 http://queue.acm.org/detail.cfm?id=2536492 It goes a bit further into the mechanics of market maker profitability. Does your simulator reflect what your queue position would be in a real electronic market? Does your market making algorithm have logic try to scratch out when it looks like the market is going to move against its position?
wglb 12y ago
The particular emphasis on one-pass algorithms might seem self-evident, but the thread running through all of this is an engineering challenge that is very relevant to HFT. If your algorithm takes 60ms to evaluate your current position on each tick, by the time you are finished, the market has moved to the next zip code. Thus, to keep up with the market, you might find yourself rewriting the network stack for better response time.

by abeppu 12y ago

kasey_junk 12y ago

a) there is a pernicious and wide spread myth about the elite status of people working in HFT. My experience is that it is largely the same as any other technical profession. That is to say, it is no more or less likely for people in HFT to know about online estimation than any where else. b) Given the reactionary view that many HFT participants have to information sharing, widely known/documented algorithms are often thought of as secret sauce and are frequently not talked about. c) I agree with your sentiment specifically to one-pass algorithms being not terribly novel but am very happy that people with HFT backgrounds are publishing anything at all as it will hopefully help to remove the backwards thinking around secrecy so prevalent in the industry.

vasilipupkin 12y ago
regarding a), not all technical professions require deep knowledge of massive data processing and time series modeling, so this statement is misleading regarding b), the details of how exactly to use the algorithms are the secret sauce and those details matter.
- dllthomas 12y ago
  I think people are reading your comment as "they need to know these things, therefore they are elite" - but I think you rather meant "it's not about being elite, just that they have to know XYZ so they are more likely to know W".
  - vasilipupkin 12y ago
    yes, exactly. But, the reality is that top HFT firms at least try to hire 10x programmers. So call them elite or not, but the calibre is high
    - dllthomas 12y ago
      Eh, I don't think that follows. Top firms in many sectors try to hire elite programmers. Do they succeed? Further, how much of the industry is made up of hires of top firms? I have no particular position on the elite status of those working in HFT. Certainly there are some smart people there, but I don't know how to generalize. I don't have anything like a representative sample of either HFT or programming more broadly to compare, despite having worked in HFT and out of it.
      - vasilipupkin 12y ago
        Top firms in HFT try to hire elite programmers and they tend to succeed. Think of it this way, if you are hiring hundreds or thousands of people a year, then it would be extremely difficult to make sure most of them are elite. Plus, chances are your projects don't require it anyway. Many projects need good talent, but not elite. But if you are hiring 1-20 people per year, then making sure that at least some of them are elite is not terribly difficult
        dllthomas 12y ago
        I can see the reasoning, I've just little confidence other dynamics don't dominate. I'm not saying I'm confident they do; and even if they do it's certainly possible for them to push the matter in either direction. But I'm saying that I don't know, that nothing you've said is anything I haven't considered, and that it's likely you also know less than you seem to think. And again, I'm saying that as someone who has worked in HFT.

tptacek 12y ago

I'm a systems programmer with a pretty primitive intuitive sense for maths. I "agree" with you, in the sense that the algorithms this article mentioned were intuitively obvious to me and thus seemed like they couldn't be sophisticated. What's the state of the art for online estimators? I've been playing with simulations of market maker algorithms from the literature, and simple Bayesian stuff and MLEs are all I've really seen, which makes me think that the interesting stuff isn't in the literature.

cdavid 12y ago
There is a lot of literature, but it may not be easy to find out, and different domains use different names for it. One keyword to look for is 'stochastic approximation', with a long (> 50 years) history in engineering and stats. Another one is 'stochastic gradient', which got a resurgence following the whole deep learning craze. There is also 'particle filtering' in signal processing. One generic article I may recommend is http://leon.bottou.org/publications/pdf/online-1998.pdf http://leon.bottou.org/publications/pdf/online-1998.pdf, it does not require much math knowledge, but that may be less practical than what you want. The 'onepass' keyword mentioned in the original article is another keyword to look for (see e.g. interesting work to do Monte-Carlo based linear algebra operations for very large matrices, this used to be a reference, not sure what's the state of the art is: http://cs-www.cs.yale.edu/homes/mmahoney/pubs/matrix1_SICOMP.pdf http://cs-www.cs.yale.edu/homes/mmahoney/pubs/matrix1_SICOMP...)
- hcrisp 12y ago
  > ... different domains use different names ... I agree. In engineering, many of these algorithms can be coded using digital filters. Here's the exponential weighted variance: import scipy.signal as dsp import numpy as np b = np.array([alpha]) a = np.array([1., -(1 - alpha)]) # Compute exponential weighted average signal_ = dsp.lfilter(b, a, signal) # Compute exponential weighted variance ewv = dsp.lfilter(b, a, (signal - signal_)**2)
minimax 12y ago
There is another article in this same edition of the ACM Queue. http://queue.acm.org/detail.cfm?id=2536492 http://queue.acm.org/detail.cfm?id=2536492 It goes a bit further into the mechanics of market maker profitability. Does your simulator reflect what your queue position would be in a real electronic market? Does your market making algorithm have logic try to scratch out when it looks like the market is going to move against its position?

wglb 12y ago

The particular emphasis on one-pass algorithms might seem self-evident, but the thread running through all of this is an engineering challenge that is very relevant to HFT. If your algorithm takes 60ms to evaluate your current position on each tick, by the time you are finished, the market has moved to the next zip code. Thus, to keep up with the market, you might find yourself rewriting the network stack for better response time.