10 ms·

the technology used by HFT firms to make markets in microseconds makes the web development world look like children. pubsub is the hot new thing here in the va

the technology used by HFT firms to make markets in microseconds makes the web development world look like children. pubsub is the hot new thing here in the valley - welcome to wall street, circa ten years ago. asynchronous is the new synchronous? what is that - a new album by smash mouth? if you completely ignore HFT, you'd think there would be way more programmers and mathematicians available for hire.

zanny 14y ago
> if you completely ignore HFT, you'd think there would be way more programmers and mathematicians available for hire. Can you clarify this? My interpretation is that if you discount mathematicians / programmers working in HFT, you would expect more of them to be in the market for jobs?
- MarkPNeyer 14y ago
  sorry, my argument was very vague. if you completely ignore HFT, you will ignore that there are many very qualified people working in that field, and therefore overestimate the number of those people available for you to hire.
  - zanny 14y ago
    Thanks for the clarification!
dasil003 14y ago
On the other hand, web development is solving many many more problems for many many more people. HFT firms are solving the single problem of making their owners rich with the authoritatively, hand-wringingly repeated justification about how critically important that liquidity is to everyone and their grandmother.
- MarkPNeyer 14y ago
  right, that's why i left HFT. i was just trying (inarticulately) to make the argument that even if the "only" cost of HFT is the massive amount of talent going into that field, it's still a huge cost.
Aloisius 14y ago
As someone who has designed, built and ran systems that operated at a scale that made the software that HFT firms look like toys and built several web companies, I can honestly say that you have absolutely no idea what you are talking about. Web development and low-level systems engineering are entirely different and neither one is particularly harder than the other. If anything, both are simply tedious. And pubsub was old by the time wall street got a hold of it.
- MarkPNeyer 14y ago
  yes, pubsub is an old pattern. what HFT experience do you have that makes you think of those systems as toys? a webpage that responds to requests in 10 milliseconds is fast. a trading engine that responds to requests in 10 milliseconds is a joke.
  - true_religion 14y ago
    It's not as if many webpages couldn't be made to respond to faster. Not to brag, but I've built trading engines in the past year that can process 20 years worth of data in less than a ms. Now I'm a founder at your usually derrided photos-sharing start up. Sure we could make most requests come out of the server in less than 2ms, but what of it? It's still going to take the user upwards of 1.5 seconds to load all of the content, and at least 100ms spent on the wire. Working at a consumer level startup, there's no point in optimizing the server yet when network costs are so high, and there's no point in improving the network when there's no ROI in it. ---- Answering to his question: > what HFT experience do you have that makes you think of those systems as toys? I'd imagine that massive distributed systems are more complex to think through that HFT systems because the later are more monolithic due to speed constraints.
    - MarkPNeyer 14y ago
      HFT systems _are_ massive distributed systems. where i worked - a small shop with 10 engineers - we had 350 machines running in 5 different data centers in chicago, new jersey, and korea. each machine ran around 100 different processes, which were coordinated using a combination of several different pubsub systems: - in house reliable tcp-based broker mediated (we called it pubsub3 ) - in house unrelaiable ip-multicast system (i named it blue ray because i thought that sounded cool) - 3rd party implementation of a PGM derivative that we licensed for god knows how much the engines subscribed to various topics on the system, processed market ticks (sent by marketdata publishers over blue ray) and fired orders at whatever instrument they were handling the risk engines modeled risk scenarios inresponse to trading shifts, powering through the risk models our quant trader wrote in c. we had a 'calcparser' system that constructed a DAG rerpresenting the AST of the c code and distributed the DAG across several machines. shit was nuts. i miss it.

zanny 14y ago

> if you completely ignore HFT, you'd think there would be way more programmers and mathematicians available for hire. Can you clarify this? My interpretation is that if you discount mathematicians / programmers working in HFT, you would expect more of them to be in the market for jobs?

MarkPNeyer 14y ago
sorry, my argument was very vague. if you completely ignore HFT, you will ignore that there are many very qualified people working in that field, and therefore overestimate the number of those people available for you to hire.
- zanny 14y ago
  Thanks for the clarification!

dasil003 14y ago

On the other hand, web development is solving many many more problems for many many more people. HFT firms are solving the single problem of making their owners rich with the authoritatively, hand-wringingly repeated justification about how critically important that liquidity is to everyone and their grandmother.

MarkPNeyer 14y ago
right, that's why i left HFT. i was just trying (inarticulately) to make the argument that even if the "only" cost of HFT is the massive amount of talent going into that field, it's still a huge cost.

Aloisius 14y ago

As someone who has designed, built and ran systems that operated at a scale that made the software that HFT firms look like toys and built several web companies, I can honestly say that you have absolutely no idea what you are talking about. Web development and low-level systems engineering are entirely different and neither one is particularly harder than the other. If anything, both are simply tedious. And pubsub was old by the time wall street got a hold of it.

MarkPNeyer 14y ago
yes, pubsub is an old pattern. what HFT experience do you have that makes you think of those systems as toys? a webpage that responds to requests in 10 milliseconds is fast. a trading engine that responds to requests in 10 milliseconds is a joke.
- true_religion 14y ago
  It's not as if many webpages couldn't be made to respond to faster. Not to brag, but I've built trading engines in the past year that can process 20 years worth of data in less than a ms. Now I'm a founder at your usually derrided photos-sharing start up. Sure we could make most requests come out of the server in less than 2ms, but what of it? It's still going to take the user upwards of 1.5 seconds to load all of the content, and at least 100ms spent on the wire. Working at a consumer level startup, there's no point in optimizing the server yet when network costs are so high, and there's no point in improving the network when there's no ROI in it. ---- Answering to his question: > what HFT experience do you have that makes you think of those systems as toys? I'd imagine that massive distributed systems are more complex to think through that HFT systems because the later are more monolithic due to speed constraints.
  - MarkPNeyer 14y ago
    HFT systems _are_ massive distributed systems. where i worked - a small shop with 10 engineers - we had 350 machines running in 5 different data centers in chicago, new jersey, and korea. each machine ran around 100 different processes, which were coordinated using a combination of several different pubsub systems: - in house reliable tcp-based broker mediated (we called it pubsub3 ) - in house unrelaiable ip-multicast system (i named it blue ray because i thought that sounded cool) - 3rd party implementation of a PGM derivative that we licensed for god knows how much the engines subscribed to various topics on the system, processed market ticks (sent by marketdata publishers over blue ray) and fired orders at whatever instrument they were handling the risk engines modeled risk scenarios inresponse to trading shifts, powering through the risk models our quant trader wrote in c. we had a 'calcparser' system that constructed a DAG rerpresenting the AST of the c code and distributed the DAG across several machines. shit was nuts. i miss it.