5 ms·

How Linux mastered Wall Street

83 points by abennett 15y ago

paperwork 15y ago
Frankly, I'm often surprised that how much of wall street is still windows based. A year or two ago I spoke to a co-location vendor who told me that most of their customers run their automated trading strategies on windows machines. A friend is currently working with a very popular market data vendor who only provides windows API. Some of their customers use linux, but they are forced to use WINE! The vendor has no plans to provide linux based api, something more portable like java or even straight tcp/ip message specification.
- mynegation 15y ago
  A lot of this has to do with Excel. Average finance guy lives in Excel and no, libre/open office is not a replacement as they will not run a huge number of macros and DLLs connecting spreadsheets to external systems. Quant guys are often locked in matlab world and again - octave on Linux cannot be used due to incompatibilities.
  - paperwork 15y ago
    I forgot about Excel. I think people outside of wall street will be surprised at this. I, myself, used a visual basic app to do some trading (not visual basic.net, VB!) as recently as a couple of years ago :). Even in the linux world, PERL is used by many quants for their analytics...not just to parse files, but as an alternative to R, matlab, etc.
    - pnathan 15y ago
      It's Perl, not PERL. /sigh. Also, Perl is A Real Language, unlike.. ah, >.>, Excel.
      - paperwork 15y ago
        Let me switch sides for a moment and defend some of what happens here. In certain instances, Excel is a very good tool. In fact, if Excel's performance could be improved, it would become the best tool for some scenarios. For example, you need to manage a stream of data. You have prices coming in and you need to publish another set of prices going out (perhaps you are trying to peg quotes in the market off another set of quotes...details don't matter). Use Excel's RTD to put incoming prices into some cells. Calculate the outgoing price by typing out a simple formula which references the original cells. All other commonly used languages are more batch oriented, rather than stream oriented. In other languages, the code will implement some sort of call-back method to handle incoming data. This program will need some sort of "main" method. Perhaps the program will need to be compiled. Certainly the user will have to know how to run it. A non-programmer will obviously pick the excel solution (and many programmers will as well). The financial industry does lots of things ass backwards -- a few of them really do make sense :)
        eru 15y ago
        You sound like you might be interested in a paper about "Improving the world's most popular functional language: user-defined functions in Excel" (http://research.microsoft.com/en-us/um/people/simonpj/papers/excel/index.htm http://research.microsoft.com/en-us/um/people/simonpj/papers...).
        dmboyd 15y ago
        I came accross this recently, along with some of the other papers guys from microsoft have put out about implementing functional languages within excel. I have to remind myself that these came out prior to the 2007 & 2010 versions of excel, and they ignored the most important recommendations and went with UI based "improvements" instead. These guys at Microsoft Cambridge Research are wasted as a talent, Its as if MSFT have "bought" them as a way of keeping ideas that are disruptive to their current streams of income under control.
        shriphani 15y ago
  - sparky 15y ago
    Though there is no drop-in Matlab replacement, at least there are native versions of Matlab for OS X and Linux. The only incompatibility that would arise there is if some underlying third-party .mex files used Windows-only APIs.
- veyron 15y ago
  To be fair there are a lot of high performance windows tricks. They involve digging through the DDK and other obscure places to find documentation, but it's not impossible to build performant windows solutions. The open nature and documentation of Linux internals makes it easy to hire commodity developers on the cheap (relatively speaking), which I suspect is the real underlying trend.
  - archangel_one 15y ago
    I'm not convinced that it's really cheaper to hire (good) Linux devs for this kind of work, and even if it were I doubt that's their biggest concern. A bunch of "commodity" developers wouldn't be able to build a complex real-time system like that and meet those targets - look what happened to the London Stock Exchange a couple of years back when they tried that. It seems much more likely to me that the real underlying driver is the level of customisability they need. They're talking about making scheduler changes to the kernel and building the whole OS themselves - that's a level of control Windows (or any other closed-source option) doesn't offer.
joshaidan 15y ago
"As late as 2007, Wall Street exchanges were still largely run on Unix, such as HP's AIX and Sun Microsystems' Solaris." Isn't AIX developed by IBM? HP maintains HP-UX if I'm not mistaken.
deleted 15y ago
[deleted]
deleted 15y ago
[deleted]
sigil 15y ago
Ahem. I know of at least one HFT in NYC that's killing it with FreeBSD.
- signa11 15y ago
  is it to do with this : http://info.iet.unipi.it/~luigi/netmap/ http://info.iet.unipi.it/~luigi/netmap/ ?
- akronim 15y ago
  If you read the article you'll see it's talking about exchanges, not those trading with them.
  - sigil 15y ago
    Not so: > "The trading shops saw that the lowest-latency solutions would only be possible with Linux," Lameter said. "The older Unixes couldn't move as fast as Linux did."

83 points by abennett 15y ago

Frankly, I'm often surprised that how much of wall street is still windows based. A year or two ago I spoke to a co-location vendor who told me that most of their customers run their automated trading strategies on windows machines. A friend is currently working with a very popular market data vendor who only provides windows API. Some of their customers use linux, but they are forced to use WINE! The vendor has no plans to provide linux based api, something more portable like java or even straight tcp/ip message specification.

mynegation 15y ago
A lot of this has to do with Excel. Average finance guy lives in Excel and no, libre/open office is not a replacement as they will not run a huge number of macros and DLLs connecting spreadsheets to external systems. Quant guys are often locked in matlab world and again - octave on Linux cannot be used due to incompatibilities.
- paperwork 15y ago
  I forgot about Excel. I think people outside of wall street will be surprised at this. I, myself, used a visual basic app to do some trading (not visual basic.net, VB!) as recently as a couple of years ago :). Even in the linux world, PERL is used by many quants for their analytics...not just to parse files, but as an alternative to R, matlab, etc.
  - pnathan 15y ago
    It's Perl, not PERL. /sigh. Also, Perl is A Real Language, unlike.. ah, >.>, Excel.
    - paperwork 15y ago
      Let me switch sides for a moment and defend some of what happens here. In certain instances, Excel is a very good tool. In fact, if Excel's performance could be improved, it would become the best tool for some scenarios. For example, you need to manage a stream of data. You have prices coming in and you need to publish another set of prices going out (perhaps you are trying to peg quotes in the market off another set of quotes...details don't matter). Use Excel's RTD to put incoming prices into some cells. Calculate the outgoing price by typing out a simple formula which references the original cells. All other commonly used languages are more batch oriented, rather than stream oriented. In other languages, the code will implement some sort of call-back method to handle incoming data. This program will need some sort of "main" method. Perhaps the program will need to be compiled. Certainly the user will have to know how to run it. A non-programmer will obviously pick the excel solution (and many programmers will as well). The financial industry does lots of things ass backwards -- a few of them really do make sense :)
      - eru 15y ago
        You sound like you might be interested in a paper about "Improving the world's most popular functional language: user-defined functions in Excel" (http://research.microsoft.com/en-us/um/people/simonpj/papers/excel/index.htm http://research.microsoft.com/en-us/um/people/simonpj/papers...).
        dmboyd 15y ago
        I came accross this recently, along with some of the other papers guys from microsoft have put out about implementing functional languages within excel. I have to remind myself that these came out prior to the 2007 & 2010 versions of excel, and they ignored the most important recommendations and went with UI based "improvements" instead. These guys at Microsoft Cambridge Research are wasted as a talent, Its as if MSFT have "bought" them as a way of keeping ideas that are disruptive to their current streams of income under control.
        shriphani 15y ago
- sparky 15y ago
  Though there is no drop-in Matlab replacement, at least there are native versions of Matlab for OS X and Linux. The only incompatibility that would arise there is if some underlying third-party .mex files used Windows-only APIs.
veyron 15y ago
To be fair there are a lot of high performance windows tricks. They involve digging through the DDK and other obscure places to find documentation, but it's not impossible to build performant windows solutions. The open nature and documentation of Linux internals makes it easy to hire commodity developers on the cheap (relatively speaking), which I suspect is the real underlying trend.
- archangel_one 15y ago
  I'm not convinced that it's really cheaper to hire (good) Linux devs for this kind of work, and even if it were I doubt that's their biggest concern. A bunch of "commodity" developers wouldn't be able to build a complex real-time system like that and meet those targets - look what happened to the London Stock Exchange a couple of years back when they tried that. It seems much more likely to me that the real underlying driver is the level of customisability they need. They're talking about making scheduler changes to the kernel and building the whole OS themselves - that's a level of control Windows (or any other closed-source option) doesn't offer.

joshaidan 15y ago

"As late as 2007, Wall Street exchanges were still largely run on Unix, such as HP's AIX and Sun Microsystems' Solaris." Isn't AIX developed by IBM? HP maintains HP-UX if I'm not mistaken.

deleted 15y ago

[deleted]

deleted 15y ago

[deleted]

sigil 15y ago

Ahem. I know of at least one HFT in NYC that's killing it with FreeBSD.

signa11 15y ago
is it to do with this : http://info.iet.unipi.it/~luigi/netmap/ http://info.iet.unipi.it/~luigi/netmap/ ?
akronim 15y ago
If you read the article you'll see it's talking about exchanges, not those trading with them.
- sigil 15y ago
  Not so: > "The trading shops saw that the lowest-latency solutions would only be possible with Linux," Lameter said. "The older Unixes couldn't move as fast as Linux did."