4 ms·

>Nasdaq runs more than 40 different algorithms, using about 35,000 parameters, to look for market abuse and manipulation in real time. I laughed when I read th

>Nasdaq runs more than 40 different algorithms, using about 35,000 parameters, to look for market abuse and manipulation in real time. I laughed when I read this. 40 whole algorithms!

TheSpiceIsLife 7y ago
Isn’t the more intense number the 35,000 parameters? Why pick on the 40 algorithms?
- account73466 7y ago
  If you try to optimize them and you deal with a black-box, then yes.
- asdfasgasdgasdg 7y ago
  Nasdaq has about 3500 listed companies, so that's only ten parameters per company. Not that many. If you're talking about running the algorithm per market participant then it's an even weirder claim.
  - account73466 7y ago
    So you just assumed that parameters are defined company-wise?
    - asdfasgasdgasdg 7y ago
      I wouldn't say I assumed anything. I would just say that the statement about number of parameters and algorithms is incoherent, or at least so vastly underspecified as to communicate no meaning.
    - weego 7y ago
      Nasdaq employees drink 15,000 cups of coffee a day, that's close to 2 parameters per cup
  - projektfu 7y ago
    Yes but if Nasdaq's total market cap is $30 trillion, that's about $750 billion per algorithm. Surely we need more algorithms.
    - Ericson2314 7y ago
      No billion left behind
- jklein11 7y ago
  I think the reason why I laughed is that I can't understand what 40 algorithms actually means. It probably takes more than 40 algorithms to accept a request at this url https://news.ycombinator.com/item?id=21730608 https://news.ycombinator.com/item?id=21730608 and return the response that you are seeing on your screen. That bullet in the article added no value to me as the reader.
  - braindeath 7y ago
    As far as imprecise journalism (let alone blatant inaccuracy goes) this is pretty inoffensive IMHO. It’s not even that far off. Algorithms can be composed. When we publish a paper on a complete ML application, the whole thing is generally referred to as “the algorithm” because it can be documented as precise set of steps from initial raw inputs to some kind of output. Just because there are literally thousands of applications of sorting and searching, matrix multiplies and what have you buried under the process doesn’t make it unclear what is being referred to.
    - adatavizguy 7y ago
      Almost all algorithms are composition of algorithms. Probably a good way to define a singular algorithm is the point a human has to be the source of its input. A self driving car is a composition of thousands of algorithms by definition -- a process that step by step calculates with an input and output which is every single method in all its code. However, for sake of conversation in the number of complete autonomous systems consider the car to be a single algorithm defined by the point a human has to be the originating input at the interface where the key gets turned to turn it on. I think this definition will be important in law because it is where we will define responsibility for actions of autonomous things.
  - craftinator 7y ago
    I'm guessing you are more of an expert on the technology involved than the journalist writing this article, and thus not the target audience. Maybe look for other areas of value in it, and skip that particular bullet point. I imagine mechanical engineers often find laughable lines in Popular Mechanics as well...
- TheSpiceIsLife 7y ago
  Sorry, I meant interesting not intense. Too late to edit now.
onlyrealcuzzo 7y ago
The algorithms probably aren't that great, and more of them would likely have diminishing returns. Adding substantially more false positives could actually be a bad thing. As someone else mentioned. The 35k parameters is skeptical. Taleb and Tversky and Kahneman have good evidence that most algorithms are better with less parameters. The more parameters, the more noise.
- nuclearnice1 7y ago
  > The more parameters, the more noise. Interesting observation. Do you have more specific references? Those three are so popular it’s hard to narrow to commentary on parameters and noise.
  - onlyrealcuzzo 7y ago
    In my opinion, it's a large part of Thinking, Fast and Slow and The Black Swan. Basically, as your signals approach infinity, your chance of finding correlation approaches 1. Taleb argues your chance of finding causation is likely to decrease.
kevingadd 7y ago
I would interpret "algorithms" here as referring to entire separate machine learning models trained differently (if not trained on entirely separate data sets). It would make sense to me that you would want to train separate models for identifying specific categories of abuse instead of training one Ultimate Model, because debugging focused models is going to be easier. That would line up with the massive number of parameters as well.
BickNowstrom 7y ago
> Nasdaq runs [https://financialservices.house.gov/uploadedfiles/hhrg-116-ba00-wstate-rejsjom-20191206.pdf https://financialservices.house.gov/uploadedfiles/hhrg-116-b...] more than 40 different algorithms, using about 35,000 parameters, to look for market abuse and manipulation in real time. Click link: > Nasdaq North America Surveillance team monitors 3 equity markets, 6 options markets and one futures market with real-time surveillance and post-trade surveillance of unusual market activity. The surveillance department is monitoring the markets for Insider Trading, Fraud and Manipulation, including manipulation through trading—pump and dump—and order book manipulation—spoofing and layering, as well as handling events in the market such as clearly erroneous transactions. > The surveillance program today is using algorithmic coding to detect unusual market behavior running over 40 different algorithms in real-time, looking for market abuse and manipulation. The patterns have sophisticated algorithms that use approximately 35,000 parameters. In addition to real-time surveillance, there are over 150 patterns covering post trade surveillance, which are used to identify a wide range of potential misconduct. > The activity is monitored across equity and options markets, with some market-specific alerts and some alerts encompassing data from all markets. So they have 40 different predictive models running in production (a lot!), and 150 hand-written (or single-pattern) rules. They also employ active learning with experts and this reduces their training data needs with 95%. 40 models using 35000 features, is 875 features per model on average. All this bickering is typical.
sgt101 7y ago
No one who has got a machine learning algorithm into production in an application with substantial capital at risk would have found the 40 number low. Anyone, can demo, POC, prototype... but prod... that is a big persons game.

>Nasdaq runs more than 40 different algorithms, using about 35,000 parameters, to look for market abuse and manipulation in real time. I laughed when I read this. 40 whole algorithms!

TheSpiceIsLife 7y ago

Isn’t the more intense number the 35,000 parameters? Why pick on the 40 algorithms?

account73466 7y ago
If you try to optimize them and you deal with a black-box, then yes.
asdfasgasdgasdg 7y ago
Nasdaq has about 3500 listed companies, so that's only ten parameters per company. Not that many. If you're talking about running the algorithm per market participant then it's an even weirder claim.
- account73466 7y ago
  So you just assumed that parameters are defined company-wise?
  - asdfasgasdgasdg 7y ago
    I wouldn't say I assumed anything. I would just say that the statement about number of parameters and algorithms is incoherent, or at least so vastly underspecified as to communicate no meaning.
  - weego 7y ago
    Nasdaq employees drink 15,000 cups of coffee a day, that's close to 2 parameters per cup
- projektfu 7y ago
  Yes but if Nasdaq's total market cap is $30 trillion, that's about $750 billion per algorithm. Surely we need more algorithms.
  - Ericson2314 7y ago
    No billion left behind
jklein11 7y ago
I think the reason why I laughed is that I can't understand what 40 algorithms actually means. It probably takes more than 40 algorithms to accept a request at this url https://news.ycombinator.com/item?id=21730608 https://news.ycombinator.com/item?id=21730608 and return the response that you are seeing on your screen. That bullet in the article added no value to me as the reader.
- braindeath 7y ago
  As far as imprecise journalism (let alone blatant inaccuracy goes) this is pretty inoffensive IMHO. It’s not even that far off. Algorithms can be composed. When we publish a paper on a complete ML application, the whole thing is generally referred to as “the algorithm” because it can be documented as precise set of steps from initial raw inputs to some kind of output. Just because there are literally thousands of applications of sorting and searching, matrix multiplies and what have you buried under the process doesn’t make it unclear what is being referred to.
  - adatavizguy 7y ago
    Almost all algorithms are composition of algorithms. Probably a good way to define a singular algorithm is the point a human has to be the source of its input. A self driving car is a composition of thousands of algorithms by definition -- a process that step by step calculates with an input and output which is every single method in all its code. However, for sake of conversation in the number of complete autonomous systems consider the car to be a single algorithm defined by the point a human has to be the originating input at the interface where the key gets turned to turn it on. I think this definition will be important in law because it is where we will define responsibility for actions of autonomous things.
- craftinator 7y ago
  I'm guessing you are more of an expert on the technology involved than the journalist writing this article, and thus not the target audience. Maybe look for other areas of value in it, and skip that particular bullet point. I imagine mechanical engineers often find laughable lines in Popular Mechanics as well...
TheSpiceIsLife 7y ago
Sorry, I meant interesting not intense. Too late to edit now.

onlyrealcuzzo 7y ago

The algorithms probably aren't that great, and more of them would likely have diminishing returns. Adding substantially more false positives could actually be a bad thing. As someone else mentioned. The 35k parameters is skeptical. Taleb and Tversky and Kahneman have good evidence that most algorithms are better with less parameters. The more parameters, the more noise.

nuclearnice1 7y ago
> The more parameters, the more noise. Interesting observation. Do you have more specific references? Those three are so popular it’s hard to narrow to commentary on parameters and noise.
- onlyrealcuzzo 7y ago
  In my opinion, it's a large part of Thinking, Fast and Slow and The Black Swan. Basically, as your signals approach infinity, your chance of finding correlation approaches 1. Taleb argues your chance of finding causation is likely to decrease.

kevingadd 7y ago

I would interpret "algorithms" here as referring to entire separate machine learning models trained differently (if not trained on entirely separate data sets). It would make sense to me that you would want to train separate models for identifying specific categories of abuse instead of training one Ultimate Model, because debugging focused models is going to be easier. That would line up with the massive number of parameters as well.

BickNowstrom 7y ago

> Nasdaq runs [https://financialservices.house.gov/uploadedfiles/hhrg-116-ba00-wstate-rejsjom-20191206.pdf https://financialservices.house.gov/uploadedfiles/hhrg-116-b...] more than 40 different algorithms, using about 35,000 parameters, to look for market abuse and manipulation in real time. Click link: > Nasdaq North America Surveillance team monitors 3 equity markets, 6 options markets and one futures market with real-time surveillance and post-trade surveillance of unusual market activity. The surveillance department is monitoring the markets for Insider Trading, Fraud and Manipulation, including manipulation through trading—pump and dump—and order book manipulation—spoofing and layering, as well as handling events in the market such as clearly erroneous transactions. > The surveillance program today is using algorithmic coding to detect unusual market behavior running over 40 different algorithms in real-time, looking for market abuse and manipulation. The patterns have sophisticated algorithms that use approximately 35,000 parameters. In addition to real-time surveillance, there are over 150 patterns covering post trade surveillance, which are used to identify a wide range of potential misconduct. > The activity is monitored across equity and options markets, with some market-specific alerts and some alerts encompassing data from all markets. So they have 40 different predictive models running in production (a lot!), and 150 hand-written (or single-pattern) rules. They also employ active learning with experts and this reduces their training data needs with 95%. 40 models using 35000 features, is 875 features per model on average. All this bickering is typical.

sgt101 7y ago

No one who has got a machine learning algorithm into production in an application with substantial capital at risk would have found the 40 number low. Anyone, can demo, POC, prototype... but prod... that is a big persons game.