7 ms·

The New Gold Rush? Wall Street Wants Your Data

114 points by dwynings 10y ago

chollida1 10y ago
Great article!! And i think it can probably be summed up with the sentence, unless your product has: - a long history of data - data that has low correlation to other available data sets - a very small group of people who have access to this data( ie exclusive deals with a few funds) Then its unlikely that you will be able to sell your data for any meaningful amount of money. A few things the author wrote that I think deserve some attention..... > You do hear the occasional story where a hedge fund paid a couple of millions a year to obtain a specific data set, and occasionally more. But there’s a reasonable chance that this type of price came with some kind of exclusive. Also, those contracts also probably have a limited shelf life, as the value of a data set decays over time, as per the above. This is very true, almost certainly your data set is not worth millions a year to any specific fund. And if it is, its because you agreed to make your data set exclusive to a single fund. Each data set requires its own special mess of tools for cleaning it, integrating it, figuring out how to generate a trading signal from it, putting it into production and then once its in production constantly monitoring it not only to verify the data is good, but to make sure it actually benefits the trading system. I wrote this comment: https://news.ycombinator.com/item?id=13140352 https://news.ycombinator.com/item?id=13140352 a while ago and it got a very large number of up votes for what was essentially a rant that we're now drowing in data. Hedge funds today share alot in common with AAA video games wrt to data. You used to be able to make a game with 2 people, then it was 10, then it was 50, then 100, now its 500+ for a AAA video game. Hedge funds have the exact same issue due to the explosion of data. I know alot of funds that are skipping on data sets, due to lack of bandwidth to consume them all. > First of all, unless it’s already commoditized, you probably don’t want to sell your data to the Bloombergs of the world. Most hedge funds won't be your direct audience, they'll already have spent money to buy data from Bloomberg and Reuters. The few, say top 100, that might buy your data directly could be your market, but these funds don't exactly have huge sales teams ready to talk to you. Which means you need to hire someone who knows the landscape to sell your data. Your sales team might be the best group of 25 year old Stanford grads around but they aren't getting in the door of most quant funds without an introduction. This is the website of one of the worlds most successful funds. https://www.rentec.com/Home.action?index=true https://www.rentec.com/Home.action?index=true
- hackuser 10y ago
  > i think it can probably be summed up with the sentence Hmm ... for anyone reading it, the article covers a lot more than the specifications for selling your dataset.
- AznHisoka 10y ago
  Also just plain dumb luck. stocks dont trade based on logic like revenue growth so if Trump tweets something negative about Starbucks and the stock goes down, but your data suggested Starbucks sales went up 300% last month, you are in tough luck. this is also probably why your data is worth much much less than you think. there are very few things that correlate very strongly to stock performance. predicting human emotion is probably more important than predicting revenue growth or number of people who went inside a Starbucks last month.
  - laingc 10y ago
    If your data suggest that the stock is trading at a discount due to irrationality, then you have a perfect opportunity to buy lots of an underpriced stock.
    - aangjie 10y ago
      I think the point OP was trying to make is that, that irrationality may not be easy to capture with automated systems, therefore the value of data is limited by your trading system/algorithm's ability to detect/predict irrational human behaviour. Or atleast that's what I read of it.
      - dsacco 10y ago
        Yes, the grandparent comment makes a good point. You can have plenty of solid data and still make a losing trade. But, given enough data-informed trades that would otherwise be proditable, the law of large numbers suggests that an overall strategy will still be sound. You can afford to have the market be irrational here and there if you really are correctly forecasting earnings in advance most of the time.
hackuser 10y ago
> The good news is that hedge funds are not advertisers and don't care about particular individuals This sounds disingenuous; it's not hard for the author or anyone else to imagine how Wall Street could use personally identifiable information, and the author claims to be at least somewhat sophisticated in the 'big data' field. Also, it's an industry that has an incredible track record of fraud, even fixing the bedrock interest rates like Libor, prides itself on an anything-goes atmosphere, and disdains regulation.
- digler999 10y ago
  I could see them passing "insider consumer info" on black markets just like they pass insider information. How handy would it be for a life insurance company to know which of its customers is googling for "chest pain" or "$CANCER_TYPE symptoms". A car insurance company would love to know which drivers go to bars on saturday night, or which ones make it from point A to point B in suspiciously low amounts of time. Wall street could creep into this by saying to either industry: "let our guys crunch the numbers and let us make book on what these rates should be...because market efficiency". It'll seem like a great idea at first, the data will be "anonymized" (except for when it's not, like when that guy pays a $5k to an analyst at the insurance company for the table mapping "anonymized" customer idns to ssns. Suddenly hedge funds get really good at guessing which pools should be invested in, or which drivers should be packaged into the high-risk tier. A few years later there's a moderate scandal, a fine is levied equal to ~5% of the illicit gains made, meanwhile the funds have moved on to perpetuate their next round of fraud.
  - hackuser 10y ago
    Also, learning information about individuals: Big clients, competitors, employees, critics, journalists, politicians, etc. etc.
hackuser 10y ago
> "No man is better than a machine, and no machine is better than a man with a machine." This concept also is popular in the military AI R&D world where the human-computer combination is called a 'centaur' (based on what I've read; I'm not of that world).
rcarrigan87 10y ago
> When I was at Bloomberg, I used to cringe when startup founders would show up in hoodies, essentially destroying their credibility before the meeting even started. I understand you can't roll in looking like a slob but I have to think times have changed and the casually dressed founder wouldn't have a problem, even with Wall Street types. The cultural differences b/t the two industries are pretty interesting though...
- choward 10y ago
  People dressed up in suits lose accountability to me. It's like they are over-compensating for something.
  - zenlikethat 10y ago
    And I'm sure they are devastated by this loss of accountability, since all they got in return was a 2x - 10x multiple of income from the average programmer.
  - bogomipz 10y ago
    Does that include lawyers and financial advisors too? Would you be OK with someone in jeans and a hoodie representing you in court? Would you be OK handing your life savings over to a money manager who was wearing a t-shirt and flip flops?
    - caleblloyd 10y ago
      Only if it was Michael Burry
    - maxerickson 10y ago
      Topical PSA from the financial advice industry: https://www.youtube.com/watch?v=yJFrkNY4n1g https://www.youtube.com/watch?v=yJFrkNY4n1g If you have something of value and the people you are talking to are actually interested in making money, they will get past quite a lot. (The lawyer question is more ridiculous, they have to work with the prejudices of the judges and lawyers they interact with so only in the narrowest of circumstances would a credible attorney walk into court in jeans)
dgfgfdagasdfgfa 10y ago
> The financial services world has its own strong identity, handles massive amounts of money, and will not necessarily be in awe of your cool startup. When I was at Bloomberg, I used to cringe when startup founders would show up in hoodies, essentially destroying their credibility before the meeting even started. Who wants wall street awe? Who cares about credibility? Run a goddamn business, not a fan club. If you cared about image, you wouldn't talk about what people wear. It's completely irrelevant to why I'd use your product.
- AnimalMuppet 10y ago
  Who cares about credibility? If you're trying to sell to Wall Street, you care about credibility. If you want them to invest in your startup, you care about credibility. If they want to sell to you, wear what you please.
  - dgfgfdagasdfgfa 10y ago
    Funny, they never mentioned a link between perception and sales, so it just came out like shit talking....
- zenlikethat 10y ago
  Except that Silicon Valley is equally superficial in different ways. Consider: founder fetishism, engineering fetishism (vs. sales/marketing/product), relative ease of getting funded as a Caltech / MIT / Stanford grad, favoring growth over revenue without coherent business plan, acting like fanboys towards YC or companies such as Uber, Google, Apple, etc.
  - dgfgfdagasdfgfa 10y ago
    No argument here, startup/vc fetishism is distinctly unattractive.
- dig1 10y ago
  > Run a goddamn business,.... If you cared about image, you wouldn't talk about what people wear. Let's change perspective - what do you think, how much success will have business with web pages that looks like from 90-ties or products that looks like they are made in basement in spare time? (sure, there are exceptions, but well, they are exceptions :P). Everything is about image; if I'm going to invest in you, you better be shaved with ironed shirt and good suite. Simply, it's telling me you are serious about your business.
AznHisoka 10y ago
"You cannot sell data you don’t own. Hedge funds care immensely about the legality of the data." Thats a shame as this makes this business model almost impossible for the average reader here. There are loads of outside data that if crawled and gathered could be useful to hedge funds. things like: - growth in number of Amazon reviews for a type of product month to month(against amazon tos to scrape them) - number of job ads that mention a technology or product (against indeed.com tos to scrape them) - google rankings of websites (against google tos to scrape them) - growth in total yelp reviews for a restaurant like Mcdonalds month to month (against yelps tos) which means this business is only for already profitable companies with large user bases.
- arbitraryquant 10y ago
  The point is if you're Amazon or Indeed or Yelp (or trying to compete) you can sell the data.
  - AznHisoka 10y ago
    thats true. and the rich get richer..
    - purity_resigns 10y ago
      I've heard them called Siren Servers.
    - deleted 10y ago
      [deleted]
    - deleted 10y ago
      [deleted]
- ice109 10y ago
  you're wrong. that's public data and you absolutely can sell it. hedge funds are worried about insider trading so if you can make the case that the data you used to trade was available to everyone then it's cleared by compliance. source: I work for a startup that does that and has 1MM rar.
arbitraryquant 10y ago
Throwaway for obvious reasons. I'm a quant at a quant hedge fund. I can confirm I want your data. The longer the history, the more I want it. Don't fuck with it much, give me clean but raw info. I don't want PII, but I'll take aggregate and anonymized. Just whatever you do, start collecting it NOW so in 3 years you have 3 years of history to sell me.
- Rainymood 10y ago
  Do you guys prefer a Master's or a PhD for research roles?
  - arbitraryquant 10y ago
    Majority are PhD, but definitely some have Bachelors/masters. Care more about quality of education/skills. 4.0 undergrad at MIT vs PhD at rando state U w unknown advisor... no question.
paulsutter 10y ago
Suggestions: - Make it easy for them to get the backfile, and charge them a lot for fresh data (somehow startups are inclined to do this backwards. They need a few years of data before they even know it's useful, but they can't trade without recent data. Charge them accordingly) - It's too hard to negotiate exclusives, but you can make them bid on exclusive access to delays. One second, minute, hour, week, etc. it doesn't need to be complicated but neither should you leave money on the table with too simple pricing
- arbitraryquant 10y ago
  Yes, if you want me to buy your data, give me all history up to last year. Then I'll decide.
jusq2 10y ago
Labor unions maybe going extinct but maybe its time for information/data unions.

114 points by dwynings 10y ago

Great article!! And i think it can probably be summed up with the sentence, unless your product has: - a long history of data - data that has low correlation to other available data sets - a very small group of people who have access to this data( ie exclusive deals with a few funds) Then its unlikely that you will be able to sell your data for any meaningful amount of money. A few things the author wrote that I think deserve some attention..... > You do hear the occasional story where a hedge fund paid a couple of millions a year to obtain a specific data set, and occasionally more. But there’s a reasonable chance that this type of price came with some kind of exclusive. Also, those contracts also probably have a limited shelf life, as the value of a data set decays over time, as per the above. This is very true, almost certainly your data set is not worth millions a year to any specific fund. And if it is, its because you agreed to make your data set exclusive to a single fund. Each data set requires its own special mess of tools for cleaning it, integrating it, figuring out how to generate a trading signal from it, putting it into production and then once its in production constantly monitoring it not only to verify the data is good, but to make sure it actually benefits the trading system. I wrote this comment: https://news.ycombinator.com/item?id=13140352 https://news.ycombinator.com/item?id=13140352 a while ago and it got a very large number of up votes for what was essentially a rant that we're now drowing in data. Hedge funds today share alot in common with AAA video games wrt to data. You used to be able to make a game with 2 people, then it was 10, then it was 50, then 100, now its 500+ for a AAA video game. Hedge funds have the exact same issue due to the explosion of data. I know alot of funds that are skipping on data sets, due to lack of bandwidth to consume them all. > First of all, unless it’s already commoditized, you probably don’t want to sell your data to the Bloombergs of the world. Most hedge funds won't be your direct audience, they'll already have spent money to buy data from Bloomberg and Reuters. The few, say top 100, that might buy your data directly could be your market, but these funds don't exactly have huge sales teams ready to talk to you. Which means you need to hire someone who knows the landscape to sell your data. Your sales team might be the best group of 25 year old Stanford grads around but they aren't getting in the door of most quant funds without an introduction. This is the website of one of the worlds most successful funds. https://www.rentec.com/Home.action?index=true https://www.rentec.com/Home.action?index=true

hackuser 10y ago
> i think it can probably be summed up with the sentence Hmm ... for anyone reading it, the article covers a lot more than the specifications for selling your dataset.
AznHisoka 10y ago
Also just plain dumb luck. stocks dont trade based on logic like revenue growth so if Trump tweets something negative about Starbucks and the stock goes down, but your data suggested Starbucks sales went up 300% last month, you are in tough luck. this is also probably why your data is worth much much less than you think. there are very few things that correlate very strongly to stock performance. predicting human emotion is probably more important than predicting revenue growth or number of people who went inside a Starbucks last month.
- laingc 10y ago
  If your data suggest that the stock is trading at a discount due to irrationality, then you have a perfect opportunity to buy lots of an underpriced stock.
  - aangjie 10y ago
    I think the point OP was trying to make is that, that irrationality may not be easy to capture with automated systems, therefore the value of data is limited by your trading system/algorithm's ability to detect/predict irrational human behaviour. Or atleast that's what I read of it.
    - dsacco 10y ago
      Yes, the grandparent comment makes a good point. You can have plenty of solid data and still make a losing trade. But, given enough data-informed trades that would otherwise be proditable, the law of large numbers suggests that an overall strategy will still be sound. You can afford to have the market be irrational here and there if you really are correctly forecasting earnings in advance most of the time.

hackuser 10y ago

> The good news is that hedge funds are not advertisers and don't care about particular individuals This sounds disingenuous; it's not hard for the author or anyone else to imagine how Wall Street could use personally identifiable information, and the author claims to be at least somewhat sophisticated in the 'big data' field. Also, it's an industry that has an incredible track record of fraud, even fixing the bedrock interest rates like Libor, prides itself on an anything-goes atmosphere, and disdains regulation.

digler999 10y ago
I could see them passing "insider consumer info" on black markets just like they pass insider information. How handy would it be for a life insurance company to know which of its customers is googling for "chest pain" or "$CANCER_TYPE symptoms". A car insurance company would love to know which drivers go to bars on saturday night, or which ones make it from point A to point B in suspiciously low amounts of time. Wall street could creep into this by saying to either industry: "let our guys crunch the numbers and let us make book on what these rates should be...because market efficiency". It'll seem like a great idea at first, the data will be "anonymized" (except for when it's not, like when that guy pays a $5k to an analyst at the insurance company for the table mapping "anonymized" customer idns to ssns. Suddenly hedge funds get really good at guessing which pools should be invested in, or which drivers should be packaged into the high-risk tier. A few years later there's a moderate scandal, a fine is levied equal to ~5% of the illicit gains made, meanwhile the funds have moved on to perpetuate their next round of fraud.
- hackuser 10y ago
  Also, learning information about individuals: Big clients, competitors, employees, critics, journalists, politicians, etc. etc.

hackuser 10y ago

> "No man is better than a machine, and no machine is better than a man with a machine." This concept also is popular in the military AI R&D world where the human-computer combination is called a 'centaur' (based on what I've read; I'm not of that world).

rcarrigan87 10y ago

> When I was at Bloomberg, I used to cringe when startup founders would show up in hoodies, essentially destroying their credibility before the meeting even started. I understand you can't roll in looking like a slob but I have to think times have changed and the casually dressed founder wouldn't have a problem, even with Wall Street types. The cultural differences b/t the two industries are pretty interesting though...

choward 10y ago
People dressed up in suits lose accountability to me. It's like they are over-compensating for something.
- zenlikethat 10y ago
  And I'm sure they are devastated by this loss of accountability, since all they got in return was a 2x - 10x multiple of income from the average programmer.
- bogomipz 10y ago
  Does that include lawyers and financial advisors too? Would you be OK with someone in jeans and a hoodie representing you in court? Would you be OK handing your life savings over to a money manager who was wearing a t-shirt and flip flops?
  - caleblloyd 10y ago
    Only if it was Michael Burry
  - maxerickson 10y ago
    Topical PSA from the financial advice industry: https://www.youtube.com/watch?v=yJFrkNY4n1g https://www.youtube.com/watch?v=yJFrkNY4n1g If you have something of value and the people you are talking to are actually interested in making money, they will get past quite a lot. (The lawyer question is more ridiculous, they have to work with the prejudices of the judges and lawyers they interact with so only in the narrowest of circumstances would a credible attorney walk into court in jeans)

dgfgfdagasdfgfa 10y ago

> The financial services world has its own strong identity, handles massive amounts of money, and will not necessarily be in awe of your cool startup. When I was at Bloomberg, I used to cringe when startup founders would show up in hoodies, essentially destroying their credibility before the meeting even started. Who wants wall street awe? Who cares about credibility? Run a goddamn business, not a fan club. If you cared about image, you wouldn't talk about what people wear. It's completely irrelevant to why I'd use your product.

AnimalMuppet 10y ago
Who cares about credibility? If you're trying to sell to Wall Street, you care about credibility. If you want them to invest in your startup, you care about credibility. If they want to sell to you, wear what you please.
- dgfgfdagasdfgfa 10y ago
  Funny, they never mentioned a link between perception and sales, so it just came out like shit talking....
zenlikethat 10y ago
Except that Silicon Valley is equally superficial in different ways. Consider: founder fetishism, engineering fetishism (vs. sales/marketing/product), relative ease of getting funded as a Caltech / MIT / Stanford grad, favoring growth over revenue without coherent business plan, acting like fanboys towards YC or companies such as Uber, Google, Apple, etc.
- dgfgfdagasdfgfa 10y ago
  No argument here, startup/vc fetishism is distinctly unattractive.
dig1 10y ago
> Run a goddamn business,.... If you cared about image, you wouldn't talk about what people wear. Let's change perspective - what do you think, how much success will have business with web pages that looks like from 90-ties or products that looks like they are made in basement in spare time? (sure, there are exceptions, but well, they are exceptions :P). Everything is about image; if I'm going to invest in you, you better be shaved with ironed shirt and good suite. Simply, it's telling me you are serious about your business.

AznHisoka 10y ago

"You cannot sell data you don’t own. Hedge funds care immensely about the legality of the data." Thats a shame as this makes this business model almost impossible for the average reader here. There are loads of outside data that if crawled and gathered could be useful to hedge funds. things like: - growth in number of Amazon reviews for a type of product month to month(against amazon tos to scrape them) - number of job ads that mention a technology or product (against indeed.com tos to scrape them) - google rankings of websites (against google tos to scrape them) - growth in total yelp reviews for a restaurant like Mcdonalds month to month (against yelps tos) which means this business is only for already profitable companies with large user bases.

arbitraryquant 10y ago
The point is if you're Amazon or Indeed or Yelp (or trying to compete) you can sell the data.
- AznHisoka 10y ago
  thats true. and the rich get richer..
  - purity_resigns 10y ago
    I've heard them called Siren Servers.
  - deleted 10y ago
    [deleted]
  - deleted 10y ago
    [deleted]
ice109 10y ago
you're wrong. that's public data and you absolutely can sell it. hedge funds are worried about insider trading so if you can make the case that the data you used to trade was available to everyone then it's cleared by compliance. source: I work for a startup that does that and has 1MM rar.

arbitraryquant 10y ago

Throwaway for obvious reasons. I'm a quant at a quant hedge fund. I can confirm I want your data. The longer the history, the more I want it. Don't fuck with it much, give me clean but raw info. I don't want PII, but I'll take aggregate and anonymized. Just whatever you do, start collecting it NOW so in 3 years you have 3 years of history to sell me.

Rainymood 10y ago
Do you guys prefer a Master's or a PhD for research roles?
- arbitraryquant 10y ago
  Majority are PhD, but definitely some have Bachelors/masters. Care more about quality of education/skills. 4.0 undergrad at MIT vs PhD at rando state U w unknown advisor... no question.

paulsutter 10y ago

Suggestions: - Make it easy for them to get the backfile, and charge them a lot for fresh data (somehow startups are inclined to do this backwards. They need a few years of data before they even know it's useful, but they can't trade without recent data. Charge them accordingly) - It's too hard to negotiate exclusives, but you can make them bid on exclusive access to delays. One second, minute, hour, week, etc. it doesn't need to be complicated but neither should you leave money on the table with too simple pricing

arbitraryquant 10y ago
Yes, if you want me to buy your data, give me all history up to last year. Then I'll decide.

jusq2 10y ago

Labor unions maybe going extinct but maybe its time for information/data unions.