5 ms·

I run a business in this space. Realistically your chances of having data that is useful to a HF (of any kind) is pretty low so I wouldn't bank on it as a reven

I run a business in this space. Realistically your chances of having data that is useful to a HF (of any kind) is pretty low so I wouldn't bank on it as a revenue source unless you have a strong reason to believe (1) your data is predictive of something an investor cares about and (2) isn't already covered by other data.

r_singh 6y ago
Could I send you a sample? I’m curious about certain data I’m scraping for a project.
- fractionalhare 6y ago
  If you shoot me an email I can tell you very quickly if it's viable, and direct you to specific people at firms who would buy it. I probably don't need a sample if you give an honest description of it.
  - neolog 6y ago
    Is it better to provide the raw data, or to instead provide some interesting statistics from aggregating or running a model on the data?
    - fractionalhare 6y ago
      Usually hedge funds prefer the raw data so their own research teams can do modeling and analysis.
      - neolog 6y ago
        Once they see the raw data they'll have a good idea how to get it. Why don't they just set up their own scraper at that point, instead of paying me?
        rmah 6y ago
        If all it is some data scraped off a few web sites that they could get an intern to do in a week or three, then it's unlikely to be valuable enough for them to pay you a substantial sum of money. The most valuable data is data that is difficult to gather. Think things like proprietary (i.e. unpublished) industry data. The canonical "sexy" alternative data set sold to hedge funds is counts of cars in retail parking lots from satellite photos.
        fractionalhare 6y ago
        > If all it is some data scraped off a few web sites that they could get an intern to do in a week or three, then it's unlikely to be valuable enough for them to pay you a substantial sum of money. If the data is compelling and clearly correlates to earnings KPIs, I can tell you from experience that "some data scraped off a few websites" can be salable to the tune of $50,000/quarter. Hedge funds will frequently choose to pay that instead of setting up their own scraping operation, because scraping sucks. Not all hedge funds, of course. Some do actively try to reverse engineer your dataset. But you probably don't want to work with those anyway. > The canonical "sexy" alternative data set sold to hedge funds is counts of cars in retail parking lots from satellite photos. I've personally developed forecasts from (ostensibly) public, scraped sources which beat drone and satellite footage of manufacturing facilities. That one Bloomberg article is not representative; satellite footage sounds sexy but it's not what most alternative data looks like.
        fakedang 6y ago
        Tbh I've yet to find successful hedge funds that try to reverse engineer stuff. It's just not worth the time for most of these companies.
        fractionalhare 6y ago
        Because scraping data sucks, occasionally has compliance concerns, and is a different core competency from trading. They would rather offload all of the bullshit involved in maintaining a robust scraping operation than pay their research team to do it. Time spent on maintaining a scraping operation is time taken away from optimizing your ETL process and producing actionable research for your trading team. You know how people pay to have their pipes unclogged even when they know how it's done? Same idea.
  - cryptonautics 6y ago
    I disseminate realtime transaction information from blockchain mempools (BTC,ETH) and flag any that create large state updates, is this useful information for any hedgefunds in the cryptospace?
fractionalhare 6y ago
There are also ways to reliably find and curate this data for trading firms and asset managers. But that usually requires a somewhat uncommon blend of skills; like statistics, web scraping, reverse engineering, or some domain expertise that gives you an edge in (legally) finding and using nonpublic data. Scrappiness also helps a lot.
dhairya 6y ago
Out of curiosity what resolution and time scale is useful. Is it fair to assume the most hedge funds are relatively good at tracking recent information and the value is in older archives that's hard to collect? Also are event streams and large connect event graphs like Forge.AI what sells actually useful? Curious as my phd research potentially has applications for information extraction and event linking. But not entirely sure if those applications are actually valuable
- yellowstuff 6y ago
  > Is it fair to assume the most hedge funds are relatively good at tracking recent information and the value is in older archives that's hard to collect? No, like he said the most valuable data has a signal that is independent from other data sets, which generally means something proprietary, and almost always without a long history. Having a cleaned archive of standard data with a long history is valuable but already pretty well served. It would be hard to compete with CRSP.
ed25519FUUU 6y ago
I’m curious the types of data that are useful, what goes into them, and what the curators of that data might make for them.
crb002 6y ago
Trying to get my credit union to allow users to opt in to sell bulk annon checking account transactions to a hedge fund. Having that much point of sale data would be huge.
- ponker 6y ago
  uh no, please don't do that. Individual decisions that seem harmless can be incredibly destructive in the aggregate, and this is one of them.
  - computerphage 6y ago
    Would you mind elaborating on the sorts of harms you're thinking of?
  - cbozeman 6y ago
    I'm not saying you're wrong, but I am curious what negative trends you believe will emerge from someone actioning this data.
  - tmoaad 6y ago
    For those that need an example, a great reminder is the target data mining story at https://www.forbes.com/sites/kashmirhill/2012/02/16/how-target-figured-out-a-teen-girl-was-pregnant-before-her-father-did/?sh=2e605e406668 https://www.forbes.com/sites/kashmirhill/2012/02/16/how-targ...

r_singh 6y ago

Could I send you a sample? I’m curious about certain data I’m scraping for a project.

fractionalhare 6y ago
If you shoot me an email I can tell you very quickly if it's viable, and direct you to specific people at firms who would buy it. I probably don't need a sample if you give an honest description of it.
- neolog 6y ago
  Is it better to provide the raw data, or to instead provide some interesting statistics from aggregating or running a model on the data?
  - fractionalhare 6y ago
    Usually hedge funds prefer the raw data so their own research teams can do modeling and analysis.
    - neolog 6y ago
      Once they see the raw data they'll have a good idea how to get it. Why don't they just set up their own scraper at that point, instead of paying me?
      - rmah 6y ago
        If all it is some data scraped off a few web sites that they could get an intern to do in a week or three, then it's unlikely to be valuable enough for them to pay you a substantial sum of money. The most valuable data is data that is difficult to gather. Think things like proprietary (i.e. unpublished) industry data. The canonical "sexy" alternative data set sold to hedge funds is counts of cars in retail parking lots from satellite photos.
        fractionalhare 6y ago
        > If all it is some data scraped off a few web sites that they could get an intern to do in a week or three, then it's unlikely to be valuable enough for them to pay you a substantial sum of money. If the data is compelling and clearly correlates to earnings KPIs, I can tell you from experience that "some data scraped off a few websites" can be salable to the tune of $50,000/quarter. Hedge funds will frequently choose to pay that instead of setting up their own scraping operation, because scraping sucks. Not all hedge funds, of course. Some do actively try to reverse engineer your dataset. But you probably don't want to work with those anyway. > The canonical "sexy" alternative data set sold to hedge funds is counts of cars in retail parking lots from satellite photos. I've personally developed forecasts from (ostensibly) public, scraped sources which beat drone and satellite footage of manufacturing facilities. That one Bloomberg article is not representative; satellite footage sounds sexy but it's not what most alternative data looks like.
        fakedang 6y ago
        Tbh I've yet to find successful hedge funds that try to reverse engineer stuff. It's just not worth the time for most of these companies.
      - fractionalhare 6y ago
        Because scraping data sucks, occasionally has compliance concerns, and is a different core competency from trading. They would rather offload all of the bullshit involved in maintaining a robust scraping operation than pay their research team to do it. Time spent on maintaining a scraping operation is time taken away from optimizing your ETL process and producing actionable research for your trading team. You know how people pay to have their pipes unclogged even when they know how it's done? Same idea.
- cryptonautics 6y ago
  I disseminate realtime transaction information from blockchain mempools (BTC,ETH) and flag any that create large state updates, is this useful information for any hedgefunds in the cryptospace?

fractionalhare 6y ago

There are also ways to reliably find and curate this data for trading firms and asset managers. But that usually requires a somewhat uncommon blend of skills; like statistics, web scraping, reverse engineering, or some domain expertise that gives you an edge in (legally) finding and using nonpublic data. Scrappiness also helps a lot.

dhairya 6y ago

Out of curiosity what resolution and time scale is useful. Is it fair to assume the most hedge funds are relatively good at tracking recent information and the value is in older archives that's hard to collect? Also are event streams and large connect event graphs like Forge.AI what sells actually useful? Curious as my phd research potentially has applications for information extraction and event linking. But not entirely sure if those applications are actually valuable

yellowstuff 6y ago
> Is it fair to assume the most hedge funds are relatively good at tracking recent information and the value is in older archives that's hard to collect? No, like he said the most valuable data has a signal that is independent from other data sets, which generally means something proprietary, and almost always without a long history. Having a cleaned archive of standard data with a long history is valuable but already pretty well served. It would be hard to compete with CRSP.

ed25519FUUU 6y ago

I’m curious the types of data that are useful, what goes into them, and what the curators of that data might make for them.

crb002 6y ago

Trying to get my credit union to allow users to opt in to sell bulk annon checking account transactions to a hedge fund. Having that much point of sale data would be huge.

ponker 6y ago
uh no, please don't do that. Individual decisions that seem harmless can be incredibly destructive in the aggregate, and this is one of them.
- computerphage 6y ago
  Would you mind elaborating on the sorts of harms you're thinking of?
- cbozeman 6y ago
  I'm not saying you're wrong, but I am curious what negative trends you believe will emerge from someone actioning this data.
- tmoaad 6y ago
  For those that need an example, a great reminder is the target data mining story at https://www.forbes.com/sites/kashmirhill/2012/02/16/how-target-figured-out-a-teen-girl-was-pregnant-before-her-father-did/?sh=2e605e406668 https://www.forbes.com/sites/kashmirhill/2012/02/16/how-targ...