28 ms·

Elo for VC – Founder's Choice

111 points by satuke 3y ago

noxvilleza 3y ago
Interested in some parts of the methodology (if perhaps someone knows or the creators spot this thread): * The pairwise comparison: does it freeze updates and calculate all shifts at the same time? For example if you had A > B > C do you calculate the impact of {A>B, A>C, B>C}, sum these impacts together (grouped by the VC), and then apply them? Or do you do it iteratively: if a firm had {A}, then {A>B}, then {A>B>C} do you add 0 then 1 then 2 comparisons as you get new data? * How do you handle the fact that respondents to the survey are over a large time-frame, so some VCs might get better or worse over that time frame? Is there some Elo-decay applied?
- irjustin 3y ago
  There are so many problems, but I think it's better to just accept than try to create a more complex methodology. Naturally this has "high school popularity" problem baked into it. So even fixes to problems won't solve fundamentals. As long as everyone knows up front what is going into (which they do via source), then you can make a judgement for yourself. Take it all with a grain of salt.
thesausageking 3y ago
I don't think the fund level ratings matter anymore. a16z has more than 300 partners managing $35B across separate funds in tech, bio, crypto, cultural leadership, and other areas. The partner(s) you work with matter more than the shingle outside the office.
- gabereiser 3y ago
  I think the fixation that luck is just a math exercise is the true culprit here. You’re right, your partnerships matter. The human element matters and can’t be quantified. It’s a gut instinct in leadership above the due diligence. When you find ones you work well with, you tend to keep the gravy train running.
- ramraj07 3y ago
  If you're comparing apples to apples similar size funds it still does matter - sure a16z is 300 partners but they share a culture and mission that evolves constantly and this will have a meaningful effect on your interaction with them.
  - slap_shot 3y ago
    That sounds nice in theory, but in reality, whether you raise from a firm of three partners or 300, you're going to work very closely with a single partner, and his or her competencies/style is mostly what your interaction with the "firm" is going to be. And you rarely, if ever, get to chose the partner with whom you work.
- allenleee 3y ago
  Dirty secret of VC platform/talent/marketing teams: they are more about scaling the GP than the founder.
  - moneywoes 3y ago
    How does that work in practice? They focus on building a brand for the partner?
gnicholas 3y ago
In case anyone else is fuzzy on the precise meaning of "Elo": https://en.wikipedia.org/wiki/Elo_rating_system https://en.wikipedia.org/wiki/Elo_rating_system
jacquesm 3y ago
Personal vote for Hoxton. Nice people and very knowledgeable, and ethical too. Note that many (top) EU VCs aren't in this dataset. This is probably due to 'We only include firms where we received 100 or more comparisons to other firms.', which in Europe, where the VC landscape is - fortunately - much more fragmented isn't going to happen all that often except for seed funds. Also, it might be worth it to add PE parties as well because that's one track where founders may well end up and those interactions do not always go smoothly.
kriro 3y ago
Nice. I think a (main) location field would be valuable, probably city+country. I'd like to quickly find EU based VCs for example.
- jacquesm 3y ago
  There won't be that many that make the 100 comparison cliffs.
  - satuke 3y ago
    Yeah I mean, for now it'll just be for the 200+ that are there and later it could grow in future.
    - jacquesm 3y ago
      That makes it next to useless for half the planet.
      - satuke 3y ago
        How? It should be just as useless as the list itself. Lots of EU funds aren't on the list now but they might be in future. Your argument is like saying Yahoo and Google should be useless as there weren't many sites to search through in 1990s.
        jacquesm 3y ago
        Yahoo and Google did not have arbitrary cut-offs for the stuff they included and the stuff that they didn't that led to disenfranchising half the planet (or more, in fact).
        satuke 3y ago
        But how does including a main location infront of a VC disenfranchises half the planet?
bravura 3y ago
Aside: I have an optimization algorithm and I'm curious if ELO ranking (or TrueSkill) would be a decent approximate solution. I have a sparse matrix of probabilities that I want to turn into a DAG. If x[m,n] = pr it means that m is a descendent (direct or transitively) of n with probability pr. I want to construct a DAG over these edges. Most importantly, I want a solution that maintains the DAG property, i.e. no cycles. Given that constraint, I want to maximize the total probability of edges kept in the DAG combined with the (1 - probability) edges removed from the graph. Any suggestions on how to implement this optimization algorithm? Perhaps I could use an ELO or TrueSkill ranking as an approximation. The difficulty is sampling matches, but perhaps it makes sense to sample non-zero edges randomly, uniformly. So nodes with high in-degree or high out-degree are selected more frequently, since they are more likely to impose constraints on the graph. The probability of winning is determined by the edge probability. This doesn't guarantee a DAG but would be a great initialization point. Anyway, I'm curious about alternate ideas or refinements to the above.
- durumu 3y ago
  I think that's a pretty good idea. Have you checked out the paper "Breaking Cycles in Noisy Hierarchies"? It looks pretty similar to what you're looking for and the code is open source: https://github.com/zhenv5/breaking_cycles_in_noisy_hierarchies https://github.com/zhenv5/breaking_cycles_in_noisy_hierarchi...
- kevinwang 3y ago
  Do you know that there's no polynomial time algorithm that solves the problem exactly?
- adamrezich 3y ago
  TrueSkill is dead simple to use—might as well give it a shot.

111 points by satuke 3y ago

Interested in some parts of the methodology (if perhaps someone knows or the creators spot this thread): * The pairwise comparison: does it freeze updates and calculate all shifts at the same time? For example if you had A > B > C do you calculate the impact of {A>B, A>C, B>C}, sum these impacts together (grouped by the VC), and then apply them? Or do you do it iteratively: if a firm had {A}, then {A>B}, then {A>B>C} do you add 0 then 1 then 2 comparisons as you get new data? * How do you handle the fact that respondents to the survey are over a large time-frame, so some VCs might get better or worse over that time frame? Is there some Elo-decay applied?

irjustin 3y ago
There are so many problems, but I think it's better to just accept than try to create a more complex methodology. Naturally this has "high school popularity" problem baked into it. So even fixes to problems won't solve fundamentals. As long as everyone knows up front what is going into (which they do via source), then you can make a judgement for yourself. Take it all with a grain of salt.

thesausageking 3y ago

I don't think the fund level ratings matter anymore. a16z has more than 300 partners managing $35B across separate funds in tech, bio, crypto, cultural leadership, and other areas. The partner(s) you work with matter more than the shingle outside the office.

gabereiser 3y ago
I think the fixation that luck is just a math exercise is the true culprit here. You’re right, your partnerships matter. The human element matters and can’t be quantified. It’s a gut instinct in leadership above the due diligence. When you find ones you work well with, you tend to keep the gravy train running.
ramraj07 3y ago
If you're comparing apples to apples similar size funds it still does matter - sure a16z is 300 partners but they share a culture and mission that evolves constantly and this will have a meaningful effect on your interaction with them.
- slap_shot 3y ago
  That sounds nice in theory, but in reality, whether you raise from a firm of three partners or 300, you're going to work very closely with a single partner, and his or her competencies/style is mostly what your interaction with the "firm" is going to be. And you rarely, if ever, get to chose the partner with whom you work.
allenleee 3y ago
Dirty secret of VC platform/talent/marketing teams: they are more about scaling the GP than the founder.
- moneywoes 3y ago
  How does that work in practice? They focus on building a brand for the partner?

gnicholas 3y ago

In case anyone else is fuzzy on the precise meaning of "Elo": https://en.wikipedia.org/wiki/Elo_rating_system https://en.wikipedia.org/wiki/Elo_rating_system

jacquesm 3y ago

Personal vote for Hoxton. Nice people and very knowledgeable, and ethical too. Note that many (top) EU VCs aren't in this dataset. This is probably due to 'We only include firms where we received 100 or more comparisons to other firms.', which in Europe, where the VC landscape is - fortunately - much more fragmented isn't going to happen all that often except for seed funds. Also, it might be worth it to add PE parties as well because that's one track where founders may well end up and those interactions do not always go smoothly.

kriro 3y ago

Nice. I think a (main) location field would be valuable, probably city+country. I'd like to quickly find EU based VCs for example.

jacquesm 3y ago
There won't be that many that make the 100 comparison cliffs.
- satuke 3y ago
  Yeah I mean, for now it'll just be for the 200+ that are there and later it could grow in future.
  - jacquesm 3y ago
    That makes it next to useless for half the planet.
    - satuke 3y ago
      How? It should be just as useless as the list itself. Lots of EU funds aren't on the list now but they might be in future. Your argument is like saying Yahoo and Google should be useless as there weren't many sites to search through in 1990s.
      - jacquesm 3y ago
        Yahoo and Google did not have arbitrary cut-offs for the stuff they included and the stuff that they didn't that led to disenfranchising half the planet (or more, in fact).
        satuke 3y ago
        But how does including a main location infront of a VC disenfranchises half the planet?

bravura 3y ago

Aside: I have an optimization algorithm and I'm curious if ELO ranking (or TrueSkill) would be a decent approximate solution. I have a sparse matrix of probabilities that I want to turn into a DAG. If x[m,n] = pr it means that m is a descendent (direct or transitively) of n with probability pr. I want to construct a DAG over these edges. Most importantly, I want a solution that maintains the DAG property, i.e. no cycles. Given that constraint, I want to maximize the total probability of edges kept in the DAG combined with the (1 - probability) edges removed from the graph. Any suggestions on how to implement this optimization algorithm? Perhaps I could use an ELO or TrueSkill ranking as an approximation. The difficulty is sampling matches, but perhaps it makes sense to sample non-zero edges randomly, uniformly. So nodes with high in-degree or high out-degree are selected more frequently, since they are more likely to impose constraints on the graph. The probability of winning is determined by the edge probability. This doesn't guarantee a DAG but would be a great initialization point. Anyway, I'm curious about alternate ideas or refinements to the above.

durumu 3y ago
I think that's a pretty good idea. Have you checked out the paper "Breaking Cycles in Noisy Hierarchies"? It looks pretty similar to what you're looking for and the code is open source: https://github.com/zhenv5/breaking_cycles_in_noisy_hierarchies https://github.com/zhenv5/breaking_cycles_in_noisy_hierarchi...
kevinwang 3y ago
Do you know that there's no polynomial time algorithm that solves the problem exactly?
adamrezich 3y ago
TrueSkill is dead simple to use—might as well give it a shot.