8 ms·

Why Law Firm Rankings Are Useless

66 points by wikiwawa 10y ago

rayiner 10y ago
As they acknowledge, it is essential to account for case difficulty in comparing success rates. Having been on both sides of the "v." I can say it is easier on the defense side (at least on the civil side). Defense counsel representing Big Cos. will have a better record not only because those companies are often the target of weaker lawsuits, but because various rules intended to filter out those weaker lawsuits stack the deck against plaintiffs. Thus, there are fantastic lawyers working at Sierra Club or NRDC, but they're not going to have the same record of wins as similarly-good lawyers at a big New York firm. According to the article, their analysis does account for that. They allegedly "calculate the probability of a successful outcome for the applicant" using machine learning. But there is no description of how they do that. If I had an algorithm that produced halfway decent results in an automated fashion, I wouldn't use it to get into the legal technology market. I'd set myself up as a litigation funder and make boatloads of money with accurate valuations of potential investments.
- gohrt 10y ago
  > Thus, there are fantastic lawyers working at Sierra Club or NRDC, but they're not going to have the same record of wins as similarly-good lawyers at a big New York firm. Not a problem for the analysis of law firms. Sierra Club is a less effective law firm than the big New York firm.
  - x1798DE 10y ago
    > Not a problem for the analysis of law firms. Sierra Club is a less effective law firm than the big New York firm. Why is this not a problem? The parent comment was saying that the Sierra Club takes harder cases and is often the plaintiff, not the defendant, so they may be more effective as a law firm but have worse numbers because the win/loss numbers don't mean the same things in different contexts.
    - harryh 10y ago
      It's also worth nothing that the Sierra Club isn't a law firm at all. It's an environmental organization that, among many other things, sometimes uses the legal system to advance its goals.
- andrewfong 10y ago
  If I were interested in litigation funding, I wouldn't evaluate the ability of law firms to litigate but rather the ability of lawyers to accurate predict the outcome of cases (regardless of whether they're actually representing the client) -- e.g. associates at law firms frequently write memos to assess the probability of success before a case makes it way to trial. Seems like there's value in actually correlating memos with actual outcomes.
  - rayiner 10y ago
    The article says they have a machine-learning algorithm that can "calculate the probability of a successful outcome" of a case. Then they calculate firms' ability to improve (or not) that outcome. But if I had such an algorithm I'd just use it directly to decide what cases to fund. My understanding from talking to litigation funders is that lawyers are terrible at valuing cases. There is money to be made for anyone that can use technology to improve those predictions.
- jamesk_au 10y ago
  > If I had an algorithm that produced halfway decent results in an automated fashion, I wouldn't use it to get into the legal technology market. I'd set myself up as a litigation funder and make boatloads of money with accurate valuations of potential investments. Legalist is a recent YC startup aiming to do exactly that: http://www.newyorker.com/business/currency/what-litigation-finance-is-really-about http://www.newyorker.com/business/currency/what-litigation-f...
sandworm101 10y ago
>> "the job of a lawyer is to persuade, which is part legal reasoning and part hard work. Since the ability to persuade directly affects litigation outcomes, participants are willing to pay highly regarded barristers upwards of $28,000 a day, and some law firm partners an hourly rate of over $1,000. In this post, we analyse data from 22,000 cases to measure the differential impact law firms and barristers have on litigation outcomes." No. That's what TV lawyers are paid for. That's the equivalent of learning about doctors from watching MASH. Most doctors are not surgeons. Most lawyers are not litigators. Many great and valuable lawyers will never win any case. Imho the best lawyer is the one who's client never faces the uncertainty and protracted negativity that is modern litigation, just as a great and valuable doctor is one who's patients avoid the knife. (Apples and oranges but the point stands.) Lawyering is, in my world, mostly about listening to clients and understanding how their realities fit into external standards such as laws and industry norms. It's about hedging, documentation and interpretations. And sometimes it's about holding the client's feet to a fire. Litigators only appear once all of that has failed. Litigators may have the highest hourly rates, but do not have the highest salaries overall. Those are reserved for corporate counsels and board members.
- iplaw 10y ago
  Depends on the subject matter and the willingness of a plaintiff and defendant to negotiate a fair settlement. In some areas, big money comes with big litigation.
libertymcateer 10y ago
> By applying machine learning algorithms to data mined from court judgments... And, with that sentence, this post becomes bullshit. The overwhelming majority of disputes do not reach litigation. The overwhelming majority of litigation does not reach judgment. And then a gigantic proportion of those judgments are appealed, and a non-trivial percentage are overturned. Even at that, only on the order of a single digit-percentage of disputes ever reaches a verdict. This is like judging a hospital-system based on its success-rate for transplant surgery. It's just deeply ignorant of what constitutes the vast, vast majority of the practice of law - even in disputes. This article has no value, whatsoever, aside from a witty aside at a cocktail party about how deeply misunderstood the legal profession is. This is particularly shocking given that litimetrics is a legal service provider. > Although A has not faced B, if both have faced C, then we have some information about the hypothetical A and B matchup. This is the same reasoning that allows one to compare tennis players across different eras. The implication is that the lawyer efficacy, based on judgments, is subject to the transitive property? Sweet mercy that is ridiculous. This post is a catastrophe.
- sandworm101 10y ago
  I'd add that the majority of civil cases are resolved through non-public deals whereby no third party can tell who has won or lost. That said, it might be interesting to apply machine learning at the appellate level. There, the outcomes are public and more readable by machines. Something as simple as looking for tell-tales such as "reverse", "dismissed" or "we side with the appellant" in holdings could be mapped to firms.
  - monocasa 10y ago
    Hell, if the lawyers have done a good job, both sides walk away from a settlement thinking that they won.
    - libertymcateer 10y ago
      Or that neither won.
- 10y ago
pitt1980 10y ago
"To sidestep the randomness of observed outcomes, we focus on modeling the probability of a successful outcome for a given case. Secondly, to account for the context or difficulty of the case, we measure the change in the probability of a successful litigation outcome, from substituting a legal service provider in. The difficulty of the case is not important, but rather the impact a legal service provider has on that original probability of success. From now on, we will refer to this measure as the impact on expected performance. This substitution exercise is known as a counterfactual, as we apply the substitutions to historical data." ----------------- how exactly did they model the probability of successful outcomes for given cases? because those depend on highly detailed fact patterns, many of which are largely qualitative in nature if they were actually able to put all those in a data base that you could then model that'd be a more interesting accomplishment to read about
- Digory 10y ago
  Right. The post links to slideshow "model" for counterfactuals given at Hulu. And I won't pretend to analyze it scientifically, but it's hard to shake the feeling that it's circular in this application. We want to rank lawyers based on effect on outcome, not win-loss record. But it looks like the model is one that derives "counterfactual" outcomes by making comparisons based on win-loss records. Can someone give a better lay-level discussion of the expected outcome model?
spitfire 10y ago
Why not use a modified Elo rating? That way you can not only easily figure out if you're going to win, but simulate the outcome of different scenarios?
- robrenaud 10y ago
  What is a modified elo rating?
  - lordnacho 10y ago
    Elo rating is a system used to gauge chess players. You win/lose point depending on your points difference before a match. If the favourite loses, more points are lost to the underdog than if the underdog loses. You can apply this to a wide variety of sports where most teams don't get to play most other teams, eg you can guess that Germany are better than India at soccer despite the rarity of the tie. And of course you can modify the system to take account of various issues such as the retirement of better players, which tends to remove points from the system.
- Digory 10y ago
  The problem is the second assumption of Elo's system, per wikipedia: "if a player wins a game, he is assumed to have performed at a higher level than his opponent for that game. Conversely, if he loses, he is assumed to have performed at a lower level." But you can't assume that about lawsuits. If the client should have paid ten million in damages, the better lawyer might limit the award to nominal damages. But the same nominal damages loss might be a disaster in another case the lawyer should have won. Lawsuits aren't zero-sum, bilateral agreements to compete like most sports. Parties aren't just competing for a 'win,' but looking to maximize gain or minimize loss.
deleted 10y ago
[deleted]
deleted 10y ago
[deleted]
soufron 10y ago
I worked on applying machine learning algorithms to data mined from court judgments during my PhD. You want to know what we learned? That it's better to be white and rich than black and poor. Point. Everything else was bullshit.
- selectron 10y ago
  How much better?
  - soufron 10y ago
    Lol.
    - selectron 10y ago
      It is a legitimate question.
- steveeq1 10y ago
  What about black and rich vs white and poor?
  - sjdchid 10y ago
    Sample size too small I'd guess.
soufron 10y ago
Also, being a Lawyer, if someone comes to me, it's because of me, not because of the firm I am working in. And trying to rank lawyers? Why not. But what's the point. I wouldn't accept being ranked anyway.
- hackuser 10y ago
  > being a Lawyer, if someone comes to me, it's because of me, not because of the firm I am working in Are you speaking hypothetically, i.e., 'if I was a lawyer'? I'm not an attorney but for a variety of reasons I have relatively deep insight into the legal industry, and I think this statement is generally not true. As in any field, most business comes from referrals and networking, and much of that happens within a firm. If your career is at a top firm then you are going to build a much different network than a similar attorney who opens their own office in a strip mall. When Goldman Sachs needs an outside attorney, where do they go and who do they ask? Even ignoring referrals, just imagine who is there when you are invited to dinner by your colleague: Is it the mayor and local Fortune 500 CEO, one of whom takes a liking to you and asks you to join the board of a local charity, or is it the lovely neighbors who drive Uber or run a small medical practice, and ask for some legal tips? Personally, I don't enjoy the hunt for status and I think it impedes socio-economic mobility, but I'm afraid that's the way it works.
  - soufron 10y ago
    I am a lawyer yes. But let's say I have a good cv.
  - deleted 10y ago
    [deleted]
  - charlesdm 10y ago
    I have a tax lawyer I work with, and I couldn't care less which firm he works at. However, he does work at a good firm, and a good name helps in negotiations (in this case, with the tax authority). A while back, I had a tax audit, and surprisingly the person doing the audit told me my firm can turn "black" into "white". That says quite a bit about the name and reputation of the firm.
charlesdm 10y ago
As someone who works with a tax law firm that is ranked pretty high on the legal500 (and obviously, I know a rank can be gamed relatively easily), I must say the experiences I've had with them have been excellent. For all intents and purposes, they are exactly what I would expect from a good law firm. I worked with several "independent" lawyers before, and honestly, they are nowhere near close to the level of skill of my current law firm. In addition, they're quite responsive, much more than the average independent lawyer. And yes, they're more expensive, but not outrageously so. They seem to reach a favourable outcome faster, and seem to be more effective. Tl;dr: Love the lawyer I work with in this firm, he gets the job done when I need him to + has a great team with broad experience to back him up. He's managed to resolve any tax issue before we even went to court.

66 points by wikiwawa 10y ago

As they acknowledge, it is essential to account for case difficulty in comparing success rates. Having been on both sides of the "v." I can say it is easier on the defense side (at least on the civil side). Defense counsel representing Big Cos. will have a better record not only because those companies are often the target of weaker lawsuits, but because various rules intended to filter out those weaker lawsuits stack the deck against plaintiffs. Thus, there are fantastic lawyers working at Sierra Club or NRDC, but they're not going to have the same record of wins as similarly-good lawyers at a big New York firm. According to the article, their analysis does account for that. They allegedly "calculate the probability of a successful outcome for the applicant" using machine learning. But there is no description of how they do that. If I had an algorithm that produced halfway decent results in an automated fashion, I wouldn't use it to get into the legal technology market. I'd set myself up as a litigation funder and make boatloads of money with accurate valuations of potential investments.

gohrt 10y ago
> Thus, there are fantastic lawyers working at Sierra Club or NRDC, but they're not going to have the same record of wins as similarly-good lawyers at a big New York firm. Not a problem for the analysis of law firms. Sierra Club is a less effective law firm than the big New York firm.
- x1798DE 10y ago
  > Not a problem for the analysis of law firms. Sierra Club is a less effective law firm than the big New York firm. Why is this not a problem? The parent comment was saying that the Sierra Club takes harder cases and is often the plaintiff, not the defendant, so they may be more effective as a law firm but have worse numbers because the win/loss numbers don't mean the same things in different contexts.
  - harryh 10y ago
    It's also worth nothing that the Sierra Club isn't a law firm at all. It's an environmental organization that, among many other things, sometimes uses the legal system to advance its goals.
andrewfong 10y ago
If I were interested in litigation funding, I wouldn't evaluate the ability of law firms to litigate but rather the ability of lawyers to accurate predict the outcome of cases (regardless of whether they're actually representing the client) -- e.g. associates at law firms frequently write memos to assess the probability of success before a case makes it way to trial. Seems like there's value in actually correlating memos with actual outcomes.
- rayiner 10y ago
  The article says they have a machine-learning algorithm that can "calculate the probability of a successful outcome" of a case. Then they calculate firms' ability to improve (or not) that outcome. But if I had such an algorithm I'd just use it directly to decide what cases to fund. My understanding from talking to litigation funders is that lawyers are terrible at valuing cases. There is money to be made for anyone that can use technology to improve those predictions.
jamesk_au 10y ago
> If I had an algorithm that produced halfway decent results in an automated fashion, I wouldn't use it to get into the legal technology market. I'd set myself up as a litigation funder and make boatloads of money with accurate valuations of potential investments. Legalist is a recent YC startup aiming to do exactly that: http://www.newyorker.com/business/currency/what-litigation-finance-is-really-about http://www.newyorker.com/business/currency/what-litigation-f...

sandworm101 10y ago

>> "the job of a lawyer is to persuade, which is part legal reasoning and part hard work. Since the ability to persuade directly affects litigation outcomes, participants are willing to pay highly regarded barristers upwards of $28,000 a day, and some law firm partners an hourly rate of over $1,000. In this post, we analyse data from 22,000 cases to measure the differential impact law firms and barristers have on litigation outcomes." No. That's what TV lawyers are paid for. That's the equivalent of learning about doctors from watching MASH. Most doctors are not surgeons. Most lawyers are not litigators. Many great and valuable lawyers will never win any case. Imho the best lawyer is the one who's client never faces the uncertainty and protracted negativity that is modern litigation, just as a great and valuable doctor is one who's patients avoid the knife. (Apples and oranges but the point stands.) Lawyering is, in my world, mostly about listening to clients and understanding how their realities fit into external standards such as laws and industry norms. It's about hedging, documentation and interpretations. And sometimes it's about holding the client's feet to a fire. Litigators only appear once all of that has failed. Litigators may have the highest hourly rates, but do not have the highest salaries overall. Those are reserved for corporate counsels and board members.

iplaw 10y ago
Depends on the subject matter and the willingness of a plaintiff and defendant to negotiate a fair settlement. In some areas, big money comes with big litigation.

libertymcateer 10y ago

> By applying machine learning algorithms to data mined from court judgments... And, with that sentence, this post becomes bullshit. The overwhelming majority of disputes do not reach litigation. The overwhelming majority of litigation does not reach judgment. And then a gigantic proportion of those judgments are appealed, and a non-trivial percentage are overturned. Even at that, only on the order of a single digit-percentage of disputes ever reaches a verdict. This is like judging a hospital-system based on its success-rate for transplant surgery. It's just deeply ignorant of what constitutes the vast, vast majority of the practice of law - even in disputes. This article has no value, whatsoever, aside from a witty aside at a cocktail party about how deeply misunderstood the legal profession is. This is particularly shocking given that litimetrics is a legal service provider. > Although A has not faced B, if both have faced C, then we have some information about the hypothetical A and B matchup. This is the same reasoning that allows one to compare tennis players across different eras. The implication is that the lawyer efficacy, based on judgments, is subject to the transitive property? Sweet mercy that is ridiculous. This post is a catastrophe.

sandworm101 10y ago
I'd add that the majority of civil cases are resolved through non-public deals whereby no third party can tell who has won or lost. That said, it might be interesting to apply machine learning at the appellate level. There, the outcomes are public and more readable by machines. Something as simple as looking for tell-tales such as "reverse", "dismissed" or "we side with the appellant" in holdings could be mapped to firms.
- monocasa 10y ago
  Hell, if the lawyers have done a good job, both sides walk away from a settlement thinking that they won.
  - libertymcateer 10y ago
    Or that neither won.
10y ago

pitt1980 10y ago

"To sidestep the randomness of observed outcomes, we focus on modeling the probability of a successful outcome for a given case. Secondly, to account for the context or difficulty of the case, we measure the change in the probability of a successful litigation outcome, from substituting a legal service provider in. The difficulty of the case is not important, but rather the impact a legal service provider has on that original probability of success. From now on, we will refer to this measure as the impact on expected performance. This substitution exercise is known as a counterfactual, as we apply the substitutions to historical data." ----------------- how exactly did they model the probability of successful outcomes for given cases? because those depend on highly detailed fact patterns, many of which are largely qualitative in nature if they were actually able to put all those in a data base that you could then model that'd be a more interesting accomplishment to read about

Digory 10y ago
Right. The post links to slideshow "model" for counterfactuals given at Hulu. And I won't pretend to analyze it scientifically, but it's hard to shake the feeling that it's circular in this application. We want to rank lawyers based on effect on outcome, not win-loss record. But it looks like the model is one that derives "counterfactual" outcomes by making comparisons based on win-loss records. Can someone give a better lay-level discussion of the expected outcome model?

spitfire 10y ago

Why not use a modified Elo rating? That way you can not only easily figure out if you're going to win, but simulate the outcome of different scenarios?

robrenaud 10y ago
What is a modified elo rating?
- lordnacho 10y ago
  Elo rating is a system used to gauge chess players. You win/lose point depending on your points difference before a match. If the favourite loses, more points are lost to the underdog than if the underdog loses. You can apply this to a wide variety of sports where most teams don't get to play most other teams, eg you can guess that Germany are better than India at soccer despite the rarity of the tie. And of course you can modify the system to take account of various issues such as the retirement of better players, which tends to remove points from the system.
Digory 10y ago
The problem is the second assumption of Elo's system, per wikipedia: "if a player wins a game, he is assumed to have performed at a higher level than his opponent for that game. Conversely, if he loses, he is assumed to have performed at a lower level." But you can't assume that about lawsuits. If the client should have paid ten million in damages, the better lawyer might limit the award to nominal damages. But the same nominal damages loss might be a disaster in another case the lawyer should have won. Lawsuits aren't zero-sum, bilateral agreements to compete like most sports. Parties aren't just competing for a 'win,' but looking to maximize gain or minimize loss.

deleted 10y ago

[deleted]

deleted 10y ago

[deleted]

soufron 10y ago

I worked on applying machine learning algorithms to data mined from court judgments during my PhD. You want to know what we learned? That it's better to be white and rich than black and poor. Point. Everything else was bullshit.

selectron 10y ago
How much better?
- soufron 10y ago
  Lol.
  - selectron 10y ago
    It is a legitimate question.
steveeq1 10y ago
What about black and rich vs white and poor?
- sjdchid 10y ago
  Sample size too small I'd guess.

soufron 10y ago

Also, being a Lawyer, if someone comes to me, it's because of me, not because of the firm I am working in. And trying to rank lawyers? Why not. But what's the point. I wouldn't accept being ranked anyway.

hackuser 10y ago
> being a Lawyer, if someone comes to me, it's because of me, not because of the firm I am working in Are you speaking hypothetically, i.e., 'if I was a lawyer'? I'm not an attorney but for a variety of reasons I have relatively deep insight into the legal industry, and I think this statement is generally not true. As in any field, most business comes from referrals and networking, and much of that happens within a firm. If your career is at a top firm then you are going to build a much different network than a similar attorney who opens their own office in a strip mall. When Goldman Sachs needs an outside attorney, where do they go and who do they ask? Even ignoring referrals, just imagine who is there when you are invited to dinner by your colleague: Is it the mayor and local Fortune 500 CEO, one of whom takes a liking to you and asks you to join the board of a local charity, or is it the lovely neighbors who drive Uber or run a small medical practice, and ask for some legal tips? Personally, I don't enjoy the hunt for status and I think it impedes socio-economic mobility, but I'm afraid that's the way it works.
- soufron 10y ago
  I am a lawyer yes. But let's say I have a good cv.
- deleted 10y ago
  [deleted]
- charlesdm 10y ago
  I have a tax lawyer I work with, and I couldn't care less which firm he works at. However, he does work at a good firm, and a good name helps in negotiations (in this case, with the tax authority). A while back, I had a tax audit, and surprisingly the person doing the audit told me my firm can turn "black" into "white". That says quite a bit about the name and reputation of the firm.

charlesdm 10y ago

As someone who works with a tax law firm that is ranked pretty high on the legal500 (and obviously, I know a rank can be gamed relatively easily), I must say the experiences I've had with them have been excellent. For all intents and purposes, they are exactly what I would expect from a good law firm. I worked with several "independent" lawyers before, and honestly, they are nowhere near close to the level of skill of my current law firm. In addition, they're quite responsive, much more than the average independent lawyer. And yes, they're more expensive, but not outrageously so. They seem to reach a favourable outcome faster, and seem to be more effective. Tl;dr: Love the lawyer I work with in this firm, he gets the job done when I need him to + has a great team with broad experience to back him up. He's managed to resolve any tax issue before we even went to court.