6 ms·

> By applying machine learning algorithms to data mined from court judgments... And, with that sentence, this post becomes bullshit. The overwhelming majority

> By applying machine learning algorithms to data mined from court judgments... And, with that sentence, this post becomes bullshit. The overwhelming majority of disputes do not reach litigation. The overwhelming majority of litigation does not reach judgment. And then a gigantic proportion of those judgments are appealed, and a non-trivial percentage are overturned. Even at that, only on the order of a single digit-percentage of disputes ever reaches a verdict. This is like judging a hospital-system based on its success-rate for transplant surgery. It's just deeply ignorant of what constitutes the vast, vast majority of the practice of law - even in disputes. This article has no value, whatsoever, aside from a witty aside at a cocktail party about how deeply misunderstood the legal profession is. This is particularly shocking given that litimetrics is a legal service provider. > Although A has not faced B, if both have faced C, then we have some information about the hypothetical A and B matchup. This is the same reasoning that allows one to compare tennis players across different eras. The implication is that the lawyer efficacy, based on judgments, is subject to the transitive property? Sweet mercy that is ridiculous. This post is a catastrophe.

sandworm101 10y ago
I'd add that the majority of civil cases are resolved through non-public deals whereby no third party can tell who has won or lost. That said, it might be interesting to apply machine learning at the appellate level. There, the outcomes are public and more readable by machines. Something as simple as looking for tell-tales such as "reverse", "dismissed" or "we side with the appellant" in holdings could be mapped to firms.
- monocasa 10y ago
  Hell, if the lawyers have done a good job, both sides walk away from a settlement thinking that they won.
  - libertymcateer 10y ago
    Or that neither won.
wikiwawa 10y ago
A large part of statistics is devoted to making valid inference from noisy samples, largely by making sensible assumptions. Even in your example, it is common practice to assess a surgeon on outcomes of surgery, even though a lot of a surgeon's work does not lead to surgery. We aren't claiming to assess the wide practice of law, just litigation. We do Litigation Analytics.
- umanwizard 10y ago
  do you understand the difference between "noisy" and "biased"?
  - wikiwawa 10y ago
    I understand sarcasm, that's for sure. As a matter develops, parties tend to drop out if they think that their chances have declined beyond some threshold. But if they continue, or more likely, their legal representation suggests they continue, it implies that they think they have a good chance of success. Of course, you can get irrational litigants but most of the time, if it goes to judgement, both parties think they have a high chance of success. Otherwise they would have bailed.
- andrewfong 10y ago
  What do you count as a "court judgment" and how are they evaluated? Are, for instance, a successful motion to dismiss and a jury decision for the defendant weighed equally? Or do you give higher weight to the motion to dismiss since that's usually a better outcome for the defendant than making it to trial? And do you look at motions other than actual judgments? I.e. if Firm A wins a motion to suppress some piece of evidence and then there's a subsequent settlement, we should infer that Firm A "won" the case, even if there's no judgment to that effect.
- hobbyjogger 10y ago
  So you sue me and get a judgment for $100,000. If I felt that I owed you nothing, surely this counts as a "loss" for me at trial. If, however, you had originally sued for $50M and I felt I might owe you some money but no more than $1M so I decided to fight it, the $100k judgment would be considered an enormous "win" for me. What data does your analysis use to distinguish between the two scenarios given that the verdicts are identical?
hellcow 10y ago
Your comment is filled with unhelpful hyperbole that makes civil, constructive discussion much more difficult. It seems reasonable to me that a firm with a great track record of judgments is more likely to succeed in other significant areas than a firm without a similarly great track record of judgments. That is, I would expect a strong, positive correlation between winning judgments and other desirable outcomes in disputes, like winning settlements. We could even measure that correlation through a representative sample of firms to prove or disprove that hypothesis (and the OP probably should), but dismissing it outright with hyperbole isn't an effective response.
- ska 10y ago
  It seems reasonable to me that a firm with a great track record of judgments is more likely to succeed in other significant areas than a firm without a similarly great track record of judgments. That seems like as strong enough assumption to require quite careful analysis and support. I know very little about the practice of law, but I am certain many other fields of endeavor just don't work like this.
savanaly 10y ago
Doesn't your whole post rely on the assumption that outcomes in the cases they measured don't correlate closely with outcomes in the (far bigger array of) cases they didn't measure? Don't you have to show that before you can definitely say their analysis is completely invalid?
- jessaustin 10y ago
  Why should we expect a correlation? Why should we assume they even thought about the issue, if it isn't mentioned in their analysis?
  - selectron 10y ago
    Why should we expect there isn't a correlation? They do mention it in their analysis. At the end of the day, just because there might be a systematic bias in your result doesn't mean there is a systematic bias. All real-world analysis (especially for observational studies) rests on certain assumptions. It is always true that these assumptions might be wrong, but it is important to think about whether or not the assumption is plausible. It seems plausible that on average, a lawyer who is able to get better outcomes when they don't settle is also able to get better outcomes when they do settle. Furthermore, even if most cases are settled, the rare cases that do go to trial can have an outsized impact. Usually people settle because a bad judgment is devastating (as well as not wanting to pay legal costs).
- wahern 10y ago
  Depends on what is meant by bullshit. Yes, if it meant that the study was absolutely wrong. No, if it meant that the study failed to provide any evidence whatsoever that there exists a correlation. In science guess who has the burden? In a legal context the burden of proof is broken down into the burden of production and the burden of persuasion. If you fail to meet the burden of production it's fair to say a claim is bullsh*t. It necessarily follows that if you haven't met the burden of production you cannot have met the burden of persuasion as a logical matter. Remember, any claim must stand on its own terms. It doesn't matter what is the reality or truth of the matter. We can never know the absolute truth of something. We can only attain a slightly firmer grip on reality when people propose persuasive arguments. Maybe you can mine a failed argument for useful bits, but a good rule of thumb is that it's not worth the time if the argument cannot even meet a minimal burden. If it wasn't worth the claimant's time why would it be worth yours? That's strong circumstantial evidence to move on.
_rpd 10y ago
This is just a case study of law firm rankings in general. Popular law firm ranking systems use criteria that may not be relevant to your business. When ranking law firms, you should use a custom ranking system.
ramblenode 10y ago
> This post is a catastrophe. This is a pretty ungenerous reading of the article. Your complaint seems to be mostly about the generalizability of the model (a legitimate complaint, as you've explained) rather than its statistical validity. It still provides some interesting insights about the performance of firms at trial. Notably, there isn't a correlation between favorable judgments in those cases which have gone to trial and traditional law firm rankings. That's surprising, even if it doesn't generalize to overall law firm "success".
busterarm 10y ago
> This is particularly shocking given that litimetrics is a legal service provider. Not so shocking. Law firm ranking services are their competition. They drew a conclusion from the data that supports their business. Law firm rankings absolutely can provide a value. Both on the firm and client side of the transaction. Imagine you're on the client side of a qui tam (Whistleblower) or Immigration case. Are you going to bring that case to a law firm that you don't have visibility into a) how the firm is trusted by their clients and b) how the law firm is regarded among other law firms? That would be crazy! Heck, in a qui tam you can't even get quality analytics into how well a firm litigates a case because the best qui tam cases get picked up and tried by the Federal Government! (Disclosure: I'm a dev at a law firm.)
- rrr298 10y ago
  Rankings seem to be very subjective and to be a function of firm biz dev teams: http://www.chambersandpartners.com/submissions http://www.chambersandpartners.com/submissions Size of marketing team might correlate with success/positive market perception though.
downandout 10y ago
While not all cases go to trial, it is important to know what your chances would be at trial with that particular firm. This is the same metric that the other side will primarily look at when considering the terms of any settlement. Therefore, I disagree that this post "becomes bullshit" merely because it relies on trial outcomes.
- busterarm 10y ago
  Having spent the last 12 years of my life wrapped up in a civil trial related to estate law and learned a ton along the way about how lawyers have failed me, I would much rather have known how often a firm's cases go to trial and if not, which way the case went. It will take _years_ before your case actually goes to trial and you will meet plenty of lawyers along the way if you aren't careful who will happily fuck off with your money. And you'll have to sue them separately to deal with that, if you even have the energy to spare. Good luck having a full-time job through that mess.
- libertymcateer 10y ago
  > While not all cases go to trial, it is important to know what your chances would be at trial with that particular firm. Read above to my last reply. This is really a poor way to think about law firms. Is the firm a toxic tort firm? Is it an appellate advocacy firm? Is it maritime law? Is it an intellectual property dispute? A business dispute? Win percentages are totally meaningless without more information, including client-positive settlement rates. This metric is totally useless.
  - downandout 10y ago
    >This metric is totally useless. You honestly believe that an opponent won't take a 90%+ trial win rate into consideration when considering a settlement? All you hear from lawyers pretrial is "our chances at trial are X" and then based on whether X is high or low, they'll make a decision to go to trial or settle.
    - libertymcateer 10y ago
specialist 10y ago
"The overwhelming majority of disputes do not reach litigation. ..." I'm shopping for a personal injury lawyer. How to pick? The two referrals I got didn't pan out. I don't know any lawyers. Aha, search the court documents to see who the winners are! I (finally) consulted my paralegal friend. He made the same points you have. Additionally, he said the courts all use different IT systems. One could use an expensive service to do the search, but it's hardly worth it for laypersons. Too bad. I did start to wonder if one could data mine the filings, just to see who's getting the most cases, and maybe infer something actionable from that.

sandworm101 10y ago

I'd add that the majority of civil cases are resolved through non-public deals whereby no third party can tell who has won or lost. That said, it might be interesting to apply machine learning at the appellate level. There, the outcomes are public and more readable by machines. Something as simple as looking for tell-tales such as "reverse", "dismissed" or "we side with the appellant" in holdings could be mapped to firms.

monocasa 10y ago
Hell, if the lawyers have done a good job, both sides walk away from a settlement thinking that they won.
- libertymcateer 10y ago
  Or that neither won.

wikiwawa 10y ago

A large part of statistics is devoted to making valid inference from noisy samples, largely by making sensible assumptions. Even in your example, it is common practice to assess a surgeon on outcomes of surgery, even though a lot of a surgeon's work does not lead to surgery. We aren't claiming to assess the wide practice of law, just litigation. We do Litigation Analytics.

umanwizard 10y ago
do you understand the difference between "noisy" and "biased"?
- wikiwawa 10y ago
  I understand sarcasm, that's for sure. As a matter develops, parties tend to drop out if they think that their chances have declined beyond some threshold. But if they continue, or more likely, their legal representation suggests they continue, it implies that they think they have a good chance of success. Of course, you can get irrational litigants but most of the time, if it goes to judgement, both parties think they have a high chance of success. Otherwise they would have bailed.
andrewfong 10y ago
What do you count as a "court judgment" and how are they evaluated? Are, for instance, a successful motion to dismiss and a jury decision for the defendant weighed equally? Or do you give higher weight to the motion to dismiss since that's usually a better outcome for the defendant than making it to trial? And do you look at motions other than actual judgments? I.e. if Firm A wins a motion to suppress some piece of evidence and then there's a subsequent settlement, we should infer that Firm A "won" the case, even if there's no judgment to that effect.
hobbyjogger 10y ago
So you sue me and get a judgment for $100,000. If I felt that I owed you nothing, surely this counts as a "loss" for me at trial. If, however, you had originally sued for $50M and I felt I might owe you some money but no more than $1M so I decided to fight it, the $100k judgment would be considered an enormous "win" for me. What data does your analysis use to distinguish between the two scenarios given that the verdicts are identical?

hellcow 10y ago

Your comment is filled with unhelpful hyperbole that makes civil, constructive discussion much more difficult. It seems reasonable to me that a firm with a great track record of judgments is more likely to succeed in other significant areas than a firm without a similarly great track record of judgments. That is, I would expect a strong, positive correlation between winning judgments and other desirable outcomes in disputes, like winning settlements. We could even measure that correlation through a representative sample of firms to prove or disprove that hypothesis (and the OP probably should), but dismissing it outright with hyperbole isn't an effective response.

ska 10y ago
It seems reasonable to me that a firm with a great track record of judgments is more likely to succeed in other significant areas than a firm without a similarly great track record of judgments. That seems like as strong enough assumption to require quite careful analysis and support. I know very little about the practice of law, but I am certain many other fields of endeavor just don't work like this.

savanaly 10y ago

Doesn't your whole post rely on the assumption that outcomes in the cases they measured don't correlate closely with outcomes in the (far bigger array of) cases they didn't measure? Don't you have to show that before you can definitely say their analysis is completely invalid?

jessaustin 10y ago
Why should we expect a correlation? Why should we assume they even thought about the issue, if it isn't mentioned in their analysis?
- selectron 10y ago
  Why should we expect there isn't a correlation? They do mention it in their analysis. At the end of the day, just because there might be a systematic bias in your result doesn't mean there is a systematic bias. All real-world analysis (especially for observational studies) rests on certain assumptions. It is always true that these assumptions might be wrong, but it is important to think about whether or not the assumption is plausible. It seems plausible that on average, a lawyer who is able to get better outcomes when they don't settle is also able to get better outcomes when they do settle. Furthermore, even if most cases are settled, the rare cases that do go to trial can have an outsized impact. Usually people settle because a bad judgment is devastating (as well as not wanting to pay legal costs).
wahern 10y ago
Depends on what is meant by bullshit. Yes, if it meant that the study was absolutely wrong. No, if it meant that the study failed to provide any evidence whatsoever that there exists a correlation. In science guess who has the burden? In a legal context the burden of proof is broken down into the burden of production and the burden of persuasion. If you fail to meet the burden of production it's fair to say a claim is bullsh*t. It necessarily follows that if you haven't met the burden of production you cannot have met the burden of persuasion as a logical matter. Remember, any claim must stand on its own terms. It doesn't matter what is the reality or truth of the matter. We can never know the absolute truth of something. We can only attain a slightly firmer grip on reality when people propose persuasive arguments. Maybe you can mine a failed argument for useful bits, but a good rule of thumb is that it's not worth the time if the argument cannot even meet a minimal burden. If it wasn't worth the claimant's time why would it be worth yours? That's strong circumstantial evidence to move on.

_rpd 10y ago

This is just a case study of law firm rankings in general. Popular law firm ranking systems use criteria that may not be relevant to your business. When ranking law firms, you should use a custom ranking system.

ramblenode 10y ago

> This post is a catastrophe. This is a pretty ungenerous reading of the article. Your complaint seems to be mostly about the generalizability of the model (a legitimate complaint, as you've explained) rather than its statistical validity. It still provides some interesting insights about the performance of firms at trial. Notably, there isn't a correlation between favorable judgments in those cases which have gone to trial and traditional law firm rankings. That's surprising, even if it doesn't generalize to overall law firm "success".

busterarm 10y ago

> This is particularly shocking given that litimetrics is a legal service provider. Not so shocking. Law firm ranking services are their competition. They drew a conclusion from the data that supports their business. Law firm rankings absolutely can provide a value. Both on the firm and client side of the transaction. Imagine you're on the client side of a qui tam (Whistleblower) or Immigration case. Are you going to bring that case to a law firm that you don't have visibility into a) how the firm is trusted by their clients and b) how the law firm is regarded among other law firms? That would be crazy! Heck, in a qui tam you can't even get quality analytics into how well a firm litigates a case because the best qui tam cases get picked up and tried by the Federal Government! (Disclosure: I'm a dev at a law firm.)

rrr298 10y ago
Rankings seem to be very subjective and to be a function of firm biz dev teams: http://www.chambersandpartners.com/submissions http://www.chambersandpartners.com/submissions Size of marketing team might correlate with success/positive market perception though.

downandout 10y ago

While not all cases go to trial, it is important to know what your chances would be at trial with that particular firm. This is the same metric that the other side will primarily look at when considering the terms of any settlement. Therefore, I disagree that this post "becomes bullshit" merely because it relies on trial outcomes.

busterarm 10y ago
Having spent the last 12 years of my life wrapped up in a civil trial related to estate law and learned a ton along the way about how lawyers have failed me, I would much rather have known how often a firm's cases go to trial and if not, which way the case went. It will take _years_ before your case actually goes to trial and you will meet plenty of lawyers along the way if you aren't careful who will happily fuck off with your money. And you'll have to sue them separately to deal with that, if you even have the energy to spare. Good luck having a full-time job through that mess.
libertymcateer 10y ago
> While not all cases go to trial, it is important to know what your chances would be at trial with that particular firm. Read above to my last reply. This is really a poor way to think about law firms. Is the firm a toxic tort firm? Is it an appellate advocacy firm? Is it maritime law? Is it an intellectual property dispute? A business dispute? Win percentages are totally meaningless without more information, including client-positive settlement rates. This metric is totally useless.
- downandout 10y ago
  >This metric is totally useless. You honestly believe that an opponent won't take a 90%+ trial win rate into consideration when considering a settlement? All you hear from lawyers pretrial is "our chances at trial are X" and then based on whether X is high or low, they'll make a decision to go to trial or settle.
  - libertymcateer 10y ago

specialist 10y ago

"The overwhelming majority of disputes do not reach litigation. ..." I'm shopping for a personal injury lawyer. How to pick? The two referrals I got didn't pan out. I don't know any lawyers. Aha, search the court documents to see who the winners are! I (finally) consulted my paralegal friend. He made the same points you have. Additionally, he said the courts all use different IT systems. One could use an expensive service to do the search, but it's hardly worth it for laypersons. Too bad. I did start to wonder if one could data mine the filings, just to see who's getting the most cases, and maybe infer something actionable from that.