3 ms·

The article didn’t include any numbers on what the general lawyer population is compared to the results. For example, they make the claim that solo and small f

The article didn’t include any numbers on what the general lawyer population is compared to the results. For example, they make the claim that solo and small firms are the most likely to file AI hallucinations because they represent 50% and 40% of the instances of legal briefs with hallucinations. However, without the base rate for briefs files by solo or small firms compare to larger firms, we don’t know if that is unusual or not. If 50% of briefs were files by solo firms and 40% were filed by small firms, then the data would actually be showing that firm size doesn’t matter.

cratermoon 1y ago
> They make the claim that solo and small firms are the most likely to file AI hallucinations because they represent 50% and 40% of the instances of legal briefs with hallucinations. Show me where she makes any predictive claims about likelihood. The analysis find that of the cases where AI was used, 90% are either solo practices or small firms. It does not conclude that there's a 90% chance a given claim using AI was done by solo or small firm, or make any other assertions about rates.
- b112 1y ago
  > This analysis confirms what many lawyers and judges may have suspected: that the archetype of misplaced reliance on AI in drafting court filings is a small or solo law practice using ChatGPT in a plaintiff’s-side representation. That is an assertion which requires numbers. If 98% of firms submitting legal briefs are solo or small firms, then the above statement is untrue. The archetype if my prior sentence is true, would be not-small/solo firms. The background data is also suspect. https://www.damiencharlotin.com/hallucinations/ https://www.damiencharlotin.com/hallucinations/ "all documents where the use of AI, whether established or merely alleged, is addressed in more than a passing reference by the court or tribunal. Notably, this does not cover mere allegations of hallucinations, but only cases where the court or tribunal has explicitly found (or implied) that the AI produced hallucinated content." While a good idea, the database is predicated upon news, or people submitting examples as well. There may be some scraping of court documents as well, it's not entirely clear. Regardless, the data is only predictive of people getting called out for such nonsense. A larger firm may have such issues with a lawyer, apologize to the court, have more clout with the court, and replace the lawyer with another employee. This is something a smaller firm cannot do, if it is a firm of one person. It's a nice writeup, and interesting. But it does lead to unverified assertions and conclusions.
dmoy 1y ago
I dunno. By revenue, legal work in the US is super top heavy - it's like 50%+ done by the top 50 firms alone. That won't map 1:1 to briefs, but I would be pretty shocked if large firms only did 10% of briefs.
District5524 1y ago
That's an important observation. It's not easy to get filing data outside the US Federal courts (PACER), because it's not typical at all that courts publish the filings themselves or information on those who file the pleadings. But you can find statistics of the legal market (mainly law firms), like class size (0-10, 10-50, ... 250+ lawyers per firm) of total number of law firms, number of employees per class size of total law firm employees, or revenue per class size. Large firms only dominate the UK, especially in terms of revenue, US is less so, EU is absolutely ruled by solo and small firms. I did some research back in 2019 on this, updates, the figures probably did not change, see page 59-60: https://ai4lawyers.eu/wp-content/uploads/2022/03/Overview-of-the-average-state-of-the-art-IT-capabilities-in-the-EU.pdf https://ai4lawyers.eu/wp-content/uploads/2022/03/Overview-of.... The revenue size statistics was not included in the final publication. You can fish similar data from the SBS dataset of Eurostat https://ec.europa.eu/eurostat/web/main/data/database https://ec.europa.eu/eurostat/web/main/data/database (But the statistical details are pretty difficult to compare with the US or Canada, using different methodologies, different terminologies.)

cratermoon 1y ago

> They make the claim that solo and small firms are the most likely to file AI hallucinations because they represent 50% and 40% of the instances of legal briefs with hallucinations. Show me where she makes any predictive claims about likelihood. The analysis find that of the cases where AI was used, 90% are either solo practices or small firms. It does not conclude that there's a 90% chance a given claim using AI was done by solo or small firm, or make any other assertions about rates.

b112 1y ago
> This analysis confirms what many lawyers and judges may have suspected: that the archetype of misplaced reliance on AI in drafting court filings is a small or solo law practice using ChatGPT in a plaintiff’s-side representation. That is an assertion which requires numbers. If 98% of firms submitting legal briefs are solo or small firms, then the above statement is untrue. The archetype if my prior sentence is true, would be not-small/solo firms. The background data is also suspect. https://www.damiencharlotin.com/hallucinations/ https://www.damiencharlotin.com/hallucinations/ "all documents where the use of AI, whether established or merely alleged, is addressed in more than a passing reference by the court or tribunal. Notably, this does not cover mere allegations of hallucinations, but only cases where the court or tribunal has explicitly found (or implied) that the AI produced hallucinated content." While a good idea, the database is predicated upon news, or people submitting examples as well. There may be some scraping of court documents as well, it's not entirely clear. Regardless, the data is only predictive of people getting called out for such nonsense. A larger firm may have such issues with a lawyer, apologize to the court, have more clout with the court, and replace the lawyer with another employee. This is something a smaller firm cannot do, if it is a firm of one person. It's a nice writeup, and interesting. But it does lead to unverified assertions and conclusions.

dmoy 1y ago

I dunno. By revenue, legal work in the US is super top heavy - it's like 50%+ done by the top 50 firms alone. That won't map 1:1 to briefs, but I would be pretty shocked if large firms only did 10% of briefs.

District5524 1y ago

That's an important observation. It's not easy to get filing data outside the US Federal courts (PACER), because it's not typical at all that courts publish the filings themselves or information on those who file the pleadings. But you can find statistics of the legal market (mainly law firms), like class size (0-10, 10-50, ... 250+ lawyers per firm) of total number of law firms, number of employees per class size of total law firm employees, or revenue per class size. Large firms only dominate the UK, especially in terms of revenue, US is less so, EU is absolutely ruled by solo and small firms. I did some research back in 2019 on this, updates, the figures probably did not change, see page 59-60: https://ai4lawyers.eu/wp-content/uploads/2022/03/Overview-of-the-average-state-of-the-art-IT-capabilities-in-the-EU.pdf https://ai4lawyers.eu/wp-content/uploads/2022/03/Overview-of.... The revenue size statistics was not included in the final publication. You can fish similar data from the SBS dataset of Eurostat https://ec.europa.eu/eurostat/web/main/data/database https://ec.europa.eu/eurostat/web/main/data/database (But the statistical details are pretty difficult to compare with the US or Canada, using different methodologies, different terminologies.)