9 ms·
India's top court angry after junior judge cites fake AI-generated orders
- voidUpdate 7mo agoHow many of these cases do we have to have before lawyers realise that they need to check that the things an LLM tells them are actually true?
- LunaSea 7mo agoIt doesn't matter anymore. LLMs just revealed what a decadent society we have setup for ourselves worldwide.
- zthrowaway 7mo agoDo we see this a lot in the US? This seems to be more unique to India.
- duskdozer 7mo agotoday: https://news.ycombinator.com/item?id=47231189 https://news.ycombinator.com/item?id=47231189
- tw04 7mo agoIt’s happening A LOT in the US too. Mainstream media just doesn’t seem to find it that newsworthy. https://arstechnica.com/tech-policy/2026/02/randomly-quoting-ray-bradbury-did-not-save-lawyer-from-losing-case-over-ai-errors/ https://arstechnica.com/tech-policy/2026/02/randomly-quoting...
- malshe 7mo agoFrom the article: > In October, two federal judges in the US were called out for the use of AI tools which led to errors in their rulings. In June 2025, the High Court of England and Wales warned lawyers not to use AI-generated case material after a series of cases cited fictitious or partially made up rulings.
- coffeefirst 7mo agoIt’s worse than that. We’re hearing about the lawyers and Ars Technica because the consequences are public and the errors are egregious. It’s likely happening to everyone.
- duskdozer 7mo agoI'm continually amazed at how much faith people have in them. I guess since they can sound like people and output really authoritative and confident text it just overrides any skepticism subconsciously?
- pjc50 7mo agoThe advertising campaign is incredible.
- ben_w 7mo agoMuch as I like them, I do frequently remind myself of two things: 1) https://en.wikipedia.org/wiki/Clever_Hans https://en.wikipedia.org/wiki/Clever_Hans 2) https://archive.org/details/nextgen-issue-26 https://archive.org/details/nextgen-issue-26 as an example of how in the 90s we has rapid cycles of a new tech (3d graphics) astounding us with how realistic each new generation was compared to the previous one, and forgetting with each new (game engine) how we'd said the same and felt the same about (graphics) we now regarded as pathetic. So yes, they do sound "authoritative and confident text it just overrides any skepticism subconsciously", but you shouldn't be amazed, we've always been like this.
- moron4hire 7mo agoIt's mind boggling how much people claim to like LLMs when you would never design any other piece of software to operate like LLMs do. Designing a system that interact with the user through natural text creates an awful experience. It slows down every interaction as you dig through all the prose to get to the key information. It turns every computer interaction into a school math word problem.
- PunchyHamster 7mo agoYes, just as with politicians. And LLMs have been thoroughly tuned to appear that
- direwolf20 7mo agohttps://en.wikipedia.org/wiki/ELIZA_effect https://en.wikipedia.org/wiki/ELIZA_effect
- Latty 7mo agoIt doesn't matter, because any process that seems right most of the time but occasionally is wrong in subtle, hard to spot ways is basically a machine to lull people into not checking, so stuff will always slip through. It's just like the cars driving themselves but you need to be able to jump in if there is a mistake, humans are not going to react as fast as if they were driving, because they aren't going to be engaged, and no one can stay as engaged as they were when they were doing it themselves. We need to stop pretending we can tell people they "just" need to check things from LLMs for accuracy, it's a process that inevitably leads to people not checking and things slipping through. Pretending it's the people's fault when essentially everyone using it would eventually end up doing that is stupid and won't solve the core problem.
- voidUpdate 7mo agoProbably worth including a "bibliography" section of citations that can be automatically checked that they actually exist then
- lazide 7mo agoNot enough - you’d also need to check that they say/mean what is being implied. Which is a real problem.
- mminer237 7mo agoTo be fair, that's a problem with human authors too. Wikipedia is really well-cited, but it's common to check a citation and find it only says half of what a sentence does, while the rest seemingly has no basis in fact. Judges are supposed to actually read the citations to not only confirm the case exists and says what's being claimed, but often to also compare & contrast the situations to ensure that principle is applicable to the case at hand.
- lazide 7mo agoYup. The issue with LLMs are not that any specific thing it is doing is unique. Rather that it does it in previously unimaginable volume, scale, and accessibility.
- YeGoblynQueenne 7mo agoWhat kind of AI is this that you constantly need a human to check its job? Do you think Jean-Luc Piccard had to constantly check the output of the Enterprise computer? No he didn't. If AI is not better than humans, then what the heck is the point? You might as well just use humans.
- AJ007 7mo agoThis whole thing is silly, LLMs can automate reference validation. If someone is a lawyer, accountant, doctor, teacher, surgeon, engineer etc, and is regurgitating answers that were pumped out with with GPT-5-extra-low or whatever mediocre throttled model they are using, they should just be fired and de-credentialed. Right now this is easy. The real problem is ahead: 99.999% of future content that exists will be made using generative AI. For many people using Facebook, Instagram, TikTok, or some other non-sequential, engagement weighted feed, 50%+ of the content they consume today is fake. As that stuff spreads in to modern culture it's going to be an endless battle to keep it out of stuff that should not be publishing fake content (e.g. the New York Times or Wall Street Journal; excluding scientific journals who seem to abandoned validation and basic statistics a long time ago.) Much of the future value and profit margins might just be in valid data?
- raincole 7mo ago> Right now this is easy. Easy? In the US you need house impeachment to fire a judge. In some countries judges are completely immune unless they are sentenced for crimes.
- voidUpdate 7mo agoDo you need impeachment to fire a lawyer, accountant, doctor, teacher, surgeon or engineer?
- raincole 7mo agoNope, and the article is about a judge. What's the point to incentive lawyers to carefully verify their references when they know the judge has no incentive to read them and can just make shit up anyway?
- mminer237 7mo agoTo fire a federal judge. Local judges, which are the vast majority, can be fired by their colleagues or replaced in elections.
- 7mo ago
- probably_wrong 7mo agoJust this week I tracked down the citations of a scientific paper (whose authors could very well be here) where 25% of the citations were made up and 50% of the remaining ones were wrong, taking ArXiv papers and citing them as belonging to (say) IJCLR. It's not just lawyers.
- codegladiator 7mo ago> She had no intention to misquote or misrepresent the rulings and that "the mistake occurred solely due to the reliance on an automatic source", the high court wrote I don't think the intention matters here. Its the same deal with every profession using llm to "automate" their work. The onus in on the professional, not the llm. Arstechnica case could have been justified by same manner otherwise. Not knowing the law isnt execuse to break law, so why is not knowing the tool an excuse to blame the tool.
- hypeatei 7mo agoThis is why LLMs won't replace humans wholesale in any profession: you can't hold a machine accountable. Most of the chatbot experiences I have with various support channels always end up with human intervention anyway when it involves money. Maybe true general intelligence would solve these issues, but LLMs aren't meeting that threshold anytime soon, imo. Stochastic parrots won't rule the world.
- lazide 7mo agoEven ‘true general intelligence’ (if we count humans as that) screws up frequently, sometimes (often?) intentionally for it’s own benefit - which is why accountability is such a necessary element. If someone won’t be held liable for the end result at some point, then there is no reason to ensure an even somewhat reasonable end result. It’s fundamental. Which is also why I suspect so many companies are pushing ‘AI’ so hard - to be able to do unreasonable things while having a smokescreen to avoid being penalized for the consequences.
- hypeatei 7mo ago> to be able to do unreasonable things while having a smokescreen Maybe, but I feel like the calculus remains unchanged for professions that already lack accountability (police, military, C-suite, three letter agencies, etc.); LLMs are yet another tool in their toolbox to obfuscate but they were going to do that anyway. Peons will continue to face consequences and sanctions if they screw up by using hallucinated output.
- alansaber 7mo agoThis is a big problem in the US and UK too. Lawyers are not technical at all and they need a robust system of governance, since currently they're (directly editing, not even diffing) documents with a chatbot which makes these mistakes inevitable. See https://insights.doughtystreet.co.uk/post/102mi96/38-uk-cases-involving-hallucinations-ai-or-otherwise-judicial-caution-in-the-f https://insights.doughtystreet.co.uk/post/102mi96/38-uk-case...
- cmiles8 7mo agoThere will be many more things like this and it’s an elephant in the room for the supposed mass replacement of people with AI. Some human still has to be accountable. Someone has to get fired / go to jail when something screws up. You can make humans more productive but for the foreseeable future you can’t take the human out of the loop to have an AI implementation that’s not a disaster/lawsuit waiting to happen. That, probably more than anything else, is why companies just aren’t seeing the much promised mass step change in productivity from AI and why so many companies are now saying they see zero ROI from AI efforts. The lowest hanging fruit will be low value rote repetitive tasks like the whole India offshoring industry, which will be the first to vaporize if AI does start replacing humans. But until companies see success on the lowest of lowest hanging fruit on en-mass labor replacement with AI things higher up on the value chain will remain relatively safe. PS: Nearly every mass layoff recently citing “AI productivity” hasn’t withstood scrutiny. They all seem to be just poorly performing companies slashing staff after overhiring, which management looking for any excuse other than just admitting that.
- fidotron 7mo ago> Some human still has to be accountable. Someone has to get fired / go to jail when something screws up. The turning point will be when threatening an AI with being unplugged for screwing up works in motivating it to stop making things up. Some people will rightly point out that is kind of what the training process is already. If we go around this loop enough times it will get there.
- Hendrikto 7mo agoYou are making a lot of assumptions here. You assume, among other things, that AI has self-preservation drive, can be threatened, can be motivated, and above all that we know how to accomplish that and are already doing so. I would dispute all of that.
- yes_man 7mo agoFor now maybe not. (Maybe). But just as evolution in nature, isn’t it likely that in the future the AIs that have a preservation drive are the ones that survive and proliferate? Seeing they optimize for their survival and proliferation, and not blindly what they were trained on. I am not discounting this happening already, not by the LLMs necessarily being sentient but at least being intelligent enough to emulate sentience. It’s just that for now, humanity is in control of what AI models are being deployed.
- chris_wot 7mo agoIn Australia, our universities are finding that a large proportion of Indian students have been using GenAI for cheating. Often they get away with it. I'm not saying that people other than Indian overseas students cheat, but it does seem more entrenched. I'd love to know why. It doesn't actually help in the long term!
- n1b0m 7mo agoIndian students have embraced GenAI at a rate significantly higher than the global average, with nearly 90% of students in some surveys actively using these tools. Government Policy and National Initiatives: The National Education Policy (NEP 2020) has shifted the focus toward digital literacy. The government has introduced AI as a skill subject for younger grades and launched programs like AI for All to promote nationwide awareness.
- deleted 7mo ago[deleted]
- love2read 7mo agoIn the United States, cheating via AI is now rampant regardless of ethnicity. I know little of Australian Universities but I would assume it’s similar over there.
- PunchyHamster 7mo agoI'd imagine they are just being worse at hiding it. GenAI is rampant pretty much everywhere in school system of most countries
- DaedalusII 7mo ago>The number of international students studying in Australia totalled 833,041 for the January-October 2025 period >The United States hosts the highest number of international students on record, with approximately 1.1 to 1.2 million The US has 32% more students than Australia and 1121% more people. Imagine if the US took on 13 million foreign college students per year lol It does help them in the long run, because it ensures they get to reside in australia. after 4 years they get permanent residence rights and benefits, etc
- arashsadrieh 7mo ago[flagged]
- love2read 7mo agoAwesome ai generated comment. Seriously when will @dang integrate pangram into comments?
- dang 7mo agoDoes pangram work?
- refulgentis 7mo agoAlas, tl;dr: no. (it caught the one above, easily, but didn't catch ones from Opus 4.6, it's very confident they're 100% human. as I would be :/ )
- love2read 7mo agoWhy is that a tldr no if it did indeed catch the blatant ai comment? Are you funded by the competitor? Lol
- refulgentis 7mo agoBecause it caught one of four :) because it's unclear what "works" means, I spelled it all out. :)
- sathish316 7mo agoNext token prediction and Hallucination as a bug. This should be of deep concern to all Frontier labs, who think Integrity and Trust is optional when LLMs are used this way in places where it's most important.
- ionwake 7mo agoone should also consider that even with fake hallucinated AI situation, the productivity and correctness of the work produced by the culprit ( in general ) may still have been of higher quality then before AI regardless of the fails
- Hendrikto 7mo agoHard to believe when this judge apparently thought that outsourcing their — extremely confidential, sensitive, and important — work to a known unreliable tool was a good idea. And then further thought that they apparently did not even need to check the results. Sound like extreme incompetence or laziness.
- ionwake 7mo agogood point. I was thinking of jobs where you can put stuff into the Ai. No need for the downvotes my point still holds
- kaptainscarlet 7mo agoThere will be loads of papers and publications with fake citation. AI will be trained on these. In the end, we'll have more and more hallucinated information that true content on the internet.
- dartharva 7mo agoThe scary thing is that Indian juduciary is infamous for being incapable of tolerating any kind of criticism against it and not hesitating to put people in jail for "contempt" for just calling out corruption. Imagine the official courts of 1.4B+ people being run by such braindead narcissists, now unhindered with having to even pretend to do their jobs as they just offload everything to AI tools.
- rramadass 7mo agoYou are way off-base here; see - https://news.ycombinator.com/item?id=47235274 https://news.ycombinator.com/item?id=47235274
- dartharva 7mo agoSee - https://news.ycombinator.com/item?id=47236336 https://news.ycombinator.com/item?id=47236336
- rramadass 7mo agoBut this is a different issue than AI usage in Indian Judiciary which is the topic of this thread. Also the supreme court gave their justifications which is not entirely wrong. It is one thing to criticize the judiciary in the open press and quite another to add deliberately mischievous passages in a school textbook.
- sillywalk 7mo ago> The scary thing is that Indian juduciary is infamous for being incapable of tolerating any kind of criticism against it and not hesitating to put people in jail for "contempt" for just calling out corruption. From [0]: "India's Supreme Court has banned a school textbook after a chapter in it made a reference to corruption in the judiciary. The revised social science book was published by the National Council of Educational Research and Training (NCERT), which designs the syllabus and textbooks for millions of schoolchildren in the country. On Wednesday, after Chief Justice Surya Kant criticised the book, saying it could damage the reputation of the judiciary, NCERT apologised and withdrew it from distribution. Now the court has ordered a complete halt on the book's publication, saying its contents were "extremely contemptuous" and "reckless". "A complete blanket ban is hereby imposed on any further publication, reprinting or digital dissemination of the book," the court said on Thursday, according to legal news website LiveLaw. The judges also issued notices to the top bureaucrat in the school education department and the NCERT director, asking them to explain why they should not be held in contempt of court for including the "offending chapter". [0] https://www.bbc.com/news/articles/c627l7zexr8o https://www.bbc.com/news/articles/c627l7zexr8o
- jfengel 7mo agoI feel like this points out a very general problem with the law: it generates a lot of boilerplate text. Lawyers don't really read it; they skim it for the relevant bits. Obviously lawyers should not be cheating with AI, especially when they don't even check it. But it does sound to me as if this is an opportunity to re-factor the process. We're carrying forward some ideas originally implemented in Latin, and which can be dramatically simplified. I'm not a lawyer; I know this only in passing. And I am aware that there are big differences between law and code. But every time I encounter the law, and hear about cases like this, what I see are vast oceans of text that can surely be made more rigorous. AI is not the problem; it's pointing out the opportunity.
- loremium 7mo agolaw texts feel like a layering problem, like just decoration around decoration to avoid breaking existing 'code' without ever simplifying it
- MagicMoonlight 7mo agoOkay so let’s try simplifying it. We’ll change the existing murder legislation to “Killing someone is a crime”. It’ll save us thousands of pages. But does that mean a soldier shooting an enemy is a crime? What about shooting someone who is raping you? What if you shoot someone by mistake, thinking they’re going to kill you? What if you hit them with a car? What if you fail to provide safety equipment which eventually results in their accidental death? Oopsie woopsie, I guess we need to add another thousand pages of exceptions back to our simplistic laws. It turns out people didn’t just write them for the fun of it.
- jfengel 7mo agoThose exceptions were accumulated over time rather than compiled into a coherent schema. I don't expect a finished product to be a single page, but neither does it require the literally millions that it currently takes up.
- deleted 7mo ago
- gracelynewhouse 7mo ago[flagged]
- MagicMoonlight 7mo agoYou wrote this slop with AI right?
- bass_girl 7mo agothe irony...
- mikelitoris 7mo ago[flagged]
- paxys 7mo agoYou are right this would never happen in an advanced country like the USA, and certainly not in a top Federal court https://www.reuters.com/sustainability/society-equity/two-federal-judges-say-use-ai-led-errors-us-court-rulings-2025-10-23/ https://www.reuters.com/sustainability/society-equity/two-fe...
- malshe 7mo agoYou had two choices: 1) read the article or 2) be racist about Indians. You chose 2. From the article: > In October, two federal judges in the US were called out for the use of AI tools which led to errors in their rulings. In June 2025, the High Court of England and Wales warned lawyers not to use AI-generated case material after a series of cases cited fictitious or partially made up rulings.
- deleted 7mo ago[deleted]
- zkmon 7mo agoAndhra is like silicon valley of India. Wouldn't blame the poor judge.
- newzino 7mo ago[flagged]
- whatever1 7mo agoThe vast majority of court cases lead to dismissal. Why not use AI to adjudicate cases, and if it is dismissal, dismissal it is. If not then move to a proper court. This way the backlog of cases will significantly drop, and we will work only on cases that there is enough meat to lead to a conviction.
- dyauspitr 7mo agoOr atleast use the AI to prioritize and sort cases will a tentative decision that is not binding.
- arunc 7mo ago> Senior judges at the Supreme Court in Delhi have threatened consequences over the use of AI Setting AI aside for a moment, this reflects a broader issue in India and elsewhere. When institutions respond to new technologies with anger or threats rather than systemic thinking, it signals a deeper problem. The real challenge is not AI itself, but how complex systems adapt to change. Instead of reacting defensively, institutions should anticipate second-order effects, build regulatory capacity, and treat this as a governance and systems problem. Mature institutions approach disruption with foresight, incentives, and feedback loops, not emotions. Without that shift, they risk reinforcing outdated hierarchies rather than serving the public effectively.
- digitalPhonix 7mo ago> Setting AI aside for a moment, this reflects a broader issue in India and elsewhere. When institutions respond to new technologies with anger or threats rather than systemic thinking, it signals a deeper problem. No, especially in this case when the first appeal to the high court resulted in the high court brushing it off as if nothing happened. It was a reprimand two to institutes (the trial court and the high court) they have a job to do and they can't shirk that responsibility.
- rramadass 7mo agoNo, you are reading it wrong; The Indian Supreme Court is doing the right thing. The lower courts in India are all overloaded with pending cases (i.e. not enough judges) and so the incentives for both judges and lawyers to "outsource" to AI is very high. This needs to be done with caution and that is what the supreme court said, viz; The Supreme Court called the case a matter of "institutional concern" and said fake AI-generated judgements had "a direct bearing on integrity of adjudicatory process". ... The defendants challenged the order in the state's high court, pointing out that the cited orders were fake. The high court acknowledged this, but accepted that the junior civil judge had made the error in "good faith" and went on to agree with the trial court's decision anyway. In its order, the high court said that "the citations may be non-existent, but if the learned trial court has considered the correct principles of law and its application to the facts of the case is also correct, mere mentioning of incorrect or non-existent rulings/citations in the order cannot be a ground to set aside the order". The high court had also sought a report from the junior judge who had used the AI-generated rulings. She told the court that this was her first time using an AI tool and she had believed the citations to be "genuine". She had no intention to misquote or misrepresent the rulings and that "the mistake occurred solely due to the reliance on an automatic source", the high court wrote. The high court also advocated for the "exercise of actual intelligence over artificial intelligence". Following this, the defendants appealed again, taking the matter to the Supreme Court, which was less forgiving about the impact of AI. Coming down sternly against the fake judgements, the top court last Friday stayed the lower court's order on the property dispute. It said the use of AI while making judgements was not simply "an error in decision making" but an act of "misconduct". "This case assumes considerable institutional concern, not because of the decision that was taken on the merits of the case, but about the process of adjudication and determination," the top court said. PS: To get an idea of how overloaded the Indian Judicial System is; this happened in a recent case in Allahabad High Court - The order then took an unusual turn. “Since I am feeling hungry, tired and physically incapacitated to dictate the judgment, the judgment is reserved,” the judge recorded. - He had been hearing more than 30 cases on that day - https://www.hindustantimes.com/india-news/hungry-tired-allahabad-hc-judge-reserves-order-after-3-hour-overtime-101772217165006.html https://www.hindustantimes.com/india-news/hungry-tired-allah...
- tamimio 7mo agoI wonder how many similar cases are happening in the engineering or software development sector that go unnoticed, and it seems no one is caring enough, only waiting for a disaster to happen so we can start seeing some regulation preventing the use of AI in engineering/coding industry.
- CagedJean 7mo ago[dead]
- deleted 7mo ago[deleted]
- thisisit 7mo agoIt's sad that it took the highest court in the country to point out lack of professionalism and misconduct. The judge took no personal responsibility. > She told the court that this was her first time using an AI tool and she had believed the citations to be "genuine". She had no intention to misquote or misrepresent the rulings and that "the mistake occurred solely due to the reliance on an automatic source", the high court wrote. She had one job. And that was to read the citations. Instead of owning up to the mistake of being lazy all she wanted to talk about "intentions". The high court also took no responsibility. > In its order, the high court said that "the citations may be non-existent, but if the learned trial court has considered the correct principles of law and its application to the facts of the case is also correct, This line of reasoning is questionable and attempt to gaslight everyone. Judges cite other cases in their judgement. But if the junior judge had no clue that the references were fake what correct principles was she applying? End of the day maybe the judgement is correct but this overall bullshit. Given that this is happening all over the world people seem to have a convenient excuse - The AI made me do it.
- boruto 7mo ago> The judge took no personal responsibility. Thats par for Indian judicial system. Simple civil cases run for several decades, the judges are all(yes everyone) is corrupt. Basically nothing works. It has been like this for several decades. LLM slop is least of its problems.
- eaglehead 7mo agoThis is going to be a huge problem in every sector. I have been exploring solution in this space for fintech and so far what Resemble AI is doing [1] is probably the best way to defend. The surface level for us is not just LLM generated text, it is also the combination of AI augmented audio (for incoming calls) and then for our own voice agents being able to protect and identify services cloning our own agent voices with watermarking. It's not fun, as we are constantly catching up. [1] https://www.resemble.ai/detect/ https://www.resemble.ai/detect/
- chrisjj 7mo ago> She had no intention to misquote or misrepresent the rulings and that "the mistake occurred solely due to the reliance on an automatic source" Next: gunman pleads death occured solely due to reliance on an automatic weapon.
- thisislife2 7mo agoThe high court also advocated for the "exercise of actual intelligence over artificial intelligence". Hehe.
- inder1 7mo agoThe failure mode here is predictable. Junior practitioners in any domain are being asked to use AI tools before they've developed the professional judgment to validate the outputs. You can't spot a hallucinated court order if you don't know what real court orders look like. The tool isn't the problem. The training pipeline that skips fundamentals is.
- yarn_ 7mo agoAll this for what?