Pakistan's AI Judge Assistant Cleared 38 Extra Cases a Month Per Judge. Here's What Actually Worked.

Key points

• A trial across 1,559 Pakistani judges found a custom AI legal tool raised cases resolved by 6.3% over nine months in 2024.
• Pakistan's courts carry a backlog of 2.26 million cases and have fewer than two judges per 100,000 people, compared to 22 in the EU.
• Trained judges completed roughly 38.5 more cases per month, saving an estimated $38.50 in judicial costs for every $1 spent running the tool.
• Judges who received six training sessions logged into the tool 56 times on average and sent 212 prompts, versus 10 logins for those who got only generic training.
• About one in five prompts involved judges asking the AI to make decisions or write opinions with little human input, raising ongoing quality concerns.

Pakistan's courts are drowning. More than 2.26 million cases sit in a backlog, and the country has fewer than two judges for every 100,000 people. The EU manages 22 per 100,000; Brazil runs 8. Something had to give.

So economist Sultan Mehmood of the New Economic School in Moscow, together with collaborators including Elliott Ash of ETH Zurich, built a tool called JudgeGPT and ran what may be the first large-scale independent test of AI inside a working court system. Their findings, first reported by IEEE Spectrum AI, land at an awkward moment: judges in several countries have already been caught quietly using commercial AI chatbots in ways courts never approved.

What is JudgeGPT and how does it work?

JudgeGPT is a custom AI assistant built specifically for Pakistani courts. It runs on OpenAI's GPT-4, the large language model (an AI system trained on vast amounts of text to generate and analyse language) that also powers many commercial chatbots, drawing from a database of 128,292 Pakistani judicial opinions and 943 statutes.

Accuracy depended on a technique called retrieval-augmented generation, or RAG: the AI searches that private database before answering, then shows its sources in footnotes rather than guessing at case law. Ash is direct about why this matters: "It turns out that actually the way to fix hallucinations isn't just more intelligent models. It's to attach the models to a tool that can do a search and verify the sources." Hallucinations, in AI terms, means the model confidently inventing facts that do not exist. Commercial chatbots tested earlier had failed badly on Pakistani legal queries, regularly fabricating case citations.

Our 17 July story on how law firms are already shaping similar specialised models showed this pattern emerging elsewhere; JudgeGPT is among the first to reach a formal government trial at scale.

Did it actually make judges better or just faster?

Faster, measurably. Better, probably, though with caveats.

By the time 487 judges had completed the training programme, the median district recorded a 6.3% jump in resolved cases. Appeal rates fell slightly, a sign that speed was not obviously coming at the cost of accuracy. Each trained judge resolved around 38.5 more cases per month than the baseline.

Metric	Result
Judges in trial	1,559
Cases in backlog	2.26 million
Productivity increase	6.3%
Extra cases per trained judge per month	38.5
Cost saving per $1 spent	$38.50
Post-training logins per judge	56
Post-training prompts per judge	212

To assess quality, the team used OpenAI's GPT-5-mini (a lighter, faster version of the latest GPT model) to compare pairs of judgements from the same judge before and after training. The AI picked the post-training judgement 59% of the time. Two experienced Pakistani lawyers agreed with those picks 70.6% of the time and with each other 73% of the time, suggesting the AI quality score is a reasonable but imperfect proxy.

David Autor, an economics professor at MIT, called the result credible. "The 6.3% productivity boost is not overwhelming," he said, "but it's credible and likely to improve as the tool is more widely used."

What is the honest concern here?

Training mattered more than the software itself. Judges who received six 90-minute sessions covering how the AI works, its limitations and the risk of bias used the tool consistently for the full nine months. Those who got only generic technology training logged in a handful of times. Judges with no training tried it for about a month, then stopped entirely.

The bigger worry is delegation. Roughly one in five prompts asked JudgeGPT what the right decision should be, or instructed it to write a legal opinion with minimal judge input. That crosses a line most legal systems draw firmly. Training reduced that proportion, but it did not eliminate it.

John Zeleznikow, a professor of law and technology at La Trobe University in Australia, frames the gap plainly: "What's not that clear is whether what you call the quality of justice is better."

For the judge who spoke to the researchers anonymously, the daily reality is simpler. They have carried more than 1,000 active cases for over a decade. "For research, it's just one prompt away," they said, "whereas before I had to search for the precedents and laws for hours."

If your workplace is piloting an AI tool, push hard for structured training, not just a login and a password. Access without education, this study shows, produces a month of curiosity and then abandonment. Six sessions changed everything: that is the number worth taking to your next budget meeting.