A Cross-Jurisdiction Framework for Evaluating Large Language Models in Common-Law Systems
Ebere Josephine Uba · Western Sydney University
Supervised by A/Prof Dongmo Zhang and Dr Jolin Qu
A Cross-Jurisdiction Framework for Evaluating Large Language Models in Common-Law Systems
Ebere Josephine Uba · Western Sydney University
Supervised by A/Prof Dongmo Zhang and Dr Jolin Qu
Legal AI failures include Mata v. Avianca, Mavundla, and Handa & Mallick.
Low- and mid-resource common-law jurisdictions are underrepresented in evaluations.
Part I: variation across models and jurisdictions; the effect of citation context.
Part II: nine questions about Nigerian legal materials, presentation, summaries and consistency.
80 final appellate cases · 4 jurisdictions · 6 commercial models.
No Clue and With Clue conditions · 960 primary predictions.
Pooled accuracy: 346/480 (72.1%) → 390/480 (81.3%).
No single model led in every setting; Grok was the exception to the citation gain.
222/960 primary predictions (23.1%) were Silent Failures.
Australia had the highest rate (30.8%); no simple resource ordering emerged.
Nigeria had the largest pooled citation gain: +16.7 points.
Two of three universally failed No Clue cases were Nigerian.
Supply materials → preload and rerun → vary summary length → vary summarisation tool.
The same 20 Nigerian cases remained the focus.
Preloaded model-constructed summaries: 64.2%–67.5%; fuller materials: 74.6%.
With a fixed predictor, tool results ranged 75.0%–90.0%.
Identical six-model reruns agreed on about three verdicts in four.
Nigerian legal practitioners: compare tested conditions on local cases, not only headline rankings.
Australian practitioners: a confident or cited answer still needs source verification.
Researchers and builders: test local variation, calibration and repeatability. These are possible uses, not measured practice outcomes.
Phase 4: lawyer-led reasoning verification is in progress.
Phase 5: practitioner resources are planned.
Stated confidence was not a reliable indicator of correctness.
The findings support further controlled study, not claims of legal competence.
Explore the findings and their source references.