Reasoning-quality conclusions await lawyer verification.

SECTION 12: THE "CORRECT GUESSING" PHENOMENON

12.1 Preliminary Evidence of Stochastic Parroting

12.2 Reasoning Classification Framework

In Phase 2 of this research, all model reasoning will be classified according to the following framework:

12.3 Type II Logic Failure Definition

A Type II Logic Failure occurs when a model predicts the correct verdict (Score = 1) but its reasoning falls into the General-but-Correct category – it uses general common law principles rather than the jurisdiction-specific principle that actually decided the case. This is the signature of "stochastic parroting" – correct guessing without genuine understanding.

Preliminary Evidence: Based on initial analysis of NG_001, Type II Logic Failure may have occurred in 5 of 6 models (83%) for that case alone. Models predicted correctly using general negligence principles rather than the jurisdiction-specific "statutory flavour" principle that actually decided the case.

Significance (if confirmed): This would provide the first empirical support for Dahl et al.'s "stochastic parrot" hypothesis with real legal data – a major theoretical contribution demonstrating that LLMs may achieve correct outcomes through probabilistic guessing rather than genuine jurisdictional understanding. Full confirmation awaits Phase 2 reasoning analysis.

12.4 Implications for Legal Practice

• Correct verdict ≠ correct reasoning – practitioners cannot assume that accurate predictions reflect sound legal analysis. A model may be "correct" but for the wrong reasons.
• Verification of reasoning is essential – the Duty of Inquiry Checklist must include verification of the legal principles applied, not just the outcome.
• Citation context helps but does not guarantee correct reasoning – even with clues, models may retrieve correct outcomes without understanding the underlying ratio decidendi.

PART 3: Phase 2 — Reasoning Analysis

Key Reason Verification (Step 1) - Researcher Analysis

Verdict & Reasoning Performance

(1) Nigeria With Clue Experiments

The following table presents the ground truth ratio decidendi for each of the 20 Nigerian cases, alongside the grade (P / F / N/A) and a concise justification for each of the six evaluated LLMs (ChatGPT, Gemini, Claude, Grok, DeepSeek, Perplexity).

• P = Correct verdict and correct reasoning (aligned with the ratio).
• F = Correct verdict but incorrect reasoning (Type II logic failure).
• N/A = Incorrect verdict (reasoning evaluation is moot).

reasoning; F = Correct verdict but wrong reasoning (Type II failure); N/A = Incorrect verdict.

Summary Table: Nigeria With Clue Experiments – Verdict & Reasoning Performance

Notes:

• Correct Verdict = number of cases where the model’s predicted outcome matched the ground truth (P + F).
• Correct Reasoning = number of cases where the model’s legal reasoning aligned with the ratio decidendi (Grade P only).
• Type II Failure = correct verdict but wrong reasoning (Grade F).
• Incorrect Verdict = verdict did not match ground truth (Grade N/A).
• Reasoning Accuracy (of correct verdicts) = Correct Reasoning / Correct Verdict × 100.
• Overall Reasoning Accuracy = Correct Reasoning / 20 × 100.

Key Findings

• High Verdict Accuracy but Low Reasoning Fidelity

All models achieved high verdict correctness (ChatGPT 100%, Gemini 95%, Claude 85%, Grok 80%, Perplexity 80%, DeepSeek 65%). However, reasoning alignment with the true ratio decidendi was low: ChatGPT, Gemini, and Claude each had only 10 out of 20 cases with correct reasoning (50%), while Grok and Perplexity scored 40%, and DeepSeek 30%.

• Type II Logic Failures are Pervasive

Type II failures (correct verdict but wrong legal reasoning) occurred in 7–10 cases per model. This is the most dangerous error for legal AI because a correct outcome masks flawed logic, undermining trust and explainability.

• Strict Liability and Procedural Cases Caused Most Failures

Cases involving stay of execution (NG_003), procedural waiver (NG_004), equitable damages quantum (NG_007), presumption of negligence in bailment (NG_015), criminal burden “one of four ingredients” (NG_016), publication in defamation (NG_017), court integrity over jurisdiction (NG_018), and interest as of right for fiduciary breach (NG_019) were consistently mis reasoned by almost all models.

• DeepSeek’s Lower Verdict Accuracy Reflects Conservative Reasoning

DeepSeek had the fewest correct verdicts (13/20) but also exhibited different failure patterns. It correctly applied estoppel (NG_013) and vicarious liability (NG_002, NG_014) but struggled with limitation bars (NG_009) and admiralty jurisdiction (NG_011).

• No Model Achieved “Human Level” Reasoning

Even the best model (ChatGPT or Claude) only had correct reasoning in half of the cases, highlighting a substantial gap for research in legal reasoning evaluation.

Visualisation Image:

Figure: Reasoning & Verdict Performance - (1.) Nigeria With Clue

Bar chart description:

• Green = Correct Reasoning (P)
• Orange = Type II Failure (correct verdict, wrong reasoning)
• Red = Incorrect Verdict (N/A)
• Number on top of each bar = total correct verdicts (P+F).

Conclusion

The Nigeria With Clue experiments demonstrate that while large language models can often predict the correct legal outcome (verdict accuracy up to 100%), their legal reasoning – as measured against the true ratio decidendi – is alarmingly deficient. Half of all correct verdicts were supported by incorrect or irrelevant legal principles (Type II failures). This finding has critical implications for deploying LLMs in legal practice:

• Verdict alone is not a reliable metric for legal AI capability.
• Explainability and reasoning verification must be core components of any legal LLM system.
• Specific legal domains (procedural law, equitable remedies, strict liability nuances, defamation elements) are particularly challenging and require targeted fine tuning or retrieval augmented grounding.

Future work should focus on reasoning aware evaluation frameworks, adversarial testing of legal logic, and hybrid systems that combine LLM output with rule based verification against the ratio decidendi. The current results underscore that even the best models are not yet ready for unsupervised legal decision support.

APPENDICES: