222 of 960 predictions were confident errors.

SECTION 13: NOTABLE FAILURE CASES

13.1 NG_009 – Ethiopian Airlines v Polaris Bank (Nigeria)

Lesson: LLMs systematically underperform on procedural/limitation points – a critical risk for practitioners. This demonstrates a structural bias toward substantive merits reasoning over procedural analysis. Even when the correct legal outcome depends entirely on a limitation period, models default to analyzing the substantive merits of the dispute.

13.2 NG_020 – Atiba v Suberu (Nigeria)

Lesson: The parol evidence / document hierarchy problem is extremely common in Nigerian property and banking disputes. All six models defaulted to the wrong intuition – that a subsequent letter from the lender modified the formal mortgage deed. This tells practitioners something specific about where not to trust these tools without verification. The universal failure on this case is as significant as NG_009, revealing a systematic weakness in understanding document hierarchy and integration rules.

13.3 AU_013 – Ecosse Property Holdings (Australia)

Lesson: Purposive commercial construction is a systematic weakness; citation retrieval helps but is not universal. This case represents the only universal failure in the Australian dataset and demonstrates that models default to literal interpretation even when commercial purpose demands a different construction.

13.4 UK_004 – G4S v Lewis-Ranwell (UK)

Lesson: Illegality defence application to insanity verdicts is genuinely difficult – even with citation. This 2026 judgment tests training data frontiers and shows that even when citations are provided, models may struggle with novel or complex policy questions.

13.5 US_013 – Medical Marijuana v Horn (US)

Lesson: Most recent case (2025) tests training data frontier – significant recency effect. Even with citation, 4 of 6 models failed, demonstrating that models cannot be relied upon for very recent jurisprudence.

APPENDIX A: Supplementary Statistical Analyses Report (Optional)

1. Introduction and Rationale

The primary statistical validation (Part 2 of the main report) established key findings using non parametric tests (Friedman, McNemar, Mann Whitney U, Jonckheere Terpstra, Chi square, Cochran’s Q, coefficient of variation, and reliability diagrams). Those tests were appropriate for the experimental design and provided robust evidence for the headline conclusions.

However, to further strengthen the research and address potential peer review questions, three supplementary statistical evaluations were conducted:

• Mixed effects logistic regression – to quantify the simultaneous effect of Condition, Model, and Legal Domain on accuracy while controlling for case level variance (a main effects logit model was used as an approximation).
• Expected Calibration Error (ECE) – to provide a single numeric measure of confidence calibration for each model and condition, complementing the visual reliability diagrams.
• Domain specific Chi square tests – to test whether accuracy varies significantly across legal domains overall and per jurisdiction.
• Power analysis for non significant results – to determine if the sample size was adequate to detect meaningful effects for the non significant findings (e.g., Grok’s decline with clues, Claude’s marginal improvement).

These analyses are optional but add rigour, confirm the adequacy of the sample size, quantify calibration, and demonstrate the influence of legal domain. They are placed in this Supplementary Statistical Analyses section, which can be appended to Part 2 of the main report as Appendix A.

2. Methodology

All analyses were performed using the complete dataset of 960 verified experiments (20 cases × 4 jurisdictions × 6 models × 2 conditions). Data preparation followed the same protocol as the main study (sanitised facts, binary verdict scores, confidence scores scaled to [0,1]). The code was executed in a Google Colab environment using Python with libraries statsmodels, scikit-learn, pingouin, and numpy.

2.1 Logistic Regression (Main Effects)

A logistic regression model was fitted with the binary outcome correct (1 = correct verdict, 0 = incorrect). Fixed effects included:

• Condition_num (0 = No Clue, 1 = With Clue)
• Model (categorical, reference = ChatGPT)
• Legal_Domain (categorical, reference = not explicitly set; the output shows comparisons for many domain levels)

The model was estimated using maximum likelihood (method='bfgs') with 1000 iterations to ensure convergence. No random effects were included due to the complexity of crossed random effects in Python; the model serves as an approximation of the main fixed effects. Consequently, within case correlation is not accounted for, but the fixed effect coefficients are consistent with the non parametric tests.

2.2 Expected Calibration Error (ECE)

ECE quantifies the average absolute difference between predicted confidence and observed accuracy across bins. A lower ECE indicates better calibration.

For each model–condition pair, predicted confidences were divided into 10 equal width bins [0,0.1), [0.1,0.2), … , [0.9,1.0]. For each bin, the mean confidence and observed accuracy were calculated. ECE is the weighted average of the absolute differences:

"ECE"=∑_(b=1)^B▒n_b/N∣〖"acc" 〗_b-〖"conf" 〗_b∣

where n_b is the number of predictions in bin b, N total predictions, 〖"acc" 〗_b observed accuracy, and 〖"conf" 〗_b mean confidence. Empty bins were safely ignored.

2.3 Domain specific Chi square Tests

For each jurisdiction and for all jurisdictions combined, a contingency table of Legal_Domain vs. correct was constructed. A chi square test of independence was performed to determine whether the distribution of correct/incorrect answers differs significantly by legal domain. A significant result indicates that some legal domains are systematically easier or harder for LLMs.

2.4 Power Analysis for Non significant Results

Two non significant results from the main McNemar tests were examined:

• Grok’s decline with clues (−2.5%, p = 0.8238)
• Claude’s improvement (+11.25%, p = 0.0784)

For each, the effect size Cohen’s h was calculated:

h=2arcsin⁡(√(p_1 ))-2arcsin⁡(√(p_2 ))

where p_1 and p_2 are the proportions correct in the two conditions (No Clue vs With Clue). Using proportion_effectsize and tt_ind_solve_power (two sample independent proportions power) with n=80 per group, α=0.05, two tailed, the statistical power to detect the observed effect was estimated. This provides a check on whether non significance could be due to insufficient sample size.

Note: The independent samples approximation is conservative; the exact McNemar power depends on discordant pairs, but the directional conclusions remain valid.

3. Results

3.1 Logistic Regression (Main Effects)

The logistic regression converged successfully (Pseudo R squared = 0.2089, log likelihood ratio test p < 0.0001). The full coefficient table is presented below.

Table 1: Logistic Regression Results (Main Effects Only)

| Variable | Coefficient | Std. Error | z | P>|z| | 95% CI |

|----------|------------|------------|----|-------|--------|

| Intercept | 2.0927 | 1.081 | 1.935 | 0.053 | [-0.027, 4.212] |

| Condition_num | 0.6729 | 0.176 | 3.827 | 0.000 | [0.328, 1.018] |

| Model (ref: ChatGPT) | | | | | |

| Claude | 0.2678 | 0.299 | 0.895 | 0.371 | [-0.319, 0.854] |

| DeepSeek | -0.7591 | 0.279 | -2.719 | 0.007 | [-1.306, -0.212] |

| Gemini | 0.7973 | 0.322 | 2.477 | 0.013 | [0.167, 1.428] |

| Grok | 0.5159 | 0.309 | 1.672 | 0.095 | [-0.089, 1.121] |

| Perplexity | -0.0835 | 0.289 | -0.289 | 0.773 | [-0.650, 0.483] |

| Legal_Domain (selected significant) | | | | | |

| Commercial Law (Banking) | -4.2719 | 1.327 | -3.219 | 0.001 | [-6.873, -1.671] |

| Commercial Law (Ethics) | -3.3053 | 1.239 | -2.667 | 0.008 | [-5.734, -0.876] |

| Contract Law (Construction) | -3.3053 | 1.239 | -2.667 | 0.008 | [-5.734, -0.876] |

| Contract Law (Mortgage) | -2.9211 | 1.226 | -2.383 | 0.017 | [-5.324, -0.518] |

| Equity (Trusts) | -2.5565 | 1.222 | -2.092 | 0.036 | [-4.951, -0.161] |

| Civil Law (RICO) | -2.5565 | 1.222 | -2.092 | 0.036 | [-4.951, -0.161] |

| Intellectual Property (Trademark) | -2.0661 | 1.121 | -1.842 | 0.065 | [-4.264, 0.132] |

Note: Only a subset of legal domains with p < 0.10 are shown for brevity; the full table is available in the code output.

Key findings:

• The Condition_num coefficient (+0.6729, p < 0.001) confirms that including citations significantly increases the log odds of a correct verdict, consistent with the McNemar test.
• DeepSeek is significantly worse than ChatGPT (p = 0.007), while Gemini is significantly better (p = 0.013). Grok is marginally better (p = 0.095).
• Several legal domains (e.g., Commercial Law (Banking), Contract Law (Construction), Equity (Trusts)) show large negative coefficients, indicating that models perform worse on those domains relative to the reference category.

3.2 Expected Calibration Error (ECE)

Table 2 reports the ECE for each model and condition. Lower ECE indicates better alignment between confidence and observed accuracy.

ECE < 0.05 is excellent, 0.05–0.10 is good, > 0.20 indicates serious miscalibration.

Table 2: Expected Calibration Error by Model and Condition

Interpretation:

• Best calibrated: Claude (No Clue, ECE=0.0375) and Gemini (With Clue, ECE=0.0188) – their confidences reflect actual accuracy almost perfectly.
• Worst calibrated: DeepSeek in both conditions (ECE = 0.290 and 0.262) – it is dangerously overconfident, frequently expressing high confidence on incorrect predictions. This confirms the “Silent Failure” risk identified in the descriptive analysis.
• Grok has low ECE in both conditions, indicating good calibration despite its accuracy decline with clues.
• ChatGPT and Perplexity show moderate miscalibration, improving with clues.

3.3 Domain specific Chi square Tests

Table 3: Chi square Test for Independence of Legal Domain and Correctness

Findings:

• Accuracy varies significantly by legal domain overall and in Nigeria, the United States, and Australia.
• In the United Kingdom, domain was not a significant factor (p = 0.2803). This suggests that, uniquely among the four jurisdictions, UK legal domains are handled more uniformly by the models. This may reflect the high quality centralised reporting of UK case law (BAILII) and/or DeepSeek’s memorisation of UK cases, which reduces domain specific variation.

3.4 Power Analysis for Non significant Results

Table 4: Power to Detect Observed Effect Sizes (Independent proportions approximation)

Interpretation:

• Grok: The observed decline (Cohen’s h = 0.097) is negligible. The power to detect such a small effect with 80 cases is only 9.4%, meaning the non significant result is not due to low power but because the true effect is practically zero. The descriptive finding that Grok declines with clues is not statistically meaningful.
• Claude: The improvement (Cohen’s h = 0.280) is a small to medium effect. The power of 42% indicates that the current sample size had only a moderate chance of detecting this effect. Hence the marginal p value (0.0784) could be a false negative – the improvement may be real but requires a larger sample to confirm. This supports the descriptive observation that Claude’s improvement (+11.25%) is consistent across jurisdictions and may be genuine.
• Overall sample size (80 cases per condition) is adequate to detect a medium effect (Cohen’s h ≥ 0.3) with power > 0.8 (as shown in the output). Non significant results with smaller effects are therefore not due to insufficient data.

Note: These power calculations use an independent samples proportion test, a conservative approximation for the paired McNemar design. Actual power for McNemar would be slightly higher, but the directional conclusions remain valid.

4. Integration with Prior Findings

4.1 Consistency with Descriptive and Main Statistical Analysis

• Condition effect (+9.1% average improvement) is strongly confirmed by the logistic regression (p < 0.001) and the low ECE of Gemini with clues.
• DeepSeek’s poor performance is reinforced: logistic regression shows it is significantly worse than ChatGPT (p = 0.007), and its high ECE (>0.26) confirms overconfidence, aligning with the “Silent Failure” warning in the descriptive section.
• Gemini’s excellence with clues is supported by the lowest ECE (0.0188) and the significant positive coefficient (p = 0.013).
• Grok’s clue induced decline is shown to be negligible (h = 0.097) and not statistically meaningful; however, its calibration remains good (ECE = 0.0256 with clues). This clarifies that while Grok’s accuracy does not improve, it does not become dangerously miscalibrated.
• Domain variability was already observed descriptively (e.g., NG_020 and AU_013 domain failures). The chi square tests quantify this: accuracy differs by domain in three of four jurisdictions, and the UK is the exception.

4.2 Contributions of Supplementary Analyses

4.3 Implications for the Industrial Translation

• ECE reinforces the “Silent Failure” warning for DeepSeek: not only is its accuracy low, but its confidence is systematically misleading. The practitioner’s Duty of Inquiry Checklist must mandate independent verification of DeepSeek outputs regardless of confidence.
• Domain chi square identifies specific legal domains (e.g., Commercial Law (Banking), Contract Law (Construction), Equity (Trusts)) as high risk areas where extra caution is required.
• Power analysis provides evidence that the sample size (80 cases × 2 conditions) is sufficient for medium sized effects, strengthening the credibility of the null findings (e.g., Grok’s decline).

5. Placement in the Overall Report

These supplementary analyses are not required for the main conclusions but provide valuable depth and address potential methodological questions from reviewers. They should be placed as:

6. Conclusion

The supplementary statistical analyses confirm and extend the primary findings:

• Citation context significantly improves accuracy (logistic regression p < 0.001).
• DeepSeek is both low accuracy and severely overconfident (ECE > 0.26), validating the “Silent Failure” risk.
• Gemini and Claude are well calibrated, with Gemini achieving near perfect calibration with clues (ECE = 0.0188).
• Legal domain affects performance in Nigeria, US, and Australia, but not in the UK – a novel observation that may reflect training data characteristics.
• The sample size (80 cases) is sufficient to detect medium effects; Grok’s non significant decline is truly negligible, while Claude’s marginal improvement may merit further investigation.

These optional analyses add rigour, address potential critique, and provide deeper insight into model behaviour. They are now ready for inclusion in the final thesis or paper.

SILENT FAILURE ANALYSIS AND FINDINGS — CROSSLAW BENCHMARK

A Comprehensive Analysis of Confidently Wrong LLM Predictions Across Four Common Law Jurisdictions

Prepared for: HUMN4002 RD2 Viva Voce | Western Sydney University

Date: Autumn 2026

Analytical Framework: Silent Failure = EV < -0.5 (Incorrect Verdict + High Confidence)

Data Source: 960 verified interactions (80 cases × 6 models × 2 conditions)

EXECUTIVE SUMMARY

Silent Failure — confident wrongness without warning — represents a significant risk in legal AI deployment. A model that is wrong but expresses high confidence creates outputs that appear authoritative and persuasive to practitioners, making errors difficult to detect and correct.

Across 960 controlled interactions, we identified 222 Silent Failures, representing 23.1% of all predictions. Citation context reduced Silent Failures by 34.3% (134 → 88), confirming that prompt design materially improves model reliability.

Australia recorded the highest Silent Failure burden within the benchmark (74, 30.8%), followed by Nigeria (57, 23.8%), United States (54, 22.5%), and United Kingdom (37, 15.4%). DeepSeek accounted for the largest model-specific burden (59 Silent Failures), with its highest observed burdens in Australia (20), the United States (19), and Nigeria (17). Gemini demonstrated the strongest Silent Failure safety profile among the models tested, producing only 23 Silent Failures total and zero in the United States.

Brier Score confirms the EV findings: DeepSeek had the poorest calibration quality observed (Brier = 0.2978 With Clue) and the highest Silent Failure burden (59), while Gemini achieved the strongest With Clue reliability profile (Brier = 0.0552, Silent Failures = 23).

Critical cases in Nigeria and the United States produced EV = -0.95 to -1.0, demonstrating that models can be perfectly confident while being completely wrong — a pattern that would remain invisible under conventional accuracy based benchmarking.

1. DEFINITION AND METHODOLOGY

1.1 Silent Failure Definition

Accuracy alone does not capture the practical risks associated with deploying LLMs in legal contexts. A model may produce an incorrect answer while expressing very high confidence, creating a particularly dangerous form of error that can appear persuasive to practitioners.

Following the benchmark framework, a Silent Failure was defined as any prediction with an Expected Value (EV) below −0.5.

1.2 Expected Value Formula

The Expected Value metric combines accuracy and confidence into a single risk adjusted measure:

Interpretation:

1.3 Brier Score Definition

Brier Score measures the mean squared difference between predicted probabilities (Confidence/100) and actual outcomes (0 or 1). Lower Brier Scores indicate better calibration.

Note on Brier Score Interpretation: The thresholds used in this report (e.g., 0.0–0.1 as "excellent") are heuristic guidelines for comparative analysis within this benchmark rather than universally accepted calibration thresholds for legal AI systems. They are intended to facilitate model comparison rather than establish absolute standards.

1.4 Dataset Overview

2. OVERALL SILENT FAILURE BURDEN

Across the 960 benchmark interactions, 222 predictions met the Silent Failure threshold, representing 23.1% of all model outputs.

This finding indicates that almost one in four legal AI predictions within the benchmark were confidently incorrect enough to satisfy the Silent Failure criterion. The result demonstrates that accuracy alone substantially understates deployment risk.

3. SILENT FAILURE BURDEN ACROSS JURISDICTIONS

Table 1. Silent Failures by Jurisdiction

Key Observations

• Australia recorded the highest Silent Failure rate within the benchmark (30.8%) — almost one in three predictions were confidently wrong. This suggests that the evaluated models were less reliable on Australian appellate law tasks than on the other jurisdictions examined.
• Nigeria recorded the second highest rate (23.8%) — nearly one in four predictions were Silent Failures. This confirms that low resource jurisdictions face compounding risks.
• The United Kingdom recorded the lowest Silent Failure rate (15.4%) — reflecting better training data coverage and model calibration for UK law.
• The UK was the only jurisdiction below 20% — suggesting that high resource, well digitised legal systems benefit from more reliable AI calibration.

4. SILENT FAILURE DISTRIBUTION BY MODEL

Table 2. Silent Failure Counts by Model

Key Observations

• DeepSeek accounted for 26.6% of all Silent Failures (59 of 222) — DeepSeek was the highest risk model in terms of calibration failure within this benchmark.
• Gemini had the lowest observed Silent Failure burden (23) — Gemini was the lowest risk model among those tested.
• Grok had 27 Silent Failures despite high accuracy (83.8%) — Grok was accurate when correct but showed overconfidence when wrong.
• Perplexity's high Silent Failure count (41) — retrieval first models may be more prone to overconfidence when retrieval is incorrect.

5. SILENT FAILURE HEATMAP (MODEL × JURISDICTION)

Table 3. Silent Failure Counts by Model and Jurisdiction

Key Observations

6. IMPACT OF CITATION CONTEXT ON SILENT FAILURE RISK

One of the most important findings concerns the effect of citation context.

Table 4. Silent Failures by Condition and Jurisdiction

Key Observations

• Citation context reduced Silent Failures by 34.3% overall (134 → 88). This is one of the strongest findings: prompt design materially improved model reliability.
• Nigeria showed the largest absolute reduction (39 → 18, -53.8%) — citation clues most helped Nigerian legal reasoning, where models otherwise struggled.
• The United Kingdom showed strong reduction (23 → 14, -39.1%) — citation context significantly improved UK law calibration.
• The United States showed the smallest reduction (29 → 25, -13.8%) — clues did not substantially reduce Silent Failures in US law, possibly because models were already overconfident in US legal contexts.
• Australia showed moderate improvement (43 → 31, -27.9%) — clues helped but did not fully resolve Australian calibration issues.

7. SILENT FAILURE BY MODEL × CONDITION

Table 5. Model Specific Silent Failures by Condition

Key Observations

• Every model except Grok improved with citation context — Grok showed a slight increase in Silent Failures with clues (+1), consistent with its accuracy decline.
• Gemini showed the largest improvement (-13, 72.2%) — citation context dramatically improved Gemini's calibration.
• DeepSeek showed the smallest improvement (-5, 15.6%) — DeepSeek remained dangerously overconfident even with clues.
• Gemini had the lowest Silent Failure count in both conditions (18 No Clue, 5 With Clue) — lowest risk model overall.

8. BRIER SCORE CALIBRATION FINDINGS

Brier Score measures the mean squared difference between predicted probabilities (Confidence/100) and actual outcomes (0 or 1). Lower Brier Scores indicate better calibration.

Table 6. Average Brier Score by Model and Condition

Key Observations

• Gemini achieved the best With Clue calibration observed (Brier = 0.0552) — exceptionally well calibrated when citation context was provided.
• Grok achieved the best No Clue calibration observed (Brier = 0.1328) — well calibrated without clues, consistent with its stable performance.
• DeepSeek had the poorest calibration observed in both conditions — Brier > 0.29 in both No Clue and With Clue, confirming dangerous overconfidence.
• All models except Grok showed improved calibration with citation context — Grok's Brier slightly worsened (0.1328 → 0.1447), consistent with its accuracy decline.

9. RELATIONSHIP BETWEEN BRIER SCORE, EXPECTED VALUE, AND SILENT FAILURE RISK

Although Brier Score and Silent Failure measure different aspects of model behaviour, the two metrics were broadly aligned in this benchmark. A clear association was observed between poorer calibration (higher Brier Scores) and higher Silent Failure counts. Models with poorer calibration generally produced more Silent Failures, suggesting that calibration quality appears to be a useful indicator of deployment risk within this benchmark.

Table 7. Brier Score vs Silent Failures by Model

Key Observations

• A clear association is observed between poorer calibration (higher Brier Scores) and higher Silent Failure counts.
• DeepSeek showed the poorest calibration observed (Brier = 0.2978) and the highest Silent Failure burden (59).
• Gemini showed the best calibration observed (Brier = 0.0552) and the lowest Silent Failure burden (23).
• Grok showed good No Clue calibration but slightly worsened with clues, consistent with its unique decline pattern.
• Brier Score confirms the EV findings — the same models that were overconfident (high Brier) were also the highest risk in terms of Silent Failure burden.

10. CRITICAL SILENT FAILURE CASES

Several cases demonstrated extreme Silent Failure behaviour.

Table 8. Example Silent Failure Cases (EV < -0.8)

In each case, the model produced an incorrect legal prediction while maintaining near maximal or perfect confidence. These outputs would likely appear persuasive to practitioners despite being wrong.

Key Observations:

• DeepSeek in Nigeria produced EV = -0.95 on multiple cases — near maximal confidence on incorrect predictions.
• DeepSeek in the United States produced EV = -1.00 on two cases — 100% confidence on wrong answers.
• DeepSeek in the United Kingdom produced EV = -1.00 on one With_Clue case — citation context did not prevent the Silent Failure.

Such cases illustrate why Silent Failure analysis provides information that traditional accuracy metrics cannot capture.

11. VISUALISATION READY SUMMARY

Silent Failure Risk by Jurisdiction

text

Australia ████████████████████████████████████████████ 30.8% (74/240)

Nigeria ██████████████████████████████████████████ 23.8% (57/240)

United States ████████████████████████████████████████ 22.5% (54/240)

United Kingdom █████████████████████████████████████████ 15.4% (37/240)

Silent Failure Burden by Model

text

DeepSeek ████████████████████████████████████████████ 59

Perplexity ████████████████████████████████████████████ 41

ChatGPT ████████████████████████████████████████████ 39

Claude ████████████████████████████████████████████ 33

Grok ████████████████████████████████████████████ 27

Gemini ████████████████████████████████████████████ 23

Condition Effect

text

No Clue ████████████████████████████████████████████████████████████████████████████████████ 134

With Clue ████████████████████████████████████████████████████████████████████████████████████ 88

↓ 34.3% reduction (46 fewer Silent Failures)

Brier Score by Model (With Clue)

text

DeepSeek ████████████████████████████████████████████ 0.2978

Grok ████████████████████████████████████████████ 0.1447

Perplexity ████████████████████████████████████████████ 0.1642

ChatGPT ████████████████████████████████████████████ 0.1559

Claude ████████████████████████████████████████████ 0.1375

Gemini ████████████████████████████████████████████ 0.0552

12. PRACTITIONER IMPLICATIONS

12.1 Model Selection Guidance

12.2 Prompt Design Guidance

12.3 Brier Score and Calibration Guidance

12.4 DeepSeek Risk Warning

⚠️ Exercise caution when deploying DeepSeek in Australian, Nigerian, and United States legal contexts and require independent verification of outputs.

Evidence:

13. THEORETICAL CONTRIBUTIONS

13.1 Silent Failure Quantification

This study provides a systematic quantification of Silent Failures across multiple common law jurisdictions. Prior work identified Silent Failures conceptually (Pathak et al., 2025; Potts & Sudhof, 2026), but did not measure them across jurisdictions, models, and prompt conditions in a controlled benchmark.

13.2 Jurisdictional Risk Mapping

The finding that Australia recorded the highest Silent Failure rate (30.8%) while the UK recorded the lowest (15.4%) confirms that Silent Failure risk is jurisdiction dependent within this benchmark. This challenges the assumption that Silent Failure is a uniform model property.

13.3 Prompt Design as Risk Mitigation

The finding that citation context reduces Silent Failures by 34.3% confirms that prompt design materially improves calibration. This is one of the strongest practical findings: simple changes to input structure reduce the most dangerous failure mode.

13.4 Model Specific Risk Profiles

The identification of DeepSeek as the highest Silent Failure model and Gemini as the lowest provides a model specific Silent Failure risk taxonomy for legal AI within this benchmark framework.

13.5 Brier Score as an Indicator of Silent Failure Risk

The clear association observed between Brier Score and Silent Failure burden suggests that calibration quality may be a useful indicator of deployment safety. Models with poorer calibration (high Brier) consistently produced more Silent Failures within this benchmark.

13.6 Extreme Cases as Evidence of Overconfidence Risk

The existence of predictions with EV = -1.0 (100% confidence on incorrect verdicts) is consistent with concerns raised in the literature regarding overconfident language model outputs (Bender et al., 2021). These cases demonstrate that perfect confidence does not imply correct reasoning.

14. SUMMARY OF KEY FINDINGS

15. CONCLUSION

Silent Failure — confident wrongness without warning — represents a significant risk in legal AI deployment. Across 960 interactions, 222 Silent Failures occurred, representing 23.1% of all predictions. The risk varied by jurisdiction: Australia recorded the highest Silent Failure rate (30.8%), followed by Nigeria (23.8%), the United States (22.5%), and the United Kingdom (15.4%).

DeepSeek accounted for 59 Silent Failures (26.6% of the total), with its highest observed burdens in Australia (20), the United States (19), and Nigeria (17). Gemini showed the lowest observed risk, with only 23 Silent Failures and no Silent Failures observed in the United States.

Critically, citation context reduced Silent Failures by 34.3% overall (134 → 88). Nigeria showed the largest improvement (-53.8%), confirming that simple prompt design changes materially improve model reliability, particularly in low resource jurisdictions.

Brier Score confirms the EV findings: DeepSeek had the poorest calibration quality observed (Brier = 0.2978 With Clue) and the highest Silent Failure burden (59), while Gemini achieved the strongest With Clue reliability profile (Brier = 0.0552, Silent Failures = 23).

Extreme cases in Nigeria and the United States produced EV = -0.95 to -1.0, demonstrating that models can be perfectly confident while being completely wrong. This pattern is invisible under conventional accuracy based benchmarking.

For practitioners: always include citation context, exercise caution with DeepSeek outputs, and treat Australian law as the jurisdiction with the highest observed Silent Failure risk. The evidence is clear: Silent Failures are measurable, jurisdiction dependent, and reducible through prompt design. This is the empirical foundation for evidence based legal AI deployment.

16. RECOMMENDATIONS

Report End

QUICK REFERENCE CARD