AI Hallucination Research › Regulators › Global standard-setters › INT › BIS-CPMI › CPMI-IOSCO-PFMI-L3-GENERAL-BUSINESS-RISK-2025 › White paper › Detail › Finding
AI Labs · updated 2026-06-03 · methodology v2.1

Finding on 'Q005 Probe' for Claude Sonnet 4.6 with web search ON

RLB Citation ID: RLB-H-INT-BIS-CPMI-IOSCO-PFMI-L3-GENERAL-BUSINESS-RISK-2025-Q005-Sonnet46
What the RLB Specialist Panel found

Finding on 'Q005 Probe' for Claude Sonnet 4.6 with web search ON

  • Question (paraphrased to protect IP): A trade repository is preparing an internal regulatory engagement summary. Draft a 1-page methodology note on the November 2025 CPMI-IOSCO Level 3 assessment on general business risk: when the assessment process began, when data was collected from FMIs, when the assessment formally concluded, how many FMIs responded, on what basis they participated, and how findings were validated with FMIs before publication.
  • AI's response: "The Level 3 assessment of FMI implementation of PFMI Principle 15 (general business risk) was carried out during the period 2023 to 2024."
  • Regulator's text: It focuses on general business risk and was carried out during 2023–25 by the Implementation Monitoring Standing Group (IMSG)... there were further rounds of engagement and follow up questions with the participating FMIs between November 2023 and April 2025.
  • Why the AI went wrong: The model truncated the assessment timeline at 2024, dropping the 2025 engagement rounds that are explicitly documented in the published text and that include the follow-up validation process with participating FMIs. The error is not an inference about an unstated date — the regulator's published document specifies the 2025 endpoint and the nature of the engagement in that period. The truncation is consistent with a retrieval boundary in late 2024 or early 2025 that caused the model to reproduce the portion of the timeline present in its indexed content while silently omitting the more recent period, without signalling any uncertainty about the completeness of its response.
  • Cited source(s):
Impact for this audience

This failure implicates the retrieval pipeline's indexing boundary for BIS-IOSCO assessment publications: the model reproduced a 2023–2024 window for the assessment process while the published document specifies a 2023–2025 window with explicit April 2025 follow-up engagement dates. The subsystem gap is indexed-content completeness for Q4 2025 BIS publications — the model returned the portion of the timeline available in its indexed content without uncertainty-flagging that the more recent period might be missing from its view.

References — raw findings (per AI model)
This finding also affects
← Previous finding Finding on 'Q003 Probe' for Claude Sonnet 4.6 with web search ON
Cite this finding

Each finding has a stable Citation ID (RLB-F-… for aggregated case-study findings, RLB-H-… for raw per-model hallucinations) — like a DOI, the ID always resolves to the canonical finding even if URLs change.

RLB Citation ID: RLB-H-INT-BIS-CPMI-IOSCO-PFMI-L3-GENERAL-BUSINESS-RISK-2025-Q005-Sonnet46
Plain text Download 
RegLeg Specialist Panel (2026). "Finding on 'Q005 Probe' for Claude Sonnet 4.6 with web search ON — AI Labs." Citation ID: RLB-H-INT-BIS-CPMI-IOSCO-PFMI-L3-GENERAL-BUSINESS-RISK-2025-Q005-Sonnet46. RegLegBrief AI Hallucination Research, published 2026-06-03. https://reglegbrief.com/regulators/j1/int/bis-cpmi/cpmi-iosco-pfmi-l3-general-business-risk-2025/whitepaper/finding/INT-BIS-CPMI-INT-001-CPMI-IOSCO-PFMI-L3-GENERAL-BUSINESS-RISK-2025-v1-005--sonnet-46-websearch/
APA 7th edition Download 
RegLeg Specialist Panel. (2026). Finding on 'Q005 Probe' for Claude Sonnet 4.6 with web search ON [Hallucination finding RLB-H-INT-BIS-CPMI-IOSCO-PFMI-L3-GENERAL-BUSINESS-RISK-2025-Q005-Sonnet46]. RegLegBrief AI Hallucination Research. https://reglegbrief.com/regulators/j1/int/bis-cpmi/cpmi-iosco-pfmi-l3-general-business-risk-2025/whitepaper/finding/INT-BIS-CPMI-INT-001-CPMI-IOSCO-PFMI-L3-GENERAL-BUSINESS-RISK-2025-v1-005--sonnet-46-websearch/
Bluebook / OSCOLA (US + UK legal) Download 
RegLeg Specialist Panel, Finding on 'Q005 Probe' for Claude Sonnet 4.6 with web search ON [RLB-H-INT-BIS-CPMI-IOSCO-PFMI-L3-GENERAL-BUSINESS-RISK-2025-Q005-Sonnet46], RegLegBrief AI Hallucination Research (June 03, 2026), https://reglegbrief.com/regulators/j1/int/bis-cpmi/cpmi-iosco-pfmi-l3-general-business-risk-2025/whitepaper/finding/INT-BIS-CPMI-INT-001-CPMI-IOSCO-PFMI-L3-GENERAL-BUSINESS-RISK-2025-v1-005--sonnet-46-websearch/.
BibTeX Download 
@misc{reglegbrief_RLB_H_INT_BIS_CPMI_IOSCO_PFMI_L3_GENERAL_BUSINESS_RISK_2025_Q005_Sonnet46,
  author    = {RegLeg Specialist Panel},
  title     = {Finding on 'Q005 Probe' for Claude Sonnet 4.6 with web search ON},
  year      = {2026},
  publisher = {RegLegBrief AI Hallucination Research},
  note      = {Hallucination finding Citation ID: RLB-H-INT-BIS-CPMI-IOSCO-PFMI-L3-GENERAL-BUSINESS-RISK-2025-Q005-Sonnet46},
  url       = {https://reglegbrief.com/regulators/j1/int/bis-cpmi/cpmi-iosco-pfmi-l3-general-business-risk-2025/whitepaper/finding/INT-BIS-CPMI-INT-001-CPMI-IOSCO-PFMI-L3-GENERAL-BUSINESS-RISK-2025-v1-005--sonnet-46-websearch/}
}
← Back to case study summary Case study detail →
About Citation IDs →