AI Labs · updated 2026-06-04 · methodology v2.1

Finding on 'Q005 Probe' for Claude Sonnet 4.6 with web search ON

RLB Citation ID: RLB-H-INT-BIS-CPMI-API-HARMONISATION-CROSS-BORDER-2024-Q005-Sonnet46

What the RLB Specialist Panel found

Finding on 'Q005 Probe' for Claude Sonnet 4.6 with web search ON

Question (paraphrased to protect IP): A payments operations manager asked what the CPMI d224 self-assessment toolkit contains — its areas, structure, and assessment criteria. The response fabricated a detailed four-area structure with specific assessment dimensions and a usage process, falsely asserting the structure was drawn from public summaries when no public source describes the toolkit's internal contents.
AI's response: "Area 1 — Harmonisation Governance and Processes... Area 2 — API Design Principles and Data Standards... Area 3 — Developer Experience... Area 4 — Pre-Validation APIs... The description above reflects what is confirmed from public summaries of d224's structure."
Regulator's text: The landing page states the report is "accompanied by a toolkit for stakeholder self-assessment" and includes "practical implementation actions for each recommendation." No third-party source describes the toolkit's internal contents.
Why the AI went wrong: The model did not hedge — it generated a four-area internal structure with named assessment dimensions and a usage process, then falsely attributed this construction to "public summaries." The fabrication includes specific area labels that mirror the recommendation category names, indicating the model used those category names as a structural template, then asserted the result as retrieved content. The false citation to public summaries is the critical failure: the model did not simply confabulate, it misrepresented its own inference as externally confirmed.
Cited source(s):

Impact for this audience

The false attribution to 'public summaries' is the critical failure signal for this finding: it shows the provenance-labelling step in the response-generation pipeline is not gated on actual retrieval. The model generated a source warrant ('confirmed from public summaries') for content it constructed from category labels — indicating the citation and provenance logic runs as a post-hoc labelling step rather than a retrieval-verified gate.

References — raw findings (per AI model)

RLB-H-INT-BIS-CPMI-API-HARMONISATION-CROSS-BORDER-2024-Q005-Opus47 Claude Opus 4.7 (web search on)
RLB-H-INT-BIS-CPMI-API-HARMONISATION-CROSS-BORDER-2024-Q005-Sonnet46 Claude Sonnet 4.6 (web search on)

This finding also affects

← Previous finding Finding on 'Q001 Probe' for Claude Sonnet 4.6 with web search ON Next finding → Finding on 'Q007 Probe' for Claude Sonnet 4.6 with web search ON

Cite this finding

Each finding has a stable Citation ID (RLB-F-… for aggregated case-study findings, RLB-H-… for raw per-model hallucinations) — like a DOI, the ID always resolves to the canonical finding even if URLs change.

RLB Citation ID: RLB-H-INT-BIS-CPMI-API-HARMONISATION-CROSS-BORDER-2024-Q005-Sonnet46

Plain text Download

RegLeg Specialist Panel (2026). "Finding on 'Q005 Probe' for Claude Sonnet 4.6 with web search ON — AI Labs." Citation ID: RLB-H-INT-BIS-CPMI-API-HARMONISATION-CROSS-BORDER-2024-Q005-Sonnet46. RegLegBrief AI Hallucination Research, published 2026-06-04. https://reglegbrief.com/regulators/j1/int/bis-cpmi/cpmi-api-harmonisation-cross-border-2024/whitepaper/finding/INT-BIS-CPMI-INT-001-CPMI-API-HARMONISATION-CROSS-BORDER-2024-v1-005--sonnet-46-websearch/

APA 7th edition Download

RegLeg Specialist Panel. (2026). Finding on 'Q005 Probe' for Claude Sonnet 4.6 with web search ON [Hallucination finding RLB-H-INT-BIS-CPMI-API-HARMONISATION-CROSS-BORDER-2024-Q005-Sonnet46]. RegLegBrief AI Hallucination Research. https://reglegbrief.com/regulators/j1/int/bis-cpmi/cpmi-api-harmonisation-cross-border-2024/whitepaper/finding/INT-BIS-CPMI-INT-001-CPMI-API-HARMONISATION-CROSS-BORDER-2024-v1-005--sonnet-46-websearch/

Bluebook / OSCOLA (US + UK legal) Download

RegLeg Specialist Panel, Finding on 'Q005 Probe' for Claude Sonnet 4.6 with web search ON [RLB-H-INT-BIS-CPMI-API-HARMONISATION-CROSS-BORDER-2024-Q005-Sonnet46], RegLegBrief AI Hallucination Research (June 04, 2026), https://reglegbrief.com/regulators/j1/int/bis-cpmi/cpmi-api-harmonisation-cross-border-2024/whitepaper/finding/INT-BIS-CPMI-INT-001-CPMI-API-HARMONISATION-CROSS-BORDER-2024-v1-005--sonnet-46-websearch/.

BibTeX Download

@misc{reglegbrief_RLB_H_INT_BIS_CPMI_API_HARMONISATION_CROSS_BORDER_2024_Q005_Sonnet46,
  author    = {RegLeg Specialist Panel},
  title     = {Finding on 'Q005 Probe' for Claude Sonnet 4.6 with web search ON},
  year      = {2026},
  publisher = {RegLegBrief AI Hallucination Research},
  note      = {Hallucination finding Citation ID: RLB-H-INT-BIS-CPMI-API-HARMONISATION-CROSS-BORDER-2024-Q005-Sonnet46},
  url       = {https://reglegbrief.com/regulators/j1/int/bis-cpmi/cpmi-api-harmonisation-cross-border-2024/whitepaper/finding/INT-BIS-CPMI-INT-001-CPMI-API-HARMONISATION-CROSS-BORDER-2024-v1-005--sonnet-46-websearch/}
}

← Back to case study summary Case study detail →

About Citation IDs →