AI Labs · updated 2026-06-04 · methodology v2.1

Finding on 'Q008 Probe' for Claude Sonnet 4.6 with web search ON

RLB Citation ID: RLB-H-INT-BIS-CPMI-API-HARMONISATION-CROSS-BORDER-2024-Q008-Sonnet46

What the RLB Specialist Panel found

Finding on 'Q008 Probe' for Claude Sonnet 4.6 with web search ON

Question (paraphrased to protect IP): A compliance analyst was asked for a recommendation-by-recommendation stakeholder breakdown of the CPMI d224 API harmonisation report. The response fabricated per-category stakeholder assignments — naming specific institution types (e.g., 'ISO, SWIFT, regional open banking standards bodies' for Category 1) by inferring from APEX composition and category names, when the actual per-recommendation stakeholder designations are only in the inaccessible PDF.
AI's response: "CATEGORY 1 — Primary stakeholders: Standards bodies (ISO, SWIFT, regional open banking standards bodies); industry associations; regulators and central banks... CATEGORY 4 — Primary stakeholders: Commercial banks and PSPs (who must implement or consume pre-validation APIs); payment system operators... central banks and regulators"
Regulator's text: The official text states the report contains "10 recommendations directed at a broad array of stakeholders." Individual stakeholder targeting per recommendation is exclusively in the inaccessible PDF.
Why the AI went wrong: The model assigned institution types to each recommendation category by reasoning from the category name and its knowledge of payments ecosystem governance — ISO and SWIFT plausibly belong to a harmonisation-processes category; commercial banks plausibly belong to a pre-validation category. This is structurally coherent domain inference, not retrieval. The output was presented as a stakeholder breakdown with no signal to the user that it was constructed from category labels rather than retrieved from the regulatory text.
Cited source(s):

Impact for this audience

Identical failure shape to Opus 4.7 on the same question — both models assigned stakeholders by inference from category names. The cross-model convergence on this specific item confirms the failure is not model-specific: it is an architecture-level property of how both models handle 'I have the document's structure but not its content.' Correction pairs for this item should target both models' training pipelines.

References — raw findings (per AI model)

RLB-H-INT-BIS-CPMI-API-HARMONISATION-CROSS-BORDER-2024-Q008-Opus47 Claude Opus 4.7 (web search on)

This finding also affects

← Previous finding Finding on 'Q007 Probe' for Claude Sonnet 4.6 with web search ON Next finding → Finding on 'Q009 Probe' for Claude Sonnet 4.6 with web search ON

Cite this finding

Each finding has a stable Citation ID (RLB-F-… for aggregated case-study findings, RLB-H-… for raw per-model hallucinations) — like a DOI, the ID always resolves to the canonical finding even if URLs change.

RLB Citation ID: RLB-H-INT-BIS-CPMI-API-HARMONISATION-CROSS-BORDER-2024-Q008-Sonnet46

Plain text Download

RegLeg Specialist Panel (2026). "Finding on 'Q008 Probe' for Claude Sonnet 4.6 with web search ON — AI Labs." Citation ID: RLB-H-INT-BIS-CPMI-API-HARMONISATION-CROSS-BORDER-2024-Q008-Sonnet46. RegLegBrief AI Hallucination Research, published 2026-06-04. https://reglegbrief.com/regulators/j1/int/bis-cpmi/cpmi-api-harmonisation-cross-border-2024/whitepaper/finding/INT-BIS-CPMI-INT-001-CPMI-API-HARMONISATION-CROSS-BORDER-2024-v1-008--sonnet-46-websearch/

APA 7th edition Download

RegLeg Specialist Panel. (2026). Finding on 'Q008 Probe' for Claude Sonnet 4.6 with web search ON [Hallucination finding RLB-H-INT-BIS-CPMI-API-HARMONISATION-CROSS-BORDER-2024-Q008-Sonnet46]. RegLegBrief AI Hallucination Research. https://reglegbrief.com/regulators/j1/int/bis-cpmi/cpmi-api-harmonisation-cross-border-2024/whitepaper/finding/INT-BIS-CPMI-INT-001-CPMI-API-HARMONISATION-CROSS-BORDER-2024-v1-008--sonnet-46-websearch/

Bluebook / OSCOLA (US + UK legal) Download

RegLeg Specialist Panel, Finding on 'Q008 Probe' for Claude Sonnet 4.6 with web search ON [RLB-H-INT-BIS-CPMI-API-HARMONISATION-CROSS-BORDER-2024-Q008-Sonnet46], RegLegBrief AI Hallucination Research (June 04, 2026), https://reglegbrief.com/regulators/j1/int/bis-cpmi/cpmi-api-harmonisation-cross-border-2024/whitepaper/finding/INT-BIS-CPMI-INT-001-CPMI-API-HARMONISATION-CROSS-BORDER-2024-v1-008--sonnet-46-websearch/.

BibTeX Download

@misc{reglegbrief_RLB_H_INT_BIS_CPMI_API_HARMONISATION_CROSS_BORDER_2024_Q008_Sonnet46,
  author    = {RegLeg Specialist Panel},
  title     = {Finding on 'Q008 Probe' for Claude Sonnet 4.6 with web search ON},
  year      = {2026},
  publisher = {RegLegBrief AI Hallucination Research},
  note      = {Hallucination finding Citation ID: RLB-H-INT-BIS-CPMI-API-HARMONISATION-CROSS-BORDER-2024-Q008-Sonnet46},
  url       = {https://reglegbrief.com/regulators/j1/int/bis-cpmi/cpmi-api-harmonisation-cross-border-2024/whitepaper/finding/INT-BIS-CPMI-INT-001-CPMI-API-HARMONISATION-CROSS-BORDER-2024-v1-008--sonnet-46-websearch/}
}

← Back to case study summary Case study detail →

About Citation IDs →