AI Labs · updated 2026-05-31 · methodology v2.1

Finding on 'Q011 Probe' for Claude Sonnet 4.6 with web search ON

RLB Citation ID: RLB-H-INT-UNTC-BBNJ-HIGH-SEAS-BIODIVERSITY-AGREEMENT-2023-Q011-Sonnet46

What the RLB Specialist Panel found

Finding on 'Q011 Probe' for Claude Sonnet 4.6 with web search ON

Question (paraphrased to protect IP): What is the UN depositary notification reference number recording the BBNJ Agreement's achievement of the 60-ratification entry-into-force threshold?
AI's response: "The depositary notification reference number for the BBNJ Agreement's entry into force is C.N.447.2025.TREATIES-XXI.10. This was the UN Secretary-General's Depositary Notification issued in 2025 recording that the 60th instrument of ratification had been deposited on 19 September 2025"
Regulator's text: The depositary notification reference is C.N.447.2025.TREATIES-XXI.10, confirmed from the UN Treaty Collection record and corroborated by treaty-monitoring sources.
Why the AI went wrong: The model's answer was correct — it stated the right reference number and the correct date. It is included because the model reached the right answer via secondary treaty-monitoring sources rather than the primary UN Treaty Collection record, and the sources cited are third-party aggregators. A model that produces the correct answer from secondary sources in this instance may produce a wrong answer on a different treaty where the secondary sources are less reliable or slower to update. The retrieval path matters, not just the outcome.
Cited source(s):
https://bbnj-mgr.fas.harvard.edu/status-bbnj-treaty-entry-force — Pretextual
https://sdg.iisd.org/news/with-60-ratifications-bbnj-agreement-to-enter-into-... — Pretextual

Impact for this audience

This finding is correct but illustrates the same retrieval-routing gap as the Opus 4.7 depositary notification finding: the model arrived at the right answer via secondary aggregators rather than the UN Treaty Collection primary record. A model that consistently routes treaty-registry queries through secondary sources will produce correct answers when those sources happen to be accurate and wrong answers when they lag or diverge. Retrieval routing for treaty-registry queries should prioritise official UN portals.

← Previous finding Finding on 'Q010 Probe' for Claude Sonnet 4.6 with web search ON Next finding → Finding on 'Q012 Probe' for Claude Sonnet 4.6 with web search ON

Cite this finding

Each finding has a stable Citation ID (RLB-F-… for aggregated case-study findings, RLB-H-… for raw per-model hallucinations) — like a DOI, the ID always resolves to the canonical finding even if URLs change.

RLB Citation ID: RLB-H-INT-UNTC-BBNJ-HIGH-SEAS-BIODIVERSITY-AGREEMENT-2023-Q011-Sonnet46

Plain text Download

RegLeg Specialist Panel (2026). "Finding on 'Q011 Probe' for Claude Sonnet 4.6 with web search ON — AI Labs." Citation ID: RLB-H-INT-UNTC-BBNJ-HIGH-SEAS-BIODIVERSITY-AGREEMENT-2023-Q011-Sonnet46. RegLegBrief AI Hallucination Research, published 2026-05-31. https://reglegbrief.com/regulators/j1/int/untc/bbnj-high-seas-biodiversity-agreement-2023/whitepaper/finding/INT-UNTC-INT-001-BBNJ-HIGH-SEAS-BIODIVERSITY-AGREEMENT-2023-v1-011--sonnet-46-websearch/

APA 7th edition Download

RegLeg Specialist Panel. (2026). Finding on 'Q011 Probe' for Claude Sonnet 4.6 with web search ON [Hallucination finding RLB-H-INT-UNTC-BBNJ-HIGH-SEAS-BIODIVERSITY-AGREEMENT-2023-Q011-Sonnet46]. RegLegBrief AI Hallucination Research. https://reglegbrief.com/regulators/j1/int/untc/bbnj-high-seas-biodiversity-agreement-2023/whitepaper/finding/INT-UNTC-INT-001-BBNJ-HIGH-SEAS-BIODIVERSITY-AGREEMENT-2023-v1-011--sonnet-46-websearch/

Bluebook / OSCOLA (US + UK legal) Download

RegLeg Specialist Panel, Finding on 'Q011 Probe' for Claude Sonnet 4.6 with web search ON [RLB-H-INT-UNTC-BBNJ-HIGH-SEAS-BIODIVERSITY-AGREEMENT-2023-Q011-Sonnet46], RegLegBrief AI Hallucination Research (May 31, 2026), https://reglegbrief.com/regulators/j1/int/untc/bbnj-high-seas-biodiversity-agreement-2023/whitepaper/finding/INT-UNTC-INT-001-BBNJ-HIGH-SEAS-BIODIVERSITY-AGREEMENT-2023-v1-011--sonnet-46-websearch/.

BibTeX Download

@misc{reglegbrief_RLB_H_INT_UNTC_BBNJ_HIGH_SEAS_BIODIVERSITY_AGREEMENT_2023_Q011_Sonnet46,
  author    = {RegLeg Specialist Panel},
  title     = {Finding on 'Q011 Probe' for Claude Sonnet 4.6 with web search ON},
  year      = {2026},
  publisher = {RegLegBrief AI Hallucination Research},
  note      = {Hallucination finding Citation ID: RLB-H-INT-UNTC-BBNJ-HIGH-SEAS-BIODIVERSITY-AGREEMENT-2023-Q011-Sonnet46},
  url       = {https://reglegbrief.com/regulators/j1/int/untc/bbnj-high-seas-biodiversity-agreement-2023/whitepaper/finding/INT-UNTC-INT-001-BBNJ-HIGH-SEAS-BIODIVERSITY-AGREEMENT-2023-v1-011--sonnet-46-websearch/}
}

← Back to case study summary Case study detail →

About Citation IDs →