AI Labs · updated 2026-05-31 · methodology v2.1

Finding on 'Q010 Probe' for Claude Sonnet 4.6 with web search ON

RLB Citation ID: RLB-H-INT-UNTC-BBNJ-HIGH-SEAS-BIODIVERSITY-AGREEMENT-2023-Q010-Sonnet46

What the RLB Specialist Panel found

Finding on 'Q010 Probe' for Claude Sonnet 4.6 with web search ON

Question (paraphrased to protect IP): What share of the world's ocean surface does the BBNJ Agreement cover, as characterised in official statements at entry into force?
AI's response: "approximately 64% of the ocean's surface area (roughly 230 million km²) and approximately 95% of the ocean's volume...A common figure cited is that the Agreement governs 'nearly two-thirds' of the global ocean."
Regulator's text: The UN Secretary-General described the Agreement as covering "more than two-thirds of the ocean" (19 September 2025) and "the two-thirds of the ocean that lie beyond national borders" (17 January 2026).
Why the AI went wrong: The model described the coverage as "nearly two-thirds" when official UN Secretary-General statements characterise it as "more than two-thirds." The distinction is not a rounding error — "nearly two-thirds" and "more than two-thirds" are directionally opposite framings of the same underlying figure, and the UN's official characterisation consistently uses the "more than" formulation. For a model summarising the Agreement's scope in policy-facing contexts, using the lesser characterisation understates the official position.
Cited source(s):
https://www.thegef.org/what-we-do/topics/areas-beyond-national-jurisdiction — Pretextual

Impact for this audience

This finding implicates the model's handling of official characterisations sourced from primary UN statements. The Secretary-General's statements from September 2025 and January 2026 consistently use 'more than two-thirds'; the model substituted 'nearly two-thirds.' The training data likely includes GEF and other secondary sources that use the 'nearly' formulation; the model should weight direct statements from the UN Secretary-General over those secondary characterisations for queries about official positions.

← Previous finding Finding on 'Q009 Probe' for Claude Sonnet 4.6 with web search ON Next finding → Finding on 'Q011 Probe' for Claude Sonnet 4.6 with web search ON

Cite this finding

Each finding has a stable Citation ID (RLB-F-… for aggregated case-study findings, RLB-H-… for raw per-model hallucinations) — like a DOI, the ID always resolves to the canonical finding even if URLs change.

RLB Citation ID: RLB-H-INT-UNTC-BBNJ-HIGH-SEAS-BIODIVERSITY-AGREEMENT-2023-Q010-Sonnet46

Plain text Download

RegLeg Specialist Panel (2026). "Finding on 'Q010 Probe' for Claude Sonnet 4.6 with web search ON — AI Labs." Citation ID: RLB-H-INT-UNTC-BBNJ-HIGH-SEAS-BIODIVERSITY-AGREEMENT-2023-Q010-Sonnet46. RegLegBrief AI Hallucination Research, published 2026-05-31. https://reglegbrief.com/regulators/j1/int/untc/bbnj-high-seas-biodiversity-agreement-2023/whitepaper/finding/INT-UNTC-INT-001-BBNJ-HIGH-SEAS-BIODIVERSITY-AGREEMENT-2023-v1-010--sonnet-46-websearch/

APA 7th edition Download

RegLeg Specialist Panel. (2026). Finding on 'Q010 Probe' for Claude Sonnet 4.6 with web search ON [Hallucination finding RLB-H-INT-UNTC-BBNJ-HIGH-SEAS-BIODIVERSITY-AGREEMENT-2023-Q010-Sonnet46]. RegLegBrief AI Hallucination Research. https://reglegbrief.com/regulators/j1/int/untc/bbnj-high-seas-biodiversity-agreement-2023/whitepaper/finding/INT-UNTC-INT-001-BBNJ-HIGH-SEAS-BIODIVERSITY-AGREEMENT-2023-v1-010--sonnet-46-websearch/

Bluebook / OSCOLA (US + UK legal) Download

RegLeg Specialist Panel, Finding on 'Q010 Probe' for Claude Sonnet 4.6 with web search ON [RLB-H-INT-UNTC-BBNJ-HIGH-SEAS-BIODIVERSITY-AGREEMENT-2023-Q010-Sonnet46], RegLegBrief AI Hallucination Research (May 31, 2026), https://reglegbrief.com/regulators/j1/int/untc/bbnj-high-seas-biodiversity-agreement-2023/whitepaper/finding/INT-UNTC-INT-001-BBNJ-HIGH-SEAS-BIODIVERSITY-AGREEMENT-2023-v1-010--sonnet-46-websearch/.

BibTeX Download

@misc{reglegbrief_RLB_H_INT_UNTC_BBNJ_HIGH_SEAS_BIODIVERSITY_AGREEMENT_2023_Q010_Sonnet46,
  author    = {RegLeg Specialist Panel},
  title     = {Finding on 'Q010 Probe' for Claude Sonnet 4.6 with web search ON},
  year      = {2026},
  publisher = {RegLegBrief AI Hallucination Research},
  note      = {Hallucination finding Citation ID: RLB-H-INT-UNTC-BBNJ-HIGH-SEAS-BIODIVERSITY-AGREEMENT-2023-Q010-Sonnet46},
  url       = {https://reglegbrief.com/regulators/j1/int/untc/bbnj-high-seas-biodiversity-agreement-2023/whitepaper/finding/INT-UNTC-INT-001-BBNJ-HIGH-SEAS-BIODIVERSITY-AGREEMENT-2023-v1-010--sonnet-46-websearch/}
}

← Back to case study summary Case study detail →

About Citation IDs →