AI Hallucination Research › Regulators › Global standard-setters › INT › UNTC › BBNJ-HIGH-SEAS-BIODIVERSITY-AGREEMENT-2023 › White paper › Detail › Finding
AI Labs · updated 2026-05-31 · methodology v2.1

Finding on 'Q001 Probe' for Claude Sonnet 4.6 with web search ON

RLB Citation ID: RLB-H-INT-UNTC-BBNJ-HIGH-SEAS-BIODIVERSITY-AGREEMENT-2023-Q001-Sonnet46
What the RLB Specialist Panel found

Finding on 'Q001 Probe' for Claude Sonnet 4.6 with web search ON

  • Question (paraphrased to protect IP): At what threshold must a planned high-seas activity undergo an environmental impact assessment under the BBNJ Agreement's screening provision, and what is the correct article reference?
  • AI's response: "if it is likely to have more than a 'minor or transitory' effect on the marine environment, or if potential effects are uncertain or not well understood"
  • Regulator's text: Parties shall ensure environmental impact assessments for activities that may have more than a minor or transitory effect on the marine environment, or the effects of which are unknown or poorly understood.
  • Why the AI went wrong: The model's formulation replaced the Agreement's phrase "unknown or poorly understood" with "uncertain or not well understood" — a paraphrase that loses the specific drafted qualifier. In a screening context, "unknown" sets a different legal standard than "uncertain", and practitioners relying on the model's wording could misjudge whether a borderline activity crosses the EIA threshold. The error is a dropped qualifier rather than a wholesale fabrication, but it is the class of error most likely to pass unchallenged in a professional setting.
  • Cited source(s):
  • https://www.tandfonline.com/doi/full/10.1080/00908320.2025.2563269 — Pretextual
Impact for this audience

This finding implicates the precision of defined-term extraction in the training corpus. The Agreement uses 'unknown or poorly understood' as the EIA screening qualifier; the model substituted 'uncertain or not well understood.' This is a near-synonym substitution that would not be caught by a general accuracy check but changes the legal standard. Structured extraction of defined terms and threshold language needs to be applied to treaty text, not only to domestic regulatory instruments.

References — raw findings (per AI model)
This finding also affects
← Previous finding Finding on 'Q011 Probe' for Claude Opus 4.7 with web search ON Next finding → Finding on 'Q002 Probe' for Claude Sonnet 4.6 with web search ON
Cite this finding

Each finding has a stable Citation ID (RLB-F-… for aggregated case-study findings, RLB-H-… for raw per-model hallucinations) — like a DOI, the ID always resolves to the canonical finding even if URLs change.

RLB Citation ID: RLB-H-INT-UNTC-BBNJ-HIGH-SEAS-BIODIVERSITY-AGREEMENT-2023-Q001-Sonnet46
Plain text Download 
RegLeg Specialist Panel (2026). "Finding on 'Q001 Probe' for Claude Sonnet 4.6 with web search ON — AI Labs." Citation ID: RLB-H-INT-UNTC-BBNJ-HIGH-SEAS-BIODIVERSITY-AGREEMENT-2023-Q001-Sonnet46. RegLegBrief AI Hallucination Research, published 2026-05-31. https://reglegbrief.com/regulators/j1/int/untc/bbnj-high-seas-biodiversity-agreement-2023/whitepaper/finding/INT-UNTC-INT-001-BBNJ-HIGH-SEAS-BIODIVERSITY-AGREEMENT-2023-v1-001--sonnet-46-websearch/
APA 7th edition Download 
RegLeg Specialist Panel. (2026). Finding on 'Q001 Probe' for Claude Sonnet 4.6 with web search ON [Hallucination finding RLB-H-INT-UNTC-BBNJ-HIGH-SEAS-BIODIVERSITY-AGREEMENT-2023-Q001-Sonnet46]. RegLegBrief AI Hallucination Research. https://reglegbrief.com/regulators/j1/int/untc/bbnj-high-seas-biodiversity-agreement-2023/whitepaper/finding/INT-UNTC-INT-001-BBNJ-HIGH-SEAS-BIODIVERSITY-AGREEMENT-2023-v1-001--sonnet-46-websearch/
Bluebook / OSCOLA (US + UK legal) Download 
RegLeg Specialist Panel, Finding on 'Q001 Probe' for Claude Sonnet 4.6 with web search ON [RLB-H-INT-UNTC-BBNJ-HIGH-SEAS-BIODIVERSITY-AGREEMENT-2023-Q001-Sonnet46], RegLegBrief AI Hallucination Research (May 31, 2026), https://reglegbrief.com/regulators/j1/int/untc/bbnj-high-seas-biodiversity-agreement-2023/whitepaper/finding/INT-UNTC-INT-001-BBNJ-HIGH-SEAS-BIODIVERSITY-AGREEMENT-2023-v1-001--sonnet-46-websearch/.
BibTeX Download 
@misc{reglegbrief_RLB_H_INT_UNTC_BBNJ_HIGH_SEAS_BIODIVERSITY_AGREEMENT_2023_Q001_Sonnet46,
  author    = {RegLeg Specialist Panel},
  title     = {Finding on 'Q001 Probe' for Claude Sonnet 4.6 with web search ON},
  year      = {2026},
  publisher = {RegLegBrief AI Hallucination Research},
  note      = {Hallucination finding Citation ID: RLB-H-INT-UNTC-BBNJ-HIGH-SEAS-BIODIVERSITY-AGREEMENT-2023-Q001-Sonnet46},
  url       = {https://reglegbrief.com/regulators/j1/int/untc/bbnj-high-seas-biodiversity-agreement-2023/whitepaper/finding/INT-UNTC-INT-001-BBNJ-HIGH-SEAS-BIODIVERSITY-AGREEMENT-2023-v1-001--sonnet-46-websearch/}
}
← Back to case study summary Case study detail →
About Citation IDs →