MAS Notice 637 Capital Adequacy: AI Model Accuracy Evaluation

This paper presents findings from RegLeg's evaluation of AI model responses to questions about MAS Notice 637 — the Monetary Authority of Singapore's risk-based capital adequacy framework for banks — covering both the consolidated notice and its 2024 amendment. Two Anthropic models were tested in web-search-enabled configurations: Claude Opus 4.7 with web search and Claude Sonnet 4.6 with web search. Across six findings, both models produced responses in which the model asserted specific regulatory details — annex content, document structure, the significance of formatting elements — that had no basis in the regulator's published text and in some cases directly contradicted it. The dominant pattern is one of confident fabrication in low-retrieval-coverage territory: when the model's search results do not surface the precise regulatory text, it generates plausible-sounding content instead of signalling uncertainty. For labs fielding these models in enterprise and regulatory contexts, this pattern represents a material gap in how the models handle authoritative technical documents under partial information retrieval.