Responsible AI in document generation: A framework for what to trust and what to govern 

Responsible AI in document generation: A framework for what to trust and what to govern 

Responsible AI in document generation is not a marketing posture. It is a three-test discipline: a named human is accountable for any consequential AI output, an auditor can reconstruct what the AI did and where it sat in the workflow, and the AI operates inside the customer’s existing governance posture rather than asking that posture to bend. A document automation platform that cannot pass these three tests is a regulatory exposure, not a productivity tool. 

Last month we set out where AI earns its place in document generation. This month, the harder question: what happens when AI shows up in the wrong place, and how compliance, risk, and audit leaders tell the difference before the regulator does it for them. 

The “it was just the chatbot” defence failed 

In February 2024, the British Columbia Civil Resolution Tribunal handed down a ruling every AI-governance committee should keep on a shelf. Jake Moffatt asked Air Canada’s chatbot about bereavement fares. The chatbot invented a policy allowing retroactive discount claims within 90 days. The actual policy did not. When Moffatt applied for the refund, Air Canada refused. The airline’s defence was that the chatbot was “a separate legal entity responsible for its own actions.” The tribunal rejected that defence and held Air Canada liable for negligent misrepresentation. (Source: cbc.ca.) 

The amount was small. The precedent is not. A regulated firm cannot route accountability for a customer-facing communication through a model layer and expect that layer to absorb the liability. The chatbot is an output of the firm, and the firm answers for it. 

For document generation, the implication is direct. Customer statements, policy documents, regulatory disclosures, and arrears letters are higher-stakes versions of the Air Canada chat. If an AI fabricates a number on page three of a million-policyholder run, “it was just the model” will not work in front of a financial regulator either. 

The 3-test gate 

The framework below is the one our enterprise customers run any candidate AI feature through, before that feature touches a regulated workflow. It works as a 30-second triage on a vendor pitch and as a structured review for a board paper. 

Test 1: Is there a named human accountable for any consequential AI output? 

Not “the AI team.” A named individual with a role and a register entry. In UK financial services, that role already exists under the Senior Managers and Certification Regime; the FCA confirmed in September 2025 that it will not introduce AI-specific rules and will use Consumer Duty and SM&CR to allocate accountability for AI-driven decisions. (Source: fca.org.uk.) In the United States, NYDFS Insurance Circular Letter No. 7, issued 11 July 2024, requires board-level governance for AI used in underwriting and pricing. (Source: dfs.ny.gov.) 

Pass condition: a named human approves before the document leaves the building, and the approval is logged at the document or template level. 

Test 2: Can an auditor verify what the AI did, what input, and where it sits in the workflow? 

The auditor’s question is forensic, not philosophical. Given a specific document a customer received, can someone reconstruct which model produced it, which inputs it received, which retrieval context it had, which version of which template it operated against, and which human approved the output? If any one of those answers is “we are not sure,” the AI is not deployable in a regulated workflow. 

NIST’s AI Risk Management Framework organises this under Map and Measure: AI systems in regulated contexts must be inventoried as such, and their outputs must be validated against ground truth before production and monitored for drift after. (Source: nist.gov.) BCBS 239’s data accuracy and completeness principles apply the same expectation to risk reporting. 

Pass condition: a per-document, per-template, or per-decision record that ties every consequential output back to inputs, model, version, and approver. 

Test 3: Does the AI fit inside the customer’s existing governance posture, or does the governance have to bend? 

This is the test that ends most vendor demos quickly. If deploying the AI requires a new sign-off layer, a new audit framework, or a carve-out from data-residency rules, the AI is not ready for the regulated workflow yet. Responsible AI features fit inside the controls already in place: role-based access, segregation of duties, change-management approval, version control, in-tenant data handling, and the existing model risk management posture. 

US banking organisations now operate under SR 26-2, the Fed, OCC, and FDIC joint guidance issued in April 2026 that supersedes SR 11-7. SR 26-2 explicitly excludes generative AI from its scope while requiring banking organisations to apply their broader risk management practices to systems not covered. (Source: federalreserve.gov.) Regulators are not waiting for an AI-specific rulebook. The existing rulebook applies. 

Pass condition: the AI runs under the same controls that already govern the rest of the document platform. 

Where AI belongs in document generation 

Apply the three tests to a real document automation workflow and the AI surface area narrows to four places, all of them design-time or advisory. None of them sits in the runtime path that produces the document the customer receives. 

Template authoring assistance. A template designer drafts a customer-statement template; the AI suggests structure, surfaces missing data fields, drafts plain-English wording for a paragraph the legal team will sign off. The human reviews, edits, and approves before the template enters the library. Every change is version-controlled. The 3-test gate passes: the named designer is accountable, approval is logged at the template level, and the governance posture is unchanged. 

Migration acceleration. A legacy template estate moves to a modern platform; the AI proposes conversions, flags ambiguous logic, generates a parity test set. A human signs off on every converted template before it goes live. The AI compresses engineering time without compressing the review. 

Output QA and template flaw detection. After a batch run produces a hundred thousand documents, an AI scans the output for anomalies: missing required clauses, layout breaks, inconsistent totals, wrong-language content for the customer’s profile. The AI flags issues for a human investigator. It does not edit the document, send a correction, or close the issue. 

Analysis across batch runs. Across millions of statements, an AI surfaces patterns a human reviewer could not see by hand: which template variants produced complaint spikes, which data-input combinations triggered exception handling, which output configurations consumed disproportionate compute. Advisory. Surfaced for human investigation, not acted on without review. 

All four share the same shape: AI does the heavy lifting, a named human stays in control, the audit trail is unbroken, the governance posture is preserved. 

Where AI does not belong 

The same three tests rule out three deployments that vendors are actively pitching into regulated buyers right now. 

Runtime generation of regulated output. A statement, policy schedule, regulatory disclosure, or arrears letter generated live by an LLM at run time fails Test 2 by construction. Generative inference is non-deterministic: token selection is probabilistic, and temperature-zero settings do not guarantee identical outputs across hardware, software versions, or batch sizes. For a regulator with a specific format specification, any variation is a technically rejected submission. For a customer expecting a contractually defined disclosure, any variation is a misrepresentation. The deterministic template engine is the only architecture an auditor can verify per-document at scale. 

Unsupervised decisions on customer-facing communications. Air Canada is the case. So is the Virgin Money chatbot incident of January 2025, where a customer typed the bank’s own name in a query and was lectured by the bank’s chatbot for using inappropriate language. (Source: fortune.com.) The CFPB’s Chatbots in Consumer Finance report documents systemic failures in US retail banking chatbots: doom-loop entrapment, incorrect dispute guidance, mortgage applicants locked out for weeks. The CFPB stated explicitly that institutions using deficient chatbots risk violating federal consumer financial laws. (Source: consumerfinance.gov.) 

Compliance-text validation without human counter-check. This failure mode produces the worst outcomes because it sounds the most reasonable. The AI reads a draft disclosure, says “this complies with [regulation],” and a busy compliance team treats the green light as the review. Confidence in the model becomes the substitute for the review. The legal profession’s recent record is the cleanest evidence of this pattern. 

Wadsworth v. Walmart Inc. (D. Wyo., February 2025) 

A Morgan & Morgan attorney used the firm’s in-house AI platform to generate case law for motions in limine. Eight of the nine cases cited did not exist. He filed without checking. Two co-signing attorneys signed without checking. The court sanctioned all three, revoked the lead attorney’s pro hac vice admission, and the judge wrote: “the duty to read is nondelegable.” (Source: lawnext.com.) Lesson for regulated document generation: AI-flagged “compliant” is not a review. It is an input to one. 

Lacey v. State Farm (C.D. Cal., May 2025) 

An Ellis George attorney used AI to generate a research outline with citations that did not exist. He passed it to K&L Gates attorneys, who filed a brief based on it without checking and without knowing AI had been used. One-third of the citations were wrong: nine of twenty-seven, either non-existent or material misstatements of the law. Joint and several sanction of $31,100 against both firms. The Special Master wrote: “I read their brief, was persuaded… only to find that the authorities didn’t exist.” (Source: abovethelaw.com.) Lesson: when AI use is invisible across a hand-off, accountability is invisible too. 

Presto Voice (SEC Administrative Proceeding, January 2025) 

The SEC charged Presto Automation, formerly Nasdaq-listed, with misleading investors about its Presto Voice AI product for drive-thru restaurants. The company described the system as AI-driven and failed to disclose that the AI was a third party’s, and that over 70 per cent of all orders required human intervention, with 100 per cent of orders at certain locations requiring manual override. The SEC’s first AI-washing enforcement against a public company. (Source: sec.gov.) Lesson: claiming AI capability you cannot demonstrate is not a positioning choice. It is a securities matter. 

The regulator already has a date in the diary 

The EU AI Act came into force in August 2024. Prohibited practices have applied since February 2025. The high-risk obligations bite this month, August 2026, for most stand-alone systems listed in Annex III. (Source: legalnodes.com.) For financial services, Annex III explicitly names credit scoring and creditworthiness assessment of natural persons, and risk assessment and pricing in life and health insurance, as high-risk applications. 

The Act’s Articles 9, 10, 13, and 14 codify the three tests almost word for word: continuous risk management across the system lifecycle, verifiable training and validation data, transparency sufficient for a deployer to interpret the output, and meaningful human oversight with the ability to intervene and override. Penalties run to 35 million euros or 7 per cent of global turnover for prohibited practices. 

ISO/IEC 42001:2023 sets the same expectations in certifiable form. (Source: iso.org.) 

The convergence across NIST AI RMF, the EU AI Act, ISO 42001, SR 26-2, BCBS 239, NYDFS Circular 7, and SM&CR is the point. Regulators in five jurisdictions are arriving at the same three questions: who is accountable, can it be audited, does it fit the existing governance. 

The 95 per cent and the 5 per cent 

In August 2025, MIT’s GenAI Divide research found that 95 per cent of enterprise generative AI pilots are failing to deliver measurable P&L impact. (Source: fortune.com.) The successful 5 per cent are not the firms with the largest model budgets. They are the firms that knew, before deployment, where AI belonged in their workflow and where it did not. They drew the line at the design-time boundary, kept the runtime path deterministic, kept the named human in the approval loop, and built the audit trail before production traffic arrived. 

Gartner’s June 2025 forecast added the negative version: over 40 per cent of agentic AI projects will be cancelled by end of 2027 due to escalating costs, unclear value, or inadequate risk controls. (Source: gartner.com.) Each cancellation failed at least one of the three tests before it was switched off. 

What this looks like in a production document estate 

A tier-1 bank we operate inside runs about 30 million pages a month through deterministic document generation. Platform-wide, the engine produces in excess of 100 million pages a year at a 99.9999 per cent success rate across 2.26 million production records. In none of these environments does a live LLM produce a customer document. AI assists template designers, accelerates migration, audits batch output for anomalies, and answers analytical questions about run-level patterns. The production path is deterministic, the audit trail is per-document, and the governance posture is unchanged. 

That is the realist’s version of “AI-embracing.” Not AI as a substitute for the production engine. AI as a force multiplier on the people who design, govern, and audit it. 

Key takeaways 

• Responsible AI in document generation is a discipline, not a feature. It runs against a three-test gate: named human accountability, auditor-verifiable execution, fit with existing governance. 

• AI belongs at design time and in advisory roles: template authoring assistance, migration acceleration, output QA, batch analysis. None of these touch the runtime path that produces the customer document. 

• AI does not belong at runtime for regulated output, in unsupervised customer-facing decisions, or as a substitute for compliance review. The Air Canada, Wadsworth v. Walmart, Lacey v. State Farm, and Presto Voice cases set the precedent. 

• The convergence across NIST AI RMF, the EU AI Act (high-risk obligations from August 2026), ISO 42001, SR 26-2, BCBS 239, NYDFS Circular Letter 7, and SM&CR is the same three questions. The regulator is not waiting on a new rulebook. 

• The successful 5 per cent of enterprise AI pilots are the ones that applied the three tests before deployment, not after. The 95 per cent did not. 

• A vendor whose answer to AI is to replace the deterministic template engine with live inference is asking the customer’s governance posture to bend. Read it as the answer it is. 

Book an AI governance review for your document operations 

We run AI governance reviews for enterprise document operations. The review applies the 3-test gate to your existing and candidate AI deployments, identifies which sit safely inside your governance posture and which do not, and sets out what an AI-augmented production architecture looks like for your regulated workloads. 

If you are preparing for the EU AI Act high-risk deadline, briefing your board on AI exposure across customer-facing communications, or scoping a responsible-AI position for procurement, this is the conversation to have first. 

Book an AI governance review for your document operations. 

Find out more & stay in the loop

🔗 Follow us on LinkedIn: https://www.linkedin.com/company/docfusion-cloud
▶️ To subscribe to our YouTube channel, click here: https://bit.ly/SubscribeDocFusion