Certification is Not Control: What Quality Discipline Adds to AI Governance in Mortgage

Roberta Pek, CMB, CRCM, is a Lean/Six Sigma Master Black Belt with more than 20 years in mortgage banking and compliance. She is Past Chair of the ASQ Financial Services Division and has published in Quality Progress. In August, she participated in an MBA webinar, AI Governance, Quality & Risk in Mortgage Operations.


Fannie Mae and Freddie Mac now require governance frameworks for sellers and servicers using artificial intelligence and machine learning in covered mortgage activities. The next question is practical: how do we know the safeguards work?

Roberta Pek, CMB, CRCM

Camillo Melchiorre’s Oct. 6 MBA NewsLink article makes the case for an AI management system using ISO/IEC 42001. I agree. I want to build on that discussion by showing how quality discipline helps lenders and servicers measure whether their safeguards work in daily operations.

A certificate is valuable. It is not a guarantee that a model produces accurate results in your operation.

What Certification Tells Us

ISO/IEC 42001 is a management system standard. Evaluation and improvement are part of that framework. Quality discipline gives teams practical ways to carry those expectations into the workflow. Certification is evidence about the management system within its assessed scope. Firms still need evidence that each tool, and the processes around it, meet their requirements.

Manufacturing offers a familiar lesson: a certified plant can still produce defective parts. Mortgage banking also has established quality control and model validation practices. The task is to apply that discipline to AI and the people and systems around it. Three practices offer a starting point.

Measure the Errors That Matter

Consider a document extraction tool that uses AI to read tax and insurance documents for escrow updates. Some scanners use conventional optical character recognition or rules instead. The testing principles apply to those tools too.

Define an error first. Is it an incorrect dollar amount, a missed due date, or a document assigned to the wrong loan? Separate errors that could affect a borrower from minor formatting issues. Measure these categories separately when they require different responses.

Select a random sample and compare the extracted information with verified source documents. Five documents with material extraction errors out of 100 is a 5% document error rate. That is different from counting incorrect fields. Define what you count and use it consistently. Then follow the information through the process. Did the correct amount reach the correct loan and produce the correct escrow update? Accurate extraction alone does not answer that question.

The sample size and testing frequency should reflect volume, risk, and the precision needed. One hundred documents is an illustration, not a universal standard. No errors in a small sample does not prove the tool is error-free.

Chart results over time and investigate changes. Separate results by document type or vendor when averages could hide a problem. Check the measurement process too: have two reviewers independently assess the same documents, resolve disagreements, and clarify the instructions.

Decide When to Intervene

A measure becomes useful when it leads to action. Set the trigger, identify the owner, and document the response before a problem occurs.

Distinguish between two decisions. An individual document may require human review because a critical field is missing, information conflicts, or the tool’s confidence falls below a tested threshold. A rise in the overall error rate may require investigation, expanded review, or a pause in automated processing.

A vendor’s confidence score is not proof of accuracy. Test whether that score reliably identifies errors in your operation.

Establish thresholds through testing and have the responsible business and risk owners approve them based on borrower impact and operating requirements. A wrong tax amount deserves different treatment from an extra space in an address. A serious error may warrant immediate escalation even when the overall rate looks acceptable.

Statistical control limits can help identify unusual variation. They are not the same as acceptable error limits. A process can be statistically stable and still produce too many errors.

Test the Human Safeguard

“A person reviews the output” describes an activity. It does not establish how well that person detects errors.

Use a controlled test environment with realistic documents containing known errors. Keep these test items out of live borrower records. Include correct documents too, so you can see whether reviewers flag accurate work unnecessarily.

Measure the share of known errors reviewers detect. Catching eight of ten gives an observed detection rate of 80%. Record what they missed and its significance. Pair these exercises with independent checks of completed production work. A staged test alone does not show how reviewers perform under normal workload and time pressure.

Missed errors call for investigation. Look at their frequency and severity, along with workload, instructions, training, and available information. Improve the process and test it again.

Apply it Across the Mortgage Lifecycle

The same questions apply to origination tools used for document classification, underwriting support, and borrower communications. Where AI affects credit decisions or loss mitigation, evaluate relevant differences across borrower groups with qualified fair lending and compliance staff. Overall accuracy alone does not establish fair treatment.

For vendor tools, request test results relevant to your use, known limitations, and information about release changes. You may not need proprietary model details to test the results in your workflow. Outsourcing the technology does not remove the need for oversight.

Three Questions to Ask

What errors matter to the borrower, and how are they measured across the process?

What triggers human review or a pause in automation, and who acts?

When was the human safeguard last tested, and what did it catch?

Policies establish expectations. Results show whether those expectations are being met. Start with one tool. Define the errors that matter, follow the output through the workflow, and test the human safeguard. That is how quality discipline helps demonstrate whether AI governance works in daily mortgage operations.

References

ISO/IEC 42001:2023 — https://www.iso.org/standard/42001

Fannie Mae Lender Letter LL-2026-04 — https://singlefamily.fanniemae.com/media/45196/display

Freddie Mac Guide Bulletin 2025-16 — https://guide.freddiemac.com/app/guide/faqs_subcategory/3334

(Views expressed in this article do not necessarily reflect policies of the Mortgage Bankers Association, nor do they connote an MBA endorsement of a specific company, product or service. MBA NewsLink welcomes submissions from member firms. Inquiries can be sent to Editor Michael Tucker or Editorial Manager Anneliese Mahoney.)