Closer to Zero: Sampling Meets Population Testing (sponsored by ACES Quality Management)

By Sharon Reichhardt

The cornerstone of every risk management program is the QC plan, and QC professionals are all too familiar with having to “define and defend” their QC plan to auditors and regulators. One key component of the plan that is frequently audited is the sampling methodologies used.  

The GSE’s are pretty clear when it comes to their sampling requirements, which focus on volume and risk . As much as the industry strives for error-free loans and servicing activities, perfection is just not possible. The goal is always to get as close to perfect as is feasible, but a 10% sample only gets a lender so far. With the right technology, lenders can now uncover far more of what stands between them and zero than ever before.

Sampling as the historical standard

The industry settled on a sample-based methodology for one simple reason: labor constraints. It is not feasible to conduct full file reviews on every loan in a lender’s pipeline or servicing book, so the industry had to find a way to ensure defects were identified in a manner that aligned with those constraints and enabled rapid identification.

The framework lenders operate under today traces to HUD Mortgagee Letter 89-32 (1989), which required FHA-approved mortgagees to review at least 10% of insured mortgages originated each month, selected randomly, plus every loan that defaulted within six payments. HUD Handbook 4060.1 REV-2 codified this in 2006 and added the alternative still used today: a statistical random sample at a 95% confidence level with 2% precision, available to mortgages originating more than 3,500 FHA loans a year, a threshold keyed to volume, not risk.

Methods of random selection include straight random and stratified random, and many lenders also run a discretionary sample alongside it. Which methods are in play depends partly on volume, since lenders and servicers that don’t meet the minimums for a statistically valid sample work from the 10% floor instead. ACES Quality Management & Control® software supports all sampling methodologies.

Random selection methods defend themselves because their logic is statistical. There is no judgment about where problems live, and each supports a valid conclusion about the population.

Discretionary selections are defended differently because they typically arise from existing findings. If there has been a historical problem with a subset of loans, the lender will run a discretionary sample to ensure that previously identified issues have been corrected. But that fix says nothing about whether the same condition exists elsewhere in the portfolio, in loans nobody looked at. The logic runs backward here, as the finding creates the sample, not the other way around. In all instances, sampling relies on reviewing parts to infer the whole.

That inference is sound, and it is also where the gap sits. Random selection produces a valid read on the population, but it lands where chance puts it. Discretionary selection goes straight at risk, but only the risk that the lender already knows to look for. Neither reaches the defect that nobody has flagged because no program knows what it does not know.

ACES’s own Q4 2025 industry data show how quickly that gap can widen. Borrower/mortgage eligibility defects rose 291.58% year-over-year, and the leading defect category flipped twice within a year. This is exactly the kind of shift a sample built on last quarter’s known problems is late to catch.

In addition, the cost of relying on a sample alone continues to grow, as mortgage repurchase and indemnification activity reached $239.39 million in the second quarter of 2026, a 16.5% increase from the first quarter and a reminder of how quickly undetected risk can resurface after a loan has closed.

Technology narrows the gap

With the technological advances available to lenders today, they no longer have to rely on sampling alone to get to the heart of what is impacting loan quality. Population testing has opened the door for lenders and servicers to become far more targeted and strategic in their loan audits, helping the industry uncover the failures still standing between it and the zero-defect goal.

ACES Population Testing™ is built around that shift. It extends quality control beyond the required sample, surfacing potential defects and concentrated pockets of risk across the entire loan portfolio and pipeline. The software evaluates every loan in a defined population against a client-defined set of policies which are written, reviewable and consistent, with coverage across pre-funding, post-closing and servicing reviews. From there, auditors receive a pass-or-fail status for each loan, with a dashboard that displays an overall quality score based on those results. That explainability helps compliance teams define and defend every flag because the same policies are applied to the same population and return the same result every time, with the reason attached.

Sampling-based loan auditing is not going anywhere and remains mandated by the GSEs and required by investors and lenders’ own policies. Population testing runs alongside it, helping lenders and servicers identify where to look outside their sample, while the review and remediation workflow within ACES Quality Management & Control software does the looking and the fixing, regardless of how the population was selected. Sourcing both from a single provider keeps the selection evidence and the review within one program.

For investors, the sharper question isn’t what a lender’s quality control found but what it didn’t test. A defect rate calculated on a sample describes only the sample. For example, a 2% critical defect rate on a month’s production of 5,000 originated loans works out to roughly 100 defective loans, and a sample uncovers only a handful of them. That gap is exactly where a repurchase demand or a GSE remediation plan takes shape.

A 2024 STRATMOR/Reggora study put the average cost of repurchasing a defective loan from Fannie Mae and Freddie Mac at $32,288 on a repurchase rate of 49 basis points. Population testing removes the stratification math altogether since every loan can be tested, so, there is no unaudited corner left for a defect to hide until an investor or examiner finds it first.

Why zero stays out of reach

So why can’t lenders get to a 0% defect rate? There are so many variables involved in mortgage production. Regulatory rules and requirements do not stay static. Investor requirements shift with market conditions, and humans are involved throughout the process, introducing errors that cannot be predicted or planned for. Even with every rule, regulation and guideline lenders must follow, human judgment still plays a role in origination.

When run together, sampling and population testing produce a more comprehensive picture of loan quality, moving an organization closer to the zero-defect endgame. The result: the same auditors working a different queue, where a loan arrives because it failed a specific test, not because a sampling frame selected it.

For as long as QC has existed, the gap between the program a lender runs and the one it could run without limits has been a matter of manpower, not willpower. ACES Population Testing takes manpower out of the equation, making portfolio size a choice rather than a constraint.

The end goal has not moved. What has changed is how much closer a lender can get to it.

(Sponsored content includes material submitted independently of the Mortgage Bankers Association and MBA NewsLink and does not connote an MBA endorsement of a specific company, product or service. For more information about sponsored content opportunities, contact Bill Farmakis at bill@jlfarmakis.com or 203/834-8832.)