First American–AVMs Take Center Stage as New Rules Spotlight Performance

By Jon Wierks, Vice President, Data & Analytics with First American

Automated valuation models (AVMs) have been a fixture in our industry for decades. These workhorse products, largely invisible to most consumers, are used to originate home equity products, test appraisal quality, value portfolios, and make servicing decisions. But, given new federal guidelines and changes in testing methodology, AVMs are about to get more attention — and possibly more scrutiny — from regulators and lenders.

By next July, lenders will be required to demonstrate the steps they are taking to track and maintain standards for accuracy and prevent bias in the AVMs they use. At the same time, new AVM testing methods are already providing more targeted insights into how various AVMs are performing.

The increasing volume of new AVM products coming to market and widening industry use of AVMs mean preparing to comply with the new guidelines takes on added importance.

In the past two years, for example, First American Data & Analytics significantly enhanced its AVM suite, as have some other major providers. The ability to fuel AVMs with larger datasets, combined with advanced ensemble modeling techniques using artificial intelligence and machine learning, have made these AVMs more accurate and improved their performance in real-world scenarios.

Home equity lending has increasingly turned to AVMs thanks to their high levels of accuracy and affordable price points. As a result, AVMs have become the preferred valuation choice for home equity lenders.

From a regulatory perspective, AVMs have become part of the broader discussion on appraisal bias. Over the past few years, the accuracy and fairness of in-person home appraisals have been called into question. These concerns about appraisal accuracy and human bias have boosted the appeal of AVMs as a supplement to in-person appraisals. By reducing human involvement, AVMs could help to correct for racial bias from appraisers evaluating homes and the conditions in minority neighborhoods. However, AVMs are under the same scrutiny to demonstrate they do not have racial bias built into the core models, as increasing use of AI technology may have the unintended consequence of promoting bias.

Finally, for some time, AVM developers and lenders have believed that there are significant real-world performance variances from model to model, but they haven’t had the ability to test for this. Now, thanks to new methodologies and industry cooperation, companies like AVMetrics and AEI Housing Center have taken AVM testing to a new level.

The New Rules

The joint agencies’ new guidelines, Quality Control Standards for Automated Valuation Models, requires mortgage originators and secondary market issuers “to maintain policies, practices, procedures, and control systems to ensure that automated valuation models used in these transactions adhere to quality control standards designed to:

(a) Ensure a high level of confidence in the estimates produced;
(b) Protect against the manipulation of data;
(c) Avoid conflicts of interest;
(d) Require random sample testing and reviews; and
(e) Comply with applicable nondiscrimination laws.”

Lenders can comply with the new guidelines by conducting their own testing or using third-party providers — like AVMetrics — that test commercially available AVMs.

Newer AVMs, like our Procision^™ AVM Suite, were designed to comply with current AVM guidelines and in anticipation of the new guidelines. To measure accuracy, First American Data and Analytics conducts daily blind testing of the valuations generated by our AVMs using multiple third-party testing providers. In addition, we offer no-cost testing to existing clients and prospects considering our AVMs.

All of our AVMs have forecasted standard deviation scores and a separate confidence score that lenders can use as a measure of accuracy.

Going Beyond 90% PP10

Although some large lenders conduct their own AVM testing, most lenders rely on specialty testing companies, like AVMetrics, that send the same property addresses to various AVM vendors, then collect and compare the valuations from each vendor against a benchmark value. This may be for an arm’s length sales price, an appraisal value, or in some cases for-sale listings.

Ninety percent of the values produced by an AVM should fall within plus or minus 10% of the benchmark value, which is the baseline industry performance target and referred to as 90% PP10.

The problem with this approach, some observers believe, is that the models may already know the sales price benchmark, or at least know the list price, which is generally very close to the ultimate sale price. AVM providers may have a “snap to” list price embedded in their models, which can inflate the actual accuracy of the model.

This past spring, industry-leading think tank AEI tested the performance of five AVMs from different providers, whose names were kept confidential. Each provider was graded on the same randomly drawn 10,000 properties, spanning 34 counties, with valuations tested over the five-year period from 2018 to 2023. AEI looked at five criteria in its evaluation: coverage, predictiveness of a subsequent sales price, springing to the list or sales price, utility as a home price index, and bounciness — how often an AVM spiked (or dropped) by at least 10% and then dropped (or spiked) by at least 5%. It also tested for AVM accuracy by household income and minority share.

When looking at specific categories, the springiness test had the widest variance in scores with some companies getting failing grades. This suggests that certain models may be more “attuned” to sales prices. As AEI noted: “If an AVM is springing to the sales or list price, it diminished the AVMs utility…” Performing better against this criterion “indicated that the AVM performs well at all times, and not just when listings or sales information (for the limited number of properties that transact on a monthly basis) become publicly available,” the report said.

Put another way, an AVM with high ‘springiness’ may not perform as well in real world applications as it does in standard testing.

Enter Blind Testing

For several years, AVMetrics has been developing a blind testing system that it will roll out later this year. Rather than sending the same addresses to various providers each month and getting back their valuations, AVM providers will now value every property in the U.S. — more than 100 million valuations each month — and send this data to AVMetrics. The testing company will ingest this data and then blind test it against future sales and listing prices as they transact. As you would expect, this is a massive undertaking for AVM vendors and AVMetrics, but it will separate the AVMs that test well from those that actually perform well in real-world conditions.

This type of true blind testing reveals how AVM models will perform in real world applications and gives users the information they need to decide when and where to utilize AVMs in their business processes.

First American Data & Analytics has addressed the influence of listing prices on AVM predictions by conducting internal blind testing and offering an AVM solution that is not influenced by recent subject values.

Accelerating the Pace of Innovation

As someone who has been involved in AVM development for more than three decades, I cannot recall a time when there has been so much interest and innovation in AVM performance or testing, nor a time when there has been so much regulatory scrutiny.

In this challenging high interest rate environment, lenders face pressure to lower costs and increase underwriting efficiency to remain competitive, and they are increasingly turning to technology to solve this challenge. The integration of artificial intelligence, machine learning, and blockchain technologies are streamlining processes, reducing costs, and enhancing overall efficiency. Additionally, automated underwriting systems are becoming more sophisticated, enabling faster and more accurate risk assessments. AVMs are being integrated into all these systems, as the need for an accurate collateral valuation remains critical.

The ability of new AVMs to deliver improved accuracy and increased efficiencies through automation will only continue to make these valuable solutions more popular among lenders. But the technology must be adopted along a parallel path with advanced and more widespread testing to continue to improve accuracy and mitigate the potential for bias. We’re glad to be on the forefront, not only in developing this new AVM technology, but also in participating with our industry colleagues in testing that will improve results for both borrowers and lenders.

(Sponsored content includes material submitted independently of the Mortgage Bankers Association and MBA NewsLink and does not connote an MBA endorsement of a specific company, product or service. For more information about sponsored content opportunities, contact Bill Farmakis at bill@jlfarmakis.com or 203/834-8832.)