Zillow’s home-buying debacle shows how hard it is to use AI to value real estate

By Rachel Metz, CNN Business

In February, Zillow appeared so confident in its ability to use artificial intelligence to estimate the value of homes that it announced a new option: for certain homes, its so-called “Zestimate” would also represent an initial cash offer from the company to purchase the property.

The move, touted by a company exec at the time as “an exciting advancement,” was intended to streamline the process for homeowners considering selling to Zillow as part of its home-flipping business. Zillow promoted this option as a way to make it convenient to sell a home while minimizing interactions with others during the pandemic. Just eight months later, however, the company is shutting down that business, Zillow Offers, entirely.

The decision, announced last week, marks a stunning defeat for Zillow. The real estate listing company took a $304 million inventory write-down in the third quarter, which it blamed on having recently purchased homes for prices that are higher than it thinks it can sell them. The company saw its stock plunge and it now plans to cut 2,000 jobs, or 25% of its staff.

The fallout from this business venture doesn’t just point to the challenges in buying and selling homes for profit, however. It also highlights how hard it is to use AI to help make expensive, real-world decisions, particularly in an ever-changing market that can be hard to predict months or even weeks out, and with prices that can be based as much on feel as on clear data points. Zillow CEO and cofounder Rich Barton explained the shuttering of Zillow Offers by citing “unpredictability in forecasting home prices” that “far exceeds” what the company had expected.

The “iBuyer” model used by Zillow and other other real estate companies entails purchasing homes directly from sellers and then re-listing them after doing minor work. For Zillow, one of the first steps in its decision to purchase any home is the “Zestimate” — a machine-learning-assisted estimate of a home’s market value that is calculated by taking into account oodles of data about the property gathered from sources including tax and property records, homeowner-submitted details such as the addition of a bathroom or bedroom, and pictures of the house. Rival platforms such as Redfin have their own estimates that take similar data into account.

“The Zestimate, facts you provided, and comparable homes nearby are used to calculate an estimated sale price,” Zillow explained on its Zillow Offers webpage to homeowners who may be interested in selling their property to the company. (The page now notes the company is “winding down” the service, and isn’t making new offers on homes.) After that estimate, the page explained, Zillow conducts an in-person evaluation of a property, determines the amount it deems necessary for repairs before it could resell the house, and then makes a final offer. Zillow has bought tens of thousands of homes since the launch of Zillow Offers, but has sold many fewer than it snapped up: according to its quarterly results, it purchased 27,000 homes from April 2018 through September 2021, and sold nearly 17,000.

Zillow declined a request for an interview with Krishna Rao, the company’s vice president of analytics. In a statement, Zillow spokesperson Viet Shelton told CNN Business the company used the Zestimate for Zillow Offers “the same way we encourage the public to use it: as a starting point.”

“The challenge we faced in Zillow Offers was the ability to accurately forecast the future price of inventory three to six months out, in a market where there were larger and more rapid changes in home values than ever before,” Shelton said.

Indeed, since Zillow entered the home-flipping business in 2018, real estate markets have changed in wildly unpredictable ways. The pandemic led to a temporary housing market freeze, followed by a supply and demand imbalance that caused an unprecedented rise in home prices. This may only have complicated the company’s decision to include the Zestimate — which Zillow points out is not an appraisal, but a “computer-generated estimate of the value of the home today, given the available data” — as part of the Zillow Offers process in more than 20 cities.

Artificial intelligence can look at far more information, far more quickly, than a single human could when considering a fair price for a home, weighing factors like comparable home sales in an area, how many people are looking in a specific neighborhood and so on. Still, “you can have a real estate agent look at a house and in one second pick out one critical factor of the valuation that just doesn’t exist as ones and zeroes in any database,” said Mike DelPrete, a real estate technology strategist and scholar-in-residence at the University of Colorado Boulder.

A key part of Zillow

The Zestimate has been a key part of Zillow’s brand since the company first launched its website in 2006. The term is featured prominently on millions of Zillow’s home listings; it’s trademarked by the company; and it’s mentioned 61 times in its IPO paperwork from 2011.

“Three times a week, we create more than 500,000 unique valuation models, built atop 3.2 terabytes of data, to generate current Zestimates on more than 70 million US homes,” the company wrote in a securities filing in 2011. More than 10 years later, the company publishes Zestimates for more than 100 million US homes.

If you’re looking up homes on Zillow’s website or app, the Zestimate is featured prominently in each listing, whether the home is for sale or not. If the house is currently for sale, a red dot is shown next to the words “House for sale,” and the Zestimate, if it’s available for that home, will appear on the same line.

Though the company points out that the Zestimate is not a home appraisal, the feature’s accuracy has been called into question over the years. For example, it became the subject of a lawsuit brought by homeowners in 2017. (That suit was dismissed.)

Zillow has spent years improving the Zestimate, going so far as to run a multi-year data science competition to improve the accuracy of the algorithm behind it. The company awarded a three-person team the $1 million prize in early 2019.

The Zestimate currently has a median error rate of 1.9% for homes that are on the market, Shelton said, meaning Zillow’s estimates for half the homes on the market come within 1.9% of the actual selling price. That percentage of error is much higher — 6.9%, according to Shelton — for off-market homes. Being off by as little as 1.9% on a property with a Zestimate of $500,000 is still nearly $10,000; that figure multiplies over many, many homes in different cities across the United States.

An art, not just a science

It’s one thing to build a model on a website that’s often reasonably accurate. It’s another to then try to use that model in the real world to make very costly bets — and do so at scale, according to Nima Shahbazi, a member of the team that won the Zestimate algorithm competition and CEO of Mindle.AI, which helps companies use AI to make predictions. For instance, if any homes Zillow purchased had hidden problems — such as a missed crack in the foundation — the Zestimate would not be able to predict those issues, he said.

“There are many different parts between a very decent model and deploying the model into production that can go wrong,” he said.

Zillow was using the Zestimate to help it make purchasing decisions for homes it hoped to make a profit off of over time. But Nikhil Malik, an assistant professor of marketing at the University of Southern California, said algorithms tend to be good at making fine-grained, short-term predictions, such as for predicting stock prices a second in advance. But there simply isn’t enough data for an algorithm to learn about longer busts and booms, according to Malik, who researches algorithmic pricing and has studied the Zestimate in particular.

There are also many unquantifiable aspects of putting a price tag on a home, DelPrete noted, such as the value of living in the same neighborhood you grew up in or down the street from your parents. These can vary from person to person, which makes it even harder to outsource a home valuation process to a computer.

“It’s a good tool for what it is,” DelPrete said of the Zestimate, but it’s a mistake to think it can be used to accurately predict house prices now or in the future. He sees it as “almost a toy,” meant more for piquing your curiosity when looking up your home or your neighbor’s home online.

“If you want to do iBuying and you’re going to make thousands of offers every day you have to be really good at valuing homes, not only today but three to six months in the future,” he said. “And that’s an art and a science.”

— CNN’s Anna Bahney contributed to this report.

The-CNN-Wire
™ & © 2021 Cable News Network, Inc., a WarnerMedia Company. All rights reserved.