Facebook knew for years ad reach estimates were based on ‘wrong data’ but blocked fixes over revenue impact, per court filing

Some more internal emails Facebook really doesn’t want you to see: Turns out in 2017 COO Sheryl Sandberg had already known for years there were problems with a free ad planning tool the company offers to marketeers to display estimates of how many people campaigns running on its platform may reach, per newly unsealed court documents.

The filing also reveals that a Facebook product manager for the “potential reach” tool warned the company was making revenue it “should never have” off of “wrong data”.

The unsealed documents pertain to a U.S. class action lawsuit, filed in 2018, which alleges that Facebook deceived advertisers by knowingly including fake and duplicate accounts in a “potential reach” metric.

Facebook denies the claim but has acknowledged accuracy issues with the “potential reach” metric as far back as 2016 — and also changed how it worked in 2019.

While the litigants have continued to accuse Facebook of continuing to misrepresent the ad reach estimate in updates to their 2018 complaint.

Redacted documents from the lawsuit, reported by the WSJ last year, included the awkward detail that a Facebook employee had asked “how long can we get away with the reach overestimation?”

But sections of the filing pertaining to Sandberg and other Facebook executives were redacted.

Newly unsealed documents from the suit — which we’ve reviewed — now reveal that in fall 2017 Sandberg “acknowledged in an internal email she had known about problems with Potential Reach for years”.

They also show Facebook repeatedly rejected internal proposals to fix the issue of fake and duplicate accounts inflating the estimates its platform showed to advertisers of the number of people who could see their ads — citing impact on revenue as a reason not to act.

In early 2018 Facebook estimated that removing duplicate accounts would cause a 10% drop in potential reach, per the unsealed filing. While Facebook management rejected an employee’s suggestion to change the language the tool showed to advertisers, declining to swap out the words “people” and “reach” for the (more accurate) term “accounts” — on the grounds that “people-based marketing was core to Facebook’s value proposition”.

The filing also reveals that a product manager for “potential reach”, Yaron Fidler, proposed a fix for the tool that would have decreased its numbers. His proposal was rejected by Facebook’s metrics leadership on the grounds that it would have a “significant” impact on the company’s revenue — to which Fidler responded: “It’s revenue we should have never made given the fact it’s based on wrong data.”

In 2016, when Facebook published an update on metrics — a few weeks after publicly disclosing it had been over-inflating average video view times, as it sought to regain advertiser trust in its reporting tools — the tech giant also announced a new channel for “regular information on metrics enhancements”, called Metrics FYI.

This is where it made the aforementioned fuzzy disclosure of accuracy issues with “potential reach” — writing then that it was “improving our methodology for sampling and extrapolating potential audience sizes” to “help to provide a more accurate estimate for a given target audience and to better account for audiences across multiple platforms (Facebook, Instagram and Audience Network)”.

“In most cases, advertisers should expect to see less than a 10% change (increase or decrease) in the audience sizes shown in the tool,” it added at the time.

However, the December 2016 blog post did not go into any detail about the nature of the accuracy problems Facebook thought needed improving — reading more like another classic slice of Facebook crisis PR.

The class action suit, meanwhile, alleges that rather than accepting internal proposals to fix the accuracy problems of “potential reach”, Facebook instead “developed talking points to deflect from the truth”.

The tech giant did announce some changes to the ad tool in March 2019 — when it said an advertiser’s campaign’s estimated potential reach “is now based on how many people have been shown an ad on a Facebook Product in the past 30 days who match your desired audience and placement criteria” (versus the estimates being previously based on “people who were active users in the past 30 days”).

But the litigants argue that the changes to the tool which displays an estimate to advertisers as they are beginning to create a campaign — and therefore when they’re deciding/considering whether/how much money to spend with Facebook — do not fully fix the issue of the metric not corresponding to the potential audience of people who could see the ad on Facebook.

An analyst report back in 2017 showed that Facebook’s ad platform claimed to reach millions more users among specific age groups in the U.S. than official census data indicated reside in the country.

At the time the company said the audience reach estimates “are based on a number of factors, including Facebook user behaviors, user demographics, location data from devices, and other factors”, per the WSJ, and claimed they are “not designed to match population or census estimates”. Facebook added then that it is “always working to improve our estimates”.

Asked about the latest batch of unsealed court documents pertaining to the lawsuit — including the revelation that Facebook’s COO had told staff she knew about “problems” with the ad tool “for years” as far back as fall 2017 — Facebook sent us this statement, attributed to a spokesperson: “These allegations are without merit and we will defend ourselves vigorously.”

Problems with self-reporting ad metrics have been a recurring theme for Facebook.

Last year the tech giant disclosed yet another issue on this front — saying its “conversion lift” ad tool had a code error that meant it had miscalculated the number of sales derived from ad impressions for a number of advertisers.

That “technical problem” with Facebook’s internal calculation of the efficacy of third parties’ ad campaigns meant advertisers saw skewed data, which they may have used to determine how much to spend on its platform.