Users of social media platforms like Facebook, Instagram and TikTok might think they're simply interacting with friends, family and followers, and seeing ads as they go. But according to research from the UBC Sauder School of Business, they're part of constant marketing experiments that are often impossible, even for the companies behind them, to fully comprehend.
For the study, the researchers examined all known published, peer-reviewed studies of the use of A/B testing by Facebook and Google — that is, when different consumers are shown different ads to determine which are most effective — and uncovered significant flaws.
UBC Sauder Associate Professors and study co-authors Dr. Yann Cornil and Dr. David Hardisty say that at any given moment, billions of social media users are being tested to see what they click on, and most importantly for marketers, what they buy. From that, one would think advertisers could tell which messages are effective and which aren't — but it turns out it isn't nearly that simple.
By using Facebook's A/B test tool, researchers can access a massive audience and observe real behaviour — and because the participants are unaware they're part of an experiment, their responses are considered more genuine and reliable.
The problem is that highly complex algorithms decide which consumers will be shown different content and ads; and as a result it's impossible for anyone — even those who created the algorithms — to fully understand why specific consumers have been targeted by an ad, and to determine why some of them decided to click on the ad. According to Dr. Cornil, it comes down to a lack of something called "random assignment" — for example, when experimenters randomly present two different ads to selected groups.
"You can't say that whatever changes you made in your ad are causing an increase in click behaviour, because within each ad there's going to be an algorithm that will select the participants most likely to click on it. If the algorithms are different, it means that there's no real random assignment," he says. "It also means we cannot say for sure that an ad generated a higher click-through rate because creatively it's a better ad. It might be because it's associated with a better algorithm."
What's more, people are often shown ads based on their search history, but if they have already decided on a particular product, and then the algorithm shows them an ad for it, researchers might wrongly conclude the ad led them to buy it.
"It will choose people not just on observable things like age or gender or location that we can easily know, but on unobservable things like past behaviour, interests, and even things that Facebook itself cannot quantify, because they're determined by machine learning and AI," says Dr. Hardisty. The targeted groups might seem similar in some ways, he adds, but the algorithm may have chosen them for completely different reasons.
"It's basically a complicated model that has somehow figured out that some type of person — we don't know what type — is more likely to click. So even if we asked people at Facebook, 'Why was this group of people selected?' they wouldn't know the answer."
So why does this matter? For one, many marketers rely on Facebook A/B testing to determine what to advertise and how; but perhaps even more importantly, different segments of the public can be excluded from important information, which can reinforce divides.
"There's one paper that explains why women are not targeted by ads for STEM (science, technology, engineering and mathematics) education purely because of algorithms," says Dr. Cornil. "Women are more expensive to target on social media, and those algorithms are going to try to generate as many clicks as possible at the lowest cost. So if women are too costly to target for the purpose of STEM education, they are not targets."
What's more, the algorithms then reinforce what's working and what isn't, so if women aren't clicking on particular ads, they will be even less exposed to them.
While the UBC study — titled On the Persistent Mischaracterization of Google and Facebook A/B Tests: How to Conduct and Report Online Platform Studies — focuses on Facebook and Google, the researchers say that all of the major social media platforms, from Instagram to TikTok, employ similar practices.
They are also ubiquitous. At a conference a Facebook employee once told Dr. Hardisty that at any given time, every Facebook user is an unwitting participant in an average of 10 different experiments. With the advent of AI-generated content and ads, that number is almost certainly on the rise.
As a result, Dr. Hardisty and Dr. Cornil — who co-authored the study with Dr. Johannes Boegershausen of Erasmus University and Dr. Shangwen Yi of Hong Kong Polytechnic University — warn that marketers should beware of reading too much into the results of Facebook A/B testing.
"If you have an ad that's going crazy and getting a lot more clicks, it may just be that Facebook successfully identified a small, particular group of people that really like it," says Dr. Hardisty. "And if you change your whole product line or campaign to match that, it might actually be alienating to most people. So you have to be very careful not to draw broader lessons from one Facebook study."
In fact, the algorithms are so complex and precise, adds Dr. Cornil, social media platforms can "micro-target" people right down to the individual level. "It's selecting the best possible ads for a specific segment — and the segment isn't even a group of people. With all the data we have about consumers, the segment is one," he says. "And it all happens in a black box. The advertiser doesn't know, but the machine knows. AI knows."
