WHAT IS AN EXPERIMENT?
The Merriam-Webster dictionary defines an experiment as “an operation or procedure carried out under controlled conditions in order to discover an unknown effect or law, to test or establish a hypothesis, or to illustrate a known law.”
The main goals of experimentation are twofold. Firstly, it allows us to compare two or more variants of a product, method, policy or theory, in order to determine which performs better. Secondly, it allows us to establish causality – whether doing X (for example, having a larger “Subscribe” button) will result in Y (an increase in subscriptions).
Imagine that a primary school recently introduced tablets to help primary school students expand their vocabulary. Take a look at Figure 1, which shows the vocabulary size of students in this hypothetical situation:
FIGURE 1.
The dotted line represents vocabulary size before the introduction of tablets, and the solid line represents vocabulary size after introducing tablets.
Looking at the larger vocabulary size after the introduction of tablets, one might be tempted to conclude that tablets do indeed improve learning. However, it is not so simple. The larger vocabulary size after the introduction of tablets could have come from a wide variety of sources. For example, perhaps the introduction of tablets coincided with a new lesson in which students were learning easier words. Or, it could simply just be due to the natural improvement in learning that a child experiences with age.
To circumvent the problem of external factors exerting an influence on our question of interest (in this case, whether tablets improve vocabulary size), we can conduct an experiment. For centuries, scientists and academics have used experiments to uncover causal relationships and test theories and hypotheses. In our hypothetical example, we could split our students into two groups – one group will receive tablets, and the other will not. Figure 2 shows some hypothetical data from this experiment.
Now, we have greater confidence that tablets do indeed improve vocabulary size. Even though vocabulary size was increasing in both groups, the group with the tablets have had a much larger increase, suggesting that tablets are indeed advantageous in boosting vocabulary learning.
However, consider a different scenario shown in Figure 3.
Here, there is no meaningful difference between the vocabulary scores of the two groups after tablets were introduced. If we had only relied on observing the scores before and after the introduction of tablets without conducting an experiment, we would have mistakenly concluded that the increase in vocabulary score was due to the tablets, when in fact it could have just been due to random factors or to the natural increase in a child’s vocabulary size throughout the semester.
Herein lies the power of experiments – they allow us to confidently make conclusions about the effectiveness of an intervention.
Although you might not realise this, if you are a user of technology (mobile apps, social media, e-commerce, the internet, and so on), you are almost certainly a participant in multiple experiments.
TECHNOLOGY, BIG DATA AND EXPERIMENTATION
Recent advances in computing power, data storage and technology in general have made it much easier, more cost-effective, and less time- consuming to conduct experiments. Indeed, the number and type of experiments that have been conducted by both tech companies and research institutes has undergone an explosive growth in the last five years. Many companies are constantly running consumer-facing experiments, testing different versions of their apps and features in order to determine which one performs the best – in terms of usability, retaining and gaining users, and, of course, profit. Tech companies such as Google, Amazon, Grab, Meta (Facebook), and Amazon have large in-house experimentation teams and platforms that allow them to quickly and effectively experiment on any product and feature launches and changes in order to make informed, data-driven decisions. Annually, each of these companies conducts more than 10,000 experiments; many of these have hundreds of thousands, if not millions, of users.
Two different versions of the Facebook page design
The new page layout (right) was designed to be less complex and to highlight key information. Facebook ran experiments to determine whether the new layout was successful in achieving this. Source: www.techcrunch.com
In the world of academia and research, many companies such as Amazon MTurk, Prolific and Pavlova were started to offer the ability to conduct online experiments, allowing them to collect data more quickly compared to the traditional method of lab-based experiments. In many cases, experiments that would have taken months of data gathering can now be completed in days or even hours.
Although you might not realise this, if you are a user of technology (mobile apps, social media, e-commerce, the internet, and so on), you are almost certainly a participant in multiple experiments. Open up your favourite app and compare it to someone else’s. Chances are, the interface, ads, or even the features you see will be different.
This is experimentation in action – companies are constantly testing different versions to find out which one performs the best. Does changing the size of the “Subscribe” button increase the number of subscribers of a newsletter? Which email title leads to a higher open rate? Will customers be more likely to make a purchase if we show them how many times an item has been purchased in the last hour? These are some of the questions that can be answered through experiments, rather than relying on ‘gut feel’ or intuition, or the nebulous concept of ‘past experience’.
ETHICS AND CONSENT
The ubiquity of experiments in our everyday lives has raised some important questions about consent and ethics. The majority of users of technology are probably in the dark about what kind of experiments they are part of – or even about the simple fact that they have been a part of multiple experiments. Even though we might not explicitly give consent, the terms and conditions that we (often blindly) accept already grant companies the right to conduct experiments (among many other things).
Thankfully, tech companies conduct experiments generally with the aim of improving user experience, increasing customer satisfaction, and bringing value to their customers. Of course, it is undeniable that the bottom line of most companies is profit – but it is also understood that the customer always comes first and that there would be no profit without customers.
Facebook's emotion experiment
For one week in 2012, Facebook’s data scientists manipulated the News Feeds of 689,003 users, removing either all of the positive posts or all of the negative posts, to see how it affected users’ moods. This raises deep concerns about the ethics of emotional manipulation and guidelines for informed consent of human subjects. Source: iStock
If experiments are inevitable, what can, and should, companies do to prevent harm to their customers in the course of their being conducted?
However, there have been unfortunate instances in which companies crossed ethical boundaries in their experimentation. In 2014, it was revealed that Facebook had run an experiment that breached ethical guidelines for informed consent. This was especially grievous – the experiment brought potential harm to their users because it was designed to change their mood and even induce some emotional distress.
Biomedical and behavioural research involving human participants in academia, healthcare and government is governed by a set of research ethics that is approved by an Institutional Review Board (IRB). One of the core principles of experiments involving human participants is that the risk of harm from the experiment should not be “greater than those ordinarily encountered in daily life”. In the Facebook experiment, a group of users were purposefully shown more emotionally negative content on their Facebook News Feed (“negative group”), while another group were shown more emotionally positive content (“positive group”).
In essence, Facebook’s experiment manipulated the mood of their users. The researchers found that those in the negative group began posting more negative updates, while those in the positive group began posting more positive updates. This effect was large enough to be published in a top-tier scientific journal, PNAS (Proceedings of the National Academy of Sciences of the United States of America), even though Facebook claimed that the effect was not large – after they were questioned on the ethics of the experiment.
If experiments are inevitable, what can, and should, companies do to prevent harm to their customers in the course of their being conducted? Companies could consider allowing users to opt out of experiments, similar to how users are allowed to opt out of data collection. However, this could ultimately have an impact on the end user themselves, since they would then not have an experience that is optimised by experiments and data.
At the very least, data scientists and researchers who design and implement experiments should receive training in ethics. Currently, researchers in academia, healthcare and government organisations receive mandatory training in ethics and compliance, especially if they work with human populations. While it might be a mere first step, it is at least one in the right direction towards the protection of consumer rights.
The Facebook watchdog
Maria Ressa was awarded the 2021 Nobel Peace Prize jointly with Dmitry Muratov for “their efforts to safeguard freedom of expression” amid growing concerns over curbs on free speech worldwide. She is calling for technology companies to face greater regulation globally to curb disinformation on social media. “Without facts, you can't have truth. Without truth, you can't have trust. Without trust, we have no shared reality, no democracy, and it becomes impossible to deal with our world's existential problems: climate, coronavirus, the battle for truth.” Photo: PACIFIC PRESS / Alamy Live News
DR GAVIN NG
Dr Gavin Ng obtained his Ph.D. in Experimental Psychology from the University of Illinois at Urbana-Champaign. There, he conducted controlled experiments to understand the human visual system – the processes and the computations that the brain carries out to perceive objects (e.g., their shape, colour, and form), deploy attention and eye movements, and search for things in the environment. He recently relocated back home to Singapore and is currently a Senior Data Scientist at Grab, where he continues to design, run and analyse experiments, albeit with larger datasets and in a less controlled setting with noisier data.
DECEMBER 2021 | ISSUE 9
Tomorrow's Technology Today
