The Blog.

Scope is not enough - what you want is engagement.

One of the problems cited for web-based experiments is the drop out. Myself, I think that data gained from a colorful but self-selecting volunteering population is in a way still more valuable than data gained from an unmotivated pool of psychology students who participate simply because they will not receive course credit if they don't.

But drop out does happen and it is a problem, and as someone looking to maximize the scope of your research, you have to deal with it, especially since you know it's selective and might skew your data. You can always resort to analyzing and describing the sample that dropped out after submitting demographic information, but this is only sufficient to describe the problem and discuss it as a shortcoming, it is not a solution to mitigate the issue. You can also incentivise participation by offering rewards, but this obviously takes funds, and is subject to ethical inquiries.

You could also motivate people to participate just for the sake of it, and what better context to do it than the web, where you already have access to a huge pool of people. This isn't quite as impossible as it sounds, but it requires a change of attitude. The first consideration is not to think of yourself as a researcher.

Congratulations, you're now a startup! You're the product owner, the developer, and you do the sales, marketing and customer support. Think of your experiment as a product. Participants are your users - quite literally. In the industry, you would approach them with your product or service, and they would assess whether it is worth their investment. With an experiment, you approach them with a request for participation, and they assess whether it is worth their time. You have to consider possible use cases for your product - whom it might benefit, who is your primary audience, what marginal strata exists that might be interested in your work. These are your possible participant pools. If you need a good cross section of the entire population, think of the different ways you approach different strata - different subpopulations need a different approach, through different channels, using different language. Here is the first difference from production: the underlying message needs to be the same, the procedure needs to be the same, and the results need to be comparable, so you only get to customize relatively cosmetic features. This can still carry you surprisingly far. Get creative.

Minimizing 'unnecessary' drop out is a question of user experience design, and user experience design is not only about usability, but also about engagement. It can be considered very well through Herzberg's motivation-hygiene theory. To explain Herzberg very shortly, he argued that task satisfaction and dissatisfaction function separately of each other. Hygiene factors are things that cause no satisfaction, but do cause dissatisfaction in their absence. For example, having a bug-free task with good instructions is a hygiene factor. Motivators on the other hand are things that give positive satisfaction - things such as a beautiful UI and extrinsic rewards. In the next two blog posts, we will discuss how to tweak the hygienic factors and motivators in a web-experiment to minimize participant drop out and to maximize engagement.