Recurring Turking: Conducting Daily Task Studies on Mechanical Turk

by   Henry Turner, et al.

In this paper, we present our system design for conducting recurring daily studies on Amazon Mechanical Turk. We implement this system to conduct a study into touch dynamics, and present our experiences, challenges and lessons learned from doing so. Study participants installed our application on their Apple iOS phones and completed two tasks daily for 31 days. Each task involves performing a series of scrolling or swiping gestures, from which behavioral information such as movement speed or pressure is extracted. Taking place over a time period of 31 days, our study utilized a self-contained app which workers used to complete daily tasks without requiring extra HITs. This differs somewhat from the typical rapid completion of one-off tasks on Amazon Mechanical Turk. This atypical use of the platform prompted us to study aspects related to long-term user retention over the study period: payment schedule (amount and structure over time), regular notifications, payment satisfaction and overall satisfaction. We also investigate the specific concern of reconciling informed consent with workers' desire to complete tasks quickly. We find that using the Mechanical Turk platform in this way leads to data of comparable quality to that of lab based studies, and that our study design choices show a statistically significant effect in keeping workers engaged.


page 7

page 9

page 11


A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk

A growing number of people are working as part of on-line crowd work, wh...

A Glimpse Far into the Future: Understanding Long-term Crowd Worker Quality

Microtask crowdsourcing is increasingly critical to the creation of extr...

Race, Gender and Beauty: The Effect of Information Provision on Online Hiring Biases

We conduct a study of hiring bias on a simulation platform where we ask ...

A Survey of NLP-Related Crowdsourcing HITs: what works and what does not

Crowdsourcing requesters on Amazon Mechanical Turk (AMT) have raised que...

The Hidden Cost of Using Amazon Mechanical Turk for Research

This work shares unexpected findings obtained from the use of the Amazon...

Satyam: Democratizing Groundtruth for Machine Vision

The democratization of machine learning (ML) has led to ML-based machine...

Indicators of retention in remote digital health studies: A cross-study evaluation of 100,000 participants

Digital technologies such as smartphones are transforming the way scient...

Please sign up or login with your details

Forgot password? Click here to reset