For researchers who run their FindingFive studies on the Mechanical Turk platform, you may have occasionally got more subjects than requested. For example, the researcher may have requested 10 participants, but at the end of a session, FindingFive reports 11 completed your study, and Amazon only has a record of 10 workers who completed your HIT. There’s one extra mechanical worker that seems to have come out of nowhere.
The Root Behavioral Cause
The underlying reason why this happens is, interestingly, behavioral. We have discovered that some MTurk workers tend to accept a HIT right away but will not work on it till much later. By the time they work on the HIT again, the HIT is about to expire as the assignment duration is almost over. Some researchers have tried to address this issue by setting a long assignment duration. Unfortunately, that backfires and encourages more of such “squatting” behavior. Once workers see the long assignment duration, they procrastinate even more.
The Technical Cause (on Amazon’s part)
The squatting behavior alone would not have caused the extra participants. Ideally, the assignment duration of a HIT should work like the “security timeout” feature on most banking websites. That is, when a HIT reaches its assignment duration, Mechanical Turk should simply close the HIT on the worker, in the same way that a banking website automatically logs you out after a certain amount of time of inaction. In that case, the worker would fail to continue, and both FindingFive and Amazon would register a worker who abandons the HIT.
But Mechanical Turk does no such thing! Instead, when a worker spends more time than the assignment duration, the HIT window remains open and functional. Only the tiny “Time Allotted” field, in the top right corner of the HIT’s page, is highlighted in red. The worker can continue to work on the FindingFive study because the web page is still operational. When they reach the end of the study, at which point FindingFive registers a completed participant, they discover that they can’t submit the HIT, since the HIT has timed out. Mechanical Turk registers an abandoned participant and another worker will accept the HIT to replace this abandoned participant.
That’s where the extra participants come from.
Our New Solution
We decide to address this issue on our end (well, would Amazon listen to our suggestions anyway?). We have now implemented a study timeout function that syncs with a HIT’s assignment duration. FindingFive is now aware of the exact time when a worker accepts a HIT, and only allows the worker to work on the study within the same time limit as specified by the HIT’s assignment duration. If the worker procrastinates, the study will time out.
This active timeout function is crucial. With this new feature, FindingFive will now close the study on workers who take too long to finish a study. As a result, there won’t be any differences between FindingFive and Mechanical Turk’s head counts! (Well, at least we hope)
Here’s what the worker will see if the study times out.
The notification bar at the top of the study window is persistent. It starts counting down the moment the worker accepts the HIT. As it comes closer to the HIT’s assignment duration, it starts flashing red to alert the worker, which is super cool. 😉
The solution implemented by FF is robust to the situation where the worker may close the accepted HIT and reopens it later. The assignment duration calculation is always anchored toward the first time the HIT is accepted or opened.
Final Word
How do you like this feature? We hope that you guys appreciate our continuing effort to make it easier for researchers to run studies. If there’s any questions, feel free to contact us!
Comments are closed.