Press "Enter" to skip to content

Launching a study on Mechanical Turk

FindingFive works seamlessly with Amazon’s Mechanical Turk. FindingFive offers an easier way to program your study and simultaneously take advantage of the participant pool and easy payment procedure that Mechanical Turk offers. We’ve done all the nitty-gritty backend work for you, so you can launch your FindingFive experiment on MTurk by following these easy steps:

  1. Make sure you have an Amazon Web Services (AWS) account
  2. Create a new Identity and Account Management (IAM) user
  3. Enter your IAM user access key and secret on FindingFive
  4. Launch and manage your study on MTurk from within FindingFive

Also be sure to check out our post detailing some best practices when running a study on Mechanical Turk!

Make sure you have an Amazon Web Services account

Go to the homepage of Amazon Web Services, and click on “Sign In to the Console”. On the next page, it should ask you to create or sign up for an AWS account.

We recommend that you sign up for an account using your university email address. If you are signing up for an account on behalf of your lab, you might want to use the lab email address.

Create a new Identity and Account Management (IAM) user

Once you have an AWS account, you’ll need to add a new IAM user to your account. This will generate access keys that allow FindingFive to do all the behind the scenes work for you on MTurk.

When you log into your AWS account, you should see a drop-down menu option at the top of the screen called “Services.” Type “IAM” into the search bar that appears at the top of this drop-down menu and hit return (or click “IAM Manage access to AWS resources” when it appears).

Add a new user

Click “Users” under the “Access management” option in the left hand menu on the screen.

Then click the “Add user” button at the top of the screen.

Set up the user’s access

  • Name the user (e.g., “FindingFive”, since we are the authorized user on your AWS account)
  • Select the box labelled “Programmatic Access” under “Select AWS access type”
  • Click the “Next: Permissions” button

Give permission for this user to access (only) Amazon Mechanical Turk

  • Click “Attach existing policies directly”
  • Type “Turk” into the policy type search bar
  • Select the box for “AmazonMechanicalTurkFullAccess”
  • Click the “Next: Tags” button
    • NOTE. Tags are optional. Click on the “Next” button until you get to the “Review” page.

Review your new user

  • Make sure your “AWS access type” says “Programmatic Access – with an access key”
  • Ensure that “AmazonMechanicalTurkFullAccess” appears under the “Permissions summary”
  • Click “Create user”

Now you should see your user’s “Access key ID” and “Secret access key”. Download the csv, keep this window open, or copy and paste both the key and secret into a local document on your computer.

Enter your IAM user keys on FindingFive

Now that you have a MTurk-only IAM user for your AWS account, you’ll need to tell FindingFive how to access that user.

First, log into FindingFive here.

Then go to your profile by clicking the “Account” menu at top right of your screen, and selecting “Profile”.

Click “MTurk integration” on the left hand side of the screen (or scroll down to the bottom) and enter your IAM user’s access and secret keys.

Click “Save changes” on the left hand side of the screen and you’re all set up to use FindingFive integrated with MTurk!

Launch your study through FindingFive

Once you’ve programmed your experiment and are ready to create a session to be launched on MTurk, head to your study’s “Session” tab, and you’ll be walked through our MTurk session wizard. (You might also want to check out our blog post on best practices when launching a study on MTurk.)

Step 1: Selecting your platform

You see two icons pop up: the FindingFive logo and the Amazon logo. To launch your study on Mechanical Turk, click the “a” icon. This will bring up a pull-down menu with three options: Sandbox, Production, and Link Only:

  • The Production option is what you should select if you’re 100% ready for collecting data from Turkers. In other words, Production allows you to start collecting data from actual participants over the Internet through Amazon’s Mechanical Turk marketplace.
  • Before putting your study into Production, we first recommend trying it out in the MTurk Sandbox. The Sandbox is a simulated environment where you can test all aspects of your study and experience it in the same way your participants will.
  • The last option, Link Only, will provide you with a direct link to your FindingFive study that can be plugged into your Mechanical Turk frame window or redirected to. We do not recommend using this option for most researchers because it means that FindingFive will not publish your study for you. If you choose to get a link from FindingFive and embed it yourself in your Turk study, FindingFive will not be able to set up any participant control for you, and most of the parameters you’ll define in the next few steps will not be automatically linked to the properties of your MTurk study. Bottom line: If your session on MTurk goes wrong in any way, FindingFive will not be able to provide any kind of help!

Step 2: Defining basic study settings

In Step 2 of the MTurk session wizard, you’ll be able to define some basic settings for your study. Keep in mind that the first three fields are the pieces of study information that your MTurk participants will see when deciding on the tasks they want to complete, so use best practices when deciding on these settings. The Session name field allows you to optionally assign a name for you to distinguish this study session from others in your session list. Participants will not be able to see this setting; it is solely for researcher use. The HIT Frame height field allows you to determine the size of your study window when it is viewed by a participant on MTurk. (Note that HIT stands for Human Intelligence Task, which is a fancy way of saying “the task a worker [a.k.a. participant] will be asked to complete.”) We recommend starting with a height of 600, but you might want to play with this setting in the Sandbox if you have a lot of size-dependent images or videos in your study.

Step 3: Initial participant settings

Next, you’ll define how many participants you want to enroll in your study, the estimated duration of your study, and some basic restrictions on your participant pool.

In particular, you can determine how long your study will remain active before a participant “times out” (the default setting is equal to the estimated duration of your study), and some simple restrictions on the participants who are eligible to enroll in your study (like blocking a participant who has failed to complete your study in the past, and setting the minimum approval rate for eligible participants). The highest quality MTurk workers will have approval ratings above 97% and will have been approved for hundreds of HITs. Trying to recruit high-quality Turkers will likely result in a lower drop-out rate and higher quality data, but being too restrictive may reduce your ability to recruit enough participants. Your mileage may vary!

You can also restrict the geographical region of your participants. This can be set on the next screen:

Step 4: Setting prerequisites and anti-prerequisites

If you have other studies or sessions running on MTurk, you can restrict your participant pool based on whether they have completed these tasks. If you see empty text boxes in this step, FindingFive hasn’t launched any other study sessions for you or your collaborators on MTurk, so you can’t restrict participation based on this setting.

Step 5: Setting up compensation

Define how much you’ll be paying participants on the next screen. (Worried about what might happen to participants who experience issues with your study? Check out our tutorial on compensation HITs!)

Step 6: Select the appropriate consent form

You’ve probably already uploaded a consent form for your study to your “Resources” section of your Researcher account. To check which consent forms you have on file (or to upload new ones), click on the “Research” tab at the top of your screen, then select “Resources.” Click on “Documents” on the left side menu to see all of the documents you’ve uploaded to your profile and upload new ones.

You will see a list of all of your consent forms in the pull-down menu in Step 6 of the MTurk session launch wizard. Select the consent form you want to use for this session (or upload a new one), and move onto the next step.

Step 7: Decide whether you’d like to share information with FindingFive

FindingFive is always looking for ways to improve the experience of our researchers and participants. If you like, you can decide to share your study information with us to help us grown and improve our platform. If you decide to share your information with us, we will uphold the highest level of security and confidentiality, as we always do! Note that you are not required to agree to any of the the conditions in this step. You can still launch your study regardless of your responses in this step.

Step 8: Schedule your session!

All FindingFive researchers have the option to schedule their MTurk session launch for some point in the next 30 days. Launch date availability is determined by how many other researchers are trying to launch at the same time, and the current load on our servers. In general, sessions are scheduled on a first-come, first-serve basis as determined by server capacity. (If you’re a premium user of FindingFive, you also have the option to launch your session immediately on Mechanical Turk.) Identify how long you want your study to remain active on Turk once it launches and type this number into the text box at the bottom of the window. Then select the date on the calendar that corresponds to your desired schedule launch date.

Click “Schedule new session,” and voila! Your study session is scheduled to be launched on Amazon’s Mechanical Turk.


Feedback

We would love to hear about your feedback on FindingFive’s integration with Mechanical Turk! Email us at researcher.help@findingfive.com.

Comments are closed.