Michael Dalili – TARG Blog

Research doesn’t just happen in the lab anymore: Mechanical Turk, Prolific Academic, and online testing.

Posted on March 20, 2015 by meryemgrabski

Over the years, from assessment to analysis, research has steadily shifted from paper to PC. The modern researcher has an ever-growing array of computer-based and online tools at their disposal for everything from data collection to live-streaming presentations of their work. While shifting to computer- or web-based platforms is easier for some areas of research than others, this has proven to work especially well in psychology. These platforms can be used for anything from simply hosting an online version of a questionnaire, to recruiting and testing participants on cognitive tasks. Throughout the course of my PhD, I have increasingly used online platforms for multiple purposes, ranging from participants completing questionnaires online on Bristol Online Survey, to recruiting participants using Amazon Mechanical Turk and completing a task hosted on the Xperiment platform. And I’m not alone! While it’s impossible to estimate just how many researchers are using computer- and web-based platforms to conduct their experiments, we have a better idea of how many researchers are using online crowdsourcing platforms such as Mechanical Turk and Prolific Academic for study recruitment. Spoiler alert: It’s A LOT! In this blog post I will describe these two platforms and give an account of my experiences using them for online testing.

Amazon Mechanical Turk, or MTurk for short, is the leading online crowdsourcing platform. Described as an Internet marketplace for work that requires human intelligence, MTurk was publicly launched in 2005, having previously been used internally to find duplicates among Amazon’s product webpages. It works as follows: workers (more commonly known as “Turkers”), who are individuals who have registered on the service, complete Human Intelligence Tasks (known as HITs) created by Requestors, who approve the completed HIT and compensate the Workers. Prior to accepting HITs, Workers are presented with information about the task, the duration of the task, and the amount of compensation they will be awarded upon successfully completing the task. Right now there are over 280,000 HITs available, ranging widely in terms of the type and duration of task as well as compensation. Amazon claims its Workers number over 500,000 ranging from 190 countries. They can be further sub-divided into “Master Categories”, who are described by Amazon as being “an elite group of Workers who have demonstrated superior performance while completing thousands of HITs across the Marketplace”. At time of writing, there are close to 22,000 Master Workers, with about 3,800 Categorization Masters and over 4,500 Photo Moderation Masters. As you might imagine, some Requestors can limit who can complete their HITs by assigning “Qualifications” that Workers must attain before participating in their tasks. Qualifications can range from requiring Master status to having approved completion of a specific number of HITs. While most Workers are based in the US, the service does boast an impressive gender balance, with about 47% of its users being women. Furthermore, Turkers are generally considered to be younger and have a lower income compared to the general US internet population, but possess a similar race composition. Additionally, many Workers worldwide cite Mechanical Turk as their main or secondary sources of income.

Since its launch, MTurk has been very popular, including among researchers. The number of articles on Web of Science with the search term “Mechanical Turk” has gone from just over 20 in 2012 to close to 100 in 2014 (see Figure 1). A similar search on PubMed produces 15 publications since the beginning of 2015.

graph — Figure 1. The number of articles found on the Web of Science prior to 2015 with the search term ‘Mechanical Turk’ within the ‘psychology’ research area. Used with permission fromWoods, Velasco, Levitan, Wan, & Spence (in preparation).

However, the popularity of MTurk has not come without controversy. Upon completing a HIT, Workers are not compensated until their task has been “approved” by the Requestor. Should the Requestor reject the HIT, the Worker receives no compensation and their reputation (% approval ratings) decreases. Many Turkers have complained about having had their HITs unfairly rejected, claiming Requestors keep their task data while withholding payment. Amazon has refused to accept responsibility for Requestors’ actions, claiming it merely creates a marketplace for Requesters and Turkers to contract freely and does not become involved in resolving disputes. Additionally, Amazon does not require Requestors to pay Workers according to any minimum wage, and a quick search of available HITs reveals many tasks requiring workers to devote a considerable amount of time for very little compensation. However, MTurk is only one of several crowdsourcing platforms, including CloudCrowd, CrowdFlower, and Prolific Academic.

Launched in 2014, Prolific Academic describes itself as “a crowdsourcing platform for academics around the globe”. Founded by collaborating academics from Oxford and Sheffield, Prolific Academic markets itself specifically as a platform for academic researchers. In fact, until August 2014, registration to the site was limited to UK-based individuals with academic emails (*.ac.uk) until it was opened up to everyone with a Facebook account (for user authentication purposes). Going a step further than its competition in appealing to academic researchers, Prolific Academic offers an extensive list of pre-screening questions (including questions about sociodemographic characteristics, levels of education or certifications, and more) that researchers can use to determine if someone is eligible to complete their study. Therefore, before someone can access and complete their study, they have to answer screening questions selected by the researcher. Individuals who have already completed screening questionnaires (available immediately upon signing up) will only be shown studies they are eligible for under the study page. At the time of writing this blog, according to the site’s homepage there are 5,081 individuals signed up to the site, with over 26,000 data submissions to date. Additionally, the site reports that participants have earned over £26,000 overall thus far. According to the site’s own demographics report from November 2014, 62% of users are male and the average age of users is about 24. Users are predominantly based in the US or UK. However, 1,500 users have joined since this report alone! Unlike MTurk and most other crowdsourcing platforms, Prolific Academic stipulates that researchers must compensate participants appropriately, which they term “Ethical Rewards”, requiring that participants be paid a minimum of £5 an hour.#

I have had experience using both MTurk and Prolific Academic in conducting and participating in research. With the assistance of Dr Andy Woods and his Xperiment platform, where my experimental task is hosted online, I was able to get an emotion recognition task up and running online. This opened up the possibility of studies on larger and more diverse samples, as well as studies being completed in MUCH shorter time frames. With Andy’s help in setting up studies on MTurk, I have run three studies on the platform since July 2014, ranging in sample size from 100 to 243 participants. Most impressively, each study was completed in a matter of hours; conducting the same study in the lab would have taken months! Similarly, given the short duration of these tasks, and the speed and ease of completing and accessing study documents on a computer, these studies cost less than they would have had they been conducted in the lab.

My experience with Prolific Academic has only been as a participant thus far but has been very positive. All the studies I completed have adhered to the “Ethical Rewards” requirement, and all researchers have been prompt in compensating me following study completion. Study duration estimates have been accurate (if anything generous) and compensation is only withheld in the case of failed catch trials (more on that below). The site is very easy to use with a user-friendly interface. It is easy to contact researchers as well, which is helpful for any queries or concerns. I know several colleagues as well who have had similar experiences and I hope to run a study on the platform in the near future.

While there have been several criticisms of conducting research on these crowd-sourcing platforms, the most common one amongst researchers is that data acquired this way will be of lesser quality than data from lab studies. Critics argue that the lack of a controlled testing environment, possible distractions during testing, and participants completing studies for compensation as quick as possible without attending to instructions are all reasons against conducting experiments on these platforms. Given the fact that research using catch trials (trials included in experiments to assess whether participants are paying attention or not) has shown failure rates ranging from 14% to 46% in a lab setting, surely participants completing tasks from their own homes would do just as badly, if not worse? We decided to investigate for ourselves. In two of our online studies, we added a catch trial as the study’s last trial, shown below.

Out of the 343 people who completed the two studies, only 3 participants failed the catch trial. That is less than 1% of participants! And we are not the only ones who have found promising results from studies using crowdsourcing platforms. Studies have shown that Turkers perform better on online attention checks than traditional subject pool participants and that MTurk Workers with high reputations can ensure high-quality data, even without the use of catch trials. Therefore, the quality of data from crowdsourcing platforms does not appear to be problematic. However, using catch trials is still a very popular and useful way of identifying participants who may not have completed tasks with enough care or attention.

Since the launch of MTurk, many similar platforms have appeared and advances have been made. MTurk has been used for everything from getting Turkers to write movie reviews to helping with missing persons searches. It’s safe to say that crowdsourcing is here to stay and has changed the way we conduct research online, with many of these sites’ tasks working on mobile and tablet platforms as well. While people have been using computers and web platforms in testing for a long time now, using crowdsourcing platforms for participant recruitment is still in its infancy. Since the launch of MTurk, many similar platforms have appeared and advances have been made. With many new possibilties emerging with the use of these platforms, it is an exciting time to be a researcher.

Across the Pond – Life as an International Postgraduate Student

Posted on April 5, 2014April 5, 2014 by michael

Deciding to pursue your studies in a foreign country is both an exciting and scary prospect. During the spring of 2012 I made the decision to leave my family and friends behind in Canada and make my way across the ocean to the great unknown that was Bristol to pursue a PhD in Experimental Psychology. Despite never having visited the city, and not having anyone I knew waiting for me on the other side, I took the leap and boarded a plane to the next few years of my life. I haven’t looked back since.

park street — My first picture on Park Street

International students have become commonplace at universities worldwide among undergraduates and postgraduates alike . If you are one of many students considering studying abroad, you will be in good company no matter which university you choose to attend. International students tend to be active members of the student population and are especially eager to immerse themselves in all aspects of student life. Having left their old lives behind, international students coming to a new city and university are looking to make new friends and have new experiences. By attending events and visiting the Bristol International Student Centre (BISC) I quickly met many other postgraduates from around the world who I had the chance to discover the city with, and formed lasting friendships. Whether they are in your programme of study or not, or even doing a Masters or PhD, there are often many more similarities than differences between you, and you will be able to quickly establish your new circle of friends in your home away from home.

One thing that may take some time is adjusting to a new academic system, which may differ from what you are used to in your home country. While this is may be more relevant to taught postgraduates rather than research students, you may have to come to terms with it anyway for your own degree requirements or if you decide to teach or mark essays (more on that later). For instance, while a PhD in experimental psychology back home entails demanding modules and coursework while conducting research, the same degree is focused almost exclusively on research output in the UK. Also, many of my postgraduate friends who were studying for a taught Masters degree felt that there was a lot crammed into one year, as the same degree in many of their countries often last two years with the same amount of content. I certainly did not envy their exams and coursework! At Bristol’s School of Experimental Psychology, PhD students who have not previously completed a Masters, like myself, are asked to attend the taught Masters lectures and have to complete a certain amount of coursework. I quite enjoyed attending these lectures and it was a nice refresher during the early months, as my PhD work was slowly starting to pick up. Like most postgraduate departments, you can expect to have colleagues from all walks of life, each with different backgrounds and usually studying quite different topics even within the same school. This only adds to your experience, as you learn from each other almost as often as you might your supervisors or your lecturers; having the luxury of all of the school’s PhD students working in the same building is invaluable. You are usually only a door away from the answer to most of your questions, and more often than not someone in your own office can answer it for you! I know that the university where I did my undergraduate degree did not have quite the same set-up, so I didn’t take the convenience of the PhD students’ office building (only across the street from our main department) for granted. While these types of things may not be at the top of your priority list when looking at foreign universities, they can have a serious impact on your quality of life as a student. Take every opportunity you can to gain insight from current students, as no one is in a better position to give you an idea of what you’ll be getting yourself into for the next few years of your life. I was lucky enough to contact a colleague with whom I share a supervisor who was not only kind enough to answer my questions but also showed me around and helped me get set up once I arrived in Bristol. While I can only speak to my experience in my own school, PhD students tend to be very kind and helpful; you only need to ask for it!

Of course, an important thing to consider when looking to study abroad is cost. Let’s face it, being an international student isn’t cheap. That being said, if you are coming from somewhere like the United States, you may be surprised to know that tuition isn’t much more than what you might pay at home. However, coming from Canada (where tuition fees are quite low) meant I would be taking on a significant increase in the cost of my schooling. While tallying the cost of tuition, rent and living expenses may be daunting, there are some important things to bear in mind. The first is funding. Even if you are deciding to study abroad, your country may very likely still have funding opportunities available to you. What you need to do is find them and apply as soon as possible. Something to bear in mind is that funding, if awarded, only begins several months after your application has been submitted, reviewed, and finally approved. And most likely they will not pay you retroactively for all the time that application was under review and this process can take up to almost one year in some cases. This means in an ideal situation you should try to apply for funding almost a full year before you expect to begin your studies. If this is not feasible, keep this in mind and start working on it almost as soon as you arrive. Acquiring funding will really help you breathe easier and not have to worry as much about keeping finances airtight. Of course, if you can’t get funding you will need to be a bit more proactive and find ways to earn money while you are studying. While being a postgraduate is already quite demanding in itself, with enough organisation and time-management you can balance it all and be successful. It just might take a bit more work. An excellent way to make some money while you are studying is to take on a research demonstrator or teaching assistant position. As a demonstrator in psychology, I attend the lab sessions for year 1 or year 2 statistics lectures and assist the students with completing the data collection, analysis and write up for their lab reports. I am also responsible for marking a group of lab reports and providing the students with feedback for future reports. This makes for a very attractive addition to your CV and is a great way to get some first-hand experience of what you might expect from a future faculty position. If you are even contemplating a career in academia in the future this is a great way to get a head start, and I would definitely recommend it. If demonstrating isn’t your thing, consider essay marking. This still gives you a small taste of the academic world and is often less time-demanding. Both these jobs are usually available in your department and also tend to pay pretty well. Another great place to look for jobs is in your university’s student union. There tend to be a lot of jobs with hours and pay rates that are really attractive to students, and as a postgraduate you are an attractive candidate who can hopefully market yourself well in an interview. I was lucky enough to nab a job involving sports with flexible hours and I really enjoy it. Finally, another great job opportunity is working as a senior resident or warden at a university hall. Rent or accommodation costs will represent one of your most significant expenses, and working as a senior resident can see your rent cut by as much as half. Depending on the hall, you may have different commitments or working hours but they tend to be quite reasonable and the savings are really worth it. These are only a couple of ways to make money or cut costs while you are studying and there are many more out there if you look for them. Although this should be a no-brainer, I will mention it here: take advantage of cheap or FREE activities, services and especially FOOD whenever the opportunity arises. These can be quite numerous, especially at the beginning of the academic year, so be sure not to miss out! While being an international student is far from cheap, remember that you are getting an invaluable experience at a new university in a new country. This is especially true if you are attending a prestigious university or working on a topic that you are very passionate about, because then you will never doubt your investment. And that is what your education is: an investment. Even if you decide to return home after your degree, you now have experiences that are likely to be radically different from the other people who may be applying for the same job as you. You have acquired more than just a degree while away from home and it’s important to remember that when you go to that job interview or write that next letter of intent.

hand_placing_coin_in_piggyban_450 — International degree – A worthwhile investment

It is quite hard to summarise my experience as an international postgraduate student but so far the experience has been nothing less than positive. I do not take my situation for granted: I am studying a topic I am very interested and passionate about at a world-class university with excellent supervisors and the best colleagues I could ask for. If you decide to study abroad for the right reasons, namely to pursue a degree that interests you in a place that inspires you and not your dream shopping destination, you will not be disappointed. If you have questions about anything I covered in this blog post or anything that I didn’t, feel free to contact me and I’ll be happy to answer any questions you might have. In closing, if you are thinking about studying abroad, do not be afraid to make the leap. Your friends and family will still be there when you visit and will support you on this journey that will change your life forever. Open up to new experiences, new people, and learning new things about yourself and you will not regret your decision.

Michael Dalili is a 2nd year PhD student in TARG.