Research doesn’t just happen in the lab anymore: Mechanical Turk, Prolific Academic, and online testing.

By Michael Dalili @michaeldalili

Over the years, from assessment to analysis, research has steadily shifted from paper to PC. The modern researcher has an ever-growing array of computer-based and online tools at their disposal for everything from data collection to live-streaming presentations of their work. While shifting to computer- or web-based platforms is easier for some areas of research than others, this has proven to work especially well in psychology. These platforms can be used for anything from simply hosting an online version of a questionnaire, to recruiting and testing participants on cognitive tasks. Throughout the course of my PhD, I have increasingly used online platforms for multiple purposes, ranging from participants completing questionnaires online on Bristol Online Survey, to recruiting participants using Amazon Mechanical Turk and completing a task hosted on the Xperiment platform. And I’m not alone! While it’s impossible to estimate just how many researchers are using computer- and web-based platforms to conduct their experiments, we have a better idea of how many researchers are using online crowdsourcing platforms such as Mechanical Turk and Prolific Academic for study recruitment. Spoiler alert: It’s A LOT! In this blog post I will describe these two platforms and give an account of my experiences using them for online testing.

amazon

Amazon Mechanical Turk, or MTurk for short, is the leading online crowdsourcing platform. Described as an Internet marketplace for work that requires human intelligence, MTurk was publicly launched in 2005, having previously been used internally to find duplicates among Amazon’s product webpages.  It works as follows: workers (more commonly known as “Turkers”), who are individuals who have registered on the service, complete Human Intelligence Tasks (known as HITs) created by Requestors, who approve the completed HIT and compensate the Workers. Prior to accepting HITs, Workers are presented with information about the task, the duration of the task, and the amount of compensation they will be awarded upon successfully completing the task. Right now there are over 280,000 HITs available, ranging widely in terms of the type and duration of task as well as compensation. Amazon claims its Workers number over 500,000 ranging from 190 countries. They can be further sub-divided into “Master Categories”, who are described by Amazon as being “an elite group of Workers who have demonstrated superior performance while completing thousands of HITs across the Marketplace”. At time of writing, there are close to 22,000 Master Workers, with about 3,800 Categorization Masters and over 4,500 Photo Moderation Masters. As you might imagine, some Requestors can limit who can complete their HITs by assigning “Qualifications” that Workers must attain before participating in their tasks. Qualifications can range from requiring Master status to having approved completion of a specific number of HITs. While most Workers are based in the US, the service does boast an impressive gender balance,  with about 47% of its users being women.  Furthermore, Turkers are generally considered to be younger and have a lower income compared to the general US internet population, but possess a similar race composition. Additionally, many Workers worldwide cite Mechanical Turk as their main or secondary sources of income.

Since its launch, MTurk has been very popular, including among researchers. The number of articles on Web of Science with the search term “Mechanical Turk” has gone from just over 20 in 2012 to close to 100 in 2014 (see Figure 1). A similar search on PubMed produces 15 publications since the beginning of 2015.

graph
Figure 1. The number of articles found on the Web of Science prior to 2015 with the search term ‘Mechanical Turk’ within the ‘psychology’ research area. Used with permission fromWoods, Velasco, Levitan, Wan, & Spence (in preparation).

However, the popularity of MTurk has not come without controversy. Upon completing a HIT, Workers are not compensated until their task has been “approved” by the Requestor. Should the Requestor reject the HIT, the Worker receives no compensation and their reputation (% approval ratings) decreases. Many Turkers have complained about having had their HITs unfairly rejected, claiming Requestors keep their task data while withholding payment. Amazon has refused to accept responsibility for Requestors’ actions, claiming it merely creates a marketplace for Requesters and Turkers to contract freely and does not become involved in resolving disputes. Additionally, Amazon does not require Requestors to pay Workers according to any minimum wage, and a quick search of available HITs reveals many tasks requiring workers to devote a considerable amount of time for very little compensation. However, MTurk is only one of several crowdsourcing platforms, including CloudCrowd, CrowdFlower, and Prolific Academic.

poracad

Launched in 2014, Prolific Academic describes itself as “a crowdsourcing platform for academics around the globe”. Founded by collaborating academics from Oxford and Sheffield, Prolific Academic markets itself specifically as a platform for academic researchers. In fact, until August 2014, registration to the site was limited to UK-based individuals with academic emails (*.ac.uk) until it was opened up to everyone with a Facebook account (for user authentication purposes). Going a step further than its competition in appealing to academic researchers, Prolific Academic offers an extensive list of pre-screening questions (including questions about sociodemographic characteristics, levels of education or certifications, and more) that researchers can use to determine if someone is eligible to complete their study. Therefore, before someone can access and complete their study, they have to answer screening questions selected by the researcher. Individuals who have already completed screening questionnaires (available immediately upon signing up) will only be shown studies they are eligible for under the study page. At the time of writing this blog, according to the site’s homepage there are 5,081 individuals signed up to the site, with over 26,000 data submissions to date. Additionally, the site reports that participants have earned over £26,000 overall thus far. According to the site’s own demographics report from November 2014, 62% of users are male and the average age of users is about 24. Users are predominantly based in the US or UK. However, 1,500 users have joined since this report alone! Unlike MTurk and most other crowdsourcing platforms, Prolific Academic stipulates that researchers must compensate participants appropriately, which they term “Ethical Rewards”, requiring that participants be paid a minimum of £5 an hour.#

lab

I have had experience using both MTurk and Prolific Academic in conducting and participating in research. With the assistance of Dr Andy Woods and his Xperiment platform, where my experimental task is hosted online, I was able to get an emotion recognition task up and running online. This opened up the possibility of studies on larger and more diverse samples, as well as studies being completed in MUCH shorter time frames. With Andy’s help in setting up studies on MTurk, I have run three studies on the platform since July 2014, ranging in sample size from 100 to 243 participants. Most impressively, each study was completed in a matter of hours; conducting the same study in the lab would have taken months! Similarly, given the short duration of these tasks, and the speed and ease of completing and accessing study documents on a computer, these studies cost less than they would have had they been conducted in the lab.

My experience with Prolific Academic has only been as a participant thus far but has been very positive. All the studies I completed have adhered to the “Ethical Rewards” requirement, and all researchers have been prompt in compensating me following study completion. Study duration estimates have been accurate (if anything generous) and compensation is only withheld in the case of failed catch trials (more on that below). The site is very easy to use with a user-friendly interface. It is easy to contact researchers as well, which is helpful for any queries or concerns. I know several colleagues as well who have had similar experiences and I hope to run a study on the platform in the near future.

While there have been several criticisms of conducting research on these crowd-sourcing platforms, the most common one amongst researchers is that data acquired this way will be of lesser quality than data from lab studies. Critics argue that the lack of a controlled testing environment, possible distractions during testing, and participants completing studies for compensation as quick as possible without attending to instructions are all reasons against conducting experiments on these platforms. Given the fact that research using catch trials (trials included in experiments to assess whether participants are paying attention or not) has shown failure rates ranging from 14% to 46% in a lab setting, surely participants completing tasks from their own homes would do just as badly, if not worse? We decided to investigate for ourselves. In two of our online studies, we added a catch trial as the study’s last trial, shown below.

glitch

Out of the 343 people who completed the two studies, only 3 participants failed the catch trial. That is less than 1% of participants! And we are not the only ones who have found promising results from studies using crowdsourcing platforms. Studies have shown that Turkers perform better on online attention checks than traditional subject pool participants and that MTurk Workers with high reputations can ensure high-quality data, even without the use of catch trials. Therefore, the quality of data from crowdsourcing platforms does not appear to be problematic. However, using catch trials is still a very popular and useful way of identifying participants who may not have completed tasks with enough care or attention.

Since the launch of MTurk, many similar platforms have appeared and advances have been made. MTurk has been used for everything from getting Turkers to write movie reviews to helping with missing persons searches. It’s safe to say that crowdsourcing is here to stay and has changed the way we conduct research online, with many of these sites’ tasks working on mobile and tablet platforms as well. While people have been using computers and web platforms in testing for a long time now, using crowdsourcing platforms for participant recruitment is still in its infancy. Since the launch of MTurk, many similar platforms have appeared and advances have been made. With many new possibilties emerging with the use of these platforms, it is an exciting time to be a researcher.

Reducing alcohol consumption in illicit drug users: new Cochrane review on psychotherapies

By Olivia Maynard @OliviaMaynard17

This blog originally appeared on the Mental Elf site on 28th January 2015

shutterstock_3084226

Whilst we all know that excessive alcohol consumption is bad for our health, illicit drug users are one group for whom problem alcohol use can be especially harmful, causing serious health consequences.

The prevalence of the hepatitis C virus is high among illicit drug users and problem alcohol use contributes to a poorer prognosis of this disease by increasing its progression to other diseases. In addition, rates of anxiety, mood and personality disorders are higher among illicit drug users, each of which is exacerbated by problem alcohol use.

Despite these health consequences, the prevalence of problem alcohol use is high among illicit drug users, with around 38% of opiate- and 45% of stimulant-using treatment-seeking individuals having co-occurring alcohol use disorders (Hartzler 2010; Hartzler 2011).

Previous Cochrane reviews have investigated the effectiveness of psychosocial interventions (or ‘talking therapies’) for either problem alcohol use, or illicit drug use alone. However, none have investigated the effectiveness of these therapies for individuals with concurrent problem alcohol and illicit drug use. Given the significant health risk and the high prevalence of concurrent problem alcohol and illicit drug use, a Cochrane review of this kind is long over-due.

Luckily, Kilmas and colleagues have done the hard work for us and their comprehensive Cochrane review of the literature evaluates the evidence for talking therapies for alcohol reduction among illicit drug users (Klimas et al, 2014).

This updated Cochrane review looks at psychotherapy for concurrent problem alcohol and illicit drug use.

The talking therapies we’re concerned with here are psychologically based interventions, which aim to reduce alcohol consumption without using any pharmacological (i.e. drug-based) treatments. Although there’s a wide range of different talking therapies currently used in practice, the ones which are discussed in this Cochrane review are:

  • Motivational interviewing (MI): this uses a client-centered approach, where the client’s readiness to change and their motivation, is a key component of the therapy.
  • Cognitive-behavioural therapy (CBT): this focuses on changing the way a client thinks and behaves. To address problem alcohol use, CBT approaches identify the triggers associated with drug use and use behavioural techniques to prevent relapse.
  • Brief interventions (BI): often BIs are based on the principles of MI and include giving advice and information. However, as implied by the name, BIs tend to be shorter and so are more suitable for non-specialist facilities.
  • The 12-step model: this is the approach used by Alcoholics Anonymous and operates by emphasising the powerlessness of the individual over their addiction. It then uses well-established therapeutic approaches, such as group cohesiveness and peer pressure to overcome this addiction.

Methods

  • The Cochrane review included all randomised controlled trials which compared psychosocial interventions with another therapy (whether that be other psychosocial therapies (to allow for comparison between therapies), pharmacological therapies, or placebo). Participants were adult illicit drug users with concurrent problem alcohol use
  • Four studies were included, involving 594 participants in total
  • The effectiveness of these interventions were assessed and the authors were most interested in the impact of these therapies on alcohol use, but were also interested in their impact on illicit drug use, participants’ engagement in further treatment and differences in alcohol related harms
  • The quality of the studies was also assessed

The quality of trials included in this review could certainly have been a lot better.

Results

The four studies were very different, each comparing different therapies:

  • Study 1: cognitive-behavioural therapy versus the 12-step model (Carroll et al, 1998)
  • Study 2: brief intervention versus treatment as usual (Feldman et al 2013)
  • Study 3: group or individual motivational interviewing versus hepatitis health promotion (Nyamathi et al, 2010)
  • Study 4: brief motivational intervention versus assessment only (Stein et al, 2002)

Due to this heterogeneity, the results could not be combined and so each study was considered separately. Of the four studies, only Study 4 found any meaningful differences between the therapies compared. Here, participants in the brief motivational intervention condition had reduced alcohol use (by seven or more days in the past month at 6-month follow up) as compared with the control group (Risk Ratio 1.67; 95% Confidence Interval 1.08 to 2.60; P value = 0.02). However, no other differences were observed for other outcome measures.

Overall, the review found little evidence that there are differences in the effectiveness of talking therapies in reducing alcohol consumption among concurrent alcohol and illicit drug users.

The authors of this review also bemoan the quality of the evidence provided by the four studies and judged them to be of either low or moderate quality, failing to account for all potential sources of bias.

The review found no evidence that any of the four therapies was a winner when it came to reducing alcohol consumption in illicit drug users.

Conclusions

So, what does this all mean for practice?

In a rather non-committal statement, which reflects the paucity of evidence available, the authors report that:

based on the low-quality evidence identified in this review, we cannot recommend using or ceasing psychosocial interventions for problem alcohol use in illicit drug users.

However, the authors suggest that similar to other conditions, early intervention for alcohol problems in primary care should be a priority. They also argue that given the high rates of co-occurrence of alcohol and drug problems, the integration of therapy for these two should be common practice, although as shown here, the evidence base to support this is currently lacking.

And what about the comparison between the different talking therapies?

Again, rather disappointingly, the authors report that:

no reliable conclusions can be drawn from these data regarding the effectiveness of different types of psychosocial interventions for the target condition.

How about the implications for research? What do we still need to find out?

This review really highlights the scarcity of well-reported, methodologically sound research investigating the effectiveness of psychosocial interventions for alcohol and illicit drug use and the authors call for trials using robust methodologies to further investigate this.

Choosing a therapy for this group of patients is difficult with insufficient evidence to support our decision.

Links

Klimas J, Tobin H, Field CA, O’Gorman CSM, Glynn LG, Keenan E, Saunders J, Bury G, Dunne C, Cullen W. Psychosocial interventions to reduce alcohol consumption in concurrent problem alcohol and illicit drug users. Cochrane Database of Systematic Reviews 2014, Issue 12. Art. No.: CD009269. DOI: 10.1002/14651858.CD009269.pub3.

Hartzler B, Donovan DM, Huang Z. Comparison of opiate-primary treatment seekers with and without alcohol use disorderJournal of Substance Abuse Treatment 2010;39 (2):114–23.

Hartzler B, DonovanDM,Huang Z. Rates and influences of alcohol use disorder comorbidity among primary stimulant misusing treatment-seekers: meta-analytic findings across eight NIDA CTN trialsThe American Journal of Drug and Alcohol Abuse 2011;37(5):460–71.

Carroll, K.M., Nich, C. Ball, S.A, McCance, E., Rounsavile, B.J. Treatment of cocaine and alcohol dependence with psychotherapy and dislfram. Addiction 1998; 93(5):713-27. [PubMed abstract]

Feldman N, Chatton A, Khan R, Khazaal Y, Zullino D. Alcohol-related brief intervention in patients treated for opiate or cocaine dependence: a randomized controlled studySubstance Abuse Treatment, Prevention, and Policy 2011;6(22):1–8.

Nyamathi A, Shoptaw S,Cohen A,Greengold B,Nyamathi K, Marfisee M, et al. Effect of motivational interviewing on reduction of alcohol useDrug Alcohol Dependence 2010;107(1):23–30. [1879–0046: (Electronic)]

Stein MD, Charuvastra A, Makstad J, Anderson BJ. A randomized trial of a brief alcohol intervention for needle exchanges (BRAINE). Addiction 2002;97(6):691. [:09652140] [PubMed abstract]

Mikhail Pogosov / Shutterstock.com

– See more at: http://www.thementalelf.net/mental-health-conditions/substance-misuse/reducing-alcohol-consumption-in-illicit-drug-users-new-cochrane-review-on-psychotherapies/#sthash.DgftSUSM.dpuf

Helping people with depression return to work

By Meg Fluharty, @MegEliz_

This blog originally appeared on the Mental Elf blog on 27th January 2015.

shutterstock_213396637

Depression is a major public health concern, with a wide range of symptoms, including hopelessness, fatigue, impaired concentration, feelings of inadequacy, as well as slowed thought and movement processing (APA 2013).

These symptoms not only impact upon an individuals’ personal life, but can impair social functioning and the ability to work (Hirschfeld 2000, Lerner 2008).

Within the US, depression was related to 27.2 lost workdays per ill worker per year, and a total of $36.6 billion capital lost in the US labour force (Kessler, 2006).

A new Cochrane systematic review and meta-analysis aims to evaluate the effectiveness of the current interventions available for reducing workplace disability in depressive disorder (Nieuwenhuijsen et al, 2014).

A US study from 2006 found that depression was related to 27.2 lost workdays per ill worker per year.

Methods

The authors searched the following databases between January 2006 and January 2014: CENTRAL, MEDLINE, psychINFO, EMBASE, and CINAHL. Studies were included if they were:

  • Randomised controlled trials (RCT) or cluster RCTs
  • Participants were adults (17+)
  • Participants were from occupational health, primary care, or outpatient care settings
  • Depressive criteria met diagnostic criteria, was assessed by a self-reported symptom scale, or by a clinical rated instrument.

Studies were excluded if participants had a primary diagnosis of a psychiatric disorder other than depressive disorder including bipolar depression and depression with psychotic tendencies.

The authors included both workplace (modify the task or hours) and clinical (antidepressant, psychological, or exercise) interventions, and the primary outcome examined was the number of illness-related absences from work during follow up (Nieuwenhuijsen et al, 2014).

Workplace adjustments

Results

The original search yielded a total of 11,776 studies, and resulted in a full text assessment of 73 studies. 50 studies were excluded at the full-text stage- resulting in 1 study included in qualitative synthesis only, and 22 studies included within the meta-analysis.

Overall there were 20 RCTs and 3 Cluster RCTs, totalling 6,278 participants ranging from 20-200 participants between studies. 7 studies recruited from primary care settings, 10 from outpatient, 2 from occupational health, 1 from a managed care setting, and 1 was conducted in a community mental health centre (Nieuwenhuijsen et al, 2014).

Work directed interventions

5 work-directed interventions were identified:

  • There was moderate evidence that a work-directed intervention plus a clinical intervention reduced sick days when compared to clinical intervention alone or a work intervention alone
  • There was low evidence that an occupational therapy and return to work program was beneficial over occupational care as usual

The review found evidence to support a combination of work-directed interventions and clinical interventions.

Antidepressants

6 studies investigated and compared the effectiveness of different antidepressant use, including SSRI, SNRI, TCA, MAO, and placebo:

  • There was no difference between SSRIs and TCAs in reducing sickness absence, while another study found low quality evidence that either TCAs or MAOs reduced absences over placebo
  • Overall, the results of this category were inconsistent

Psychological therapies

  • There was moderate evidence of online or telephone CBT against occupational care as usual for reduction of absences
  • Two studies displayed no evidence that community health nurse interventions helped any more than care-as-usual

Psychological therapies combined with antidepressants

  • Two studies found that enhanced primary care did not decrease sick days over 4-12 months, and another longer term study found similar results
  • However, there was high quality evidence that a telephone outreach management program can be effective in reducing sick leave compared to care-as-usual

Exercise

  • There was low quality evidence that exercise was more effective than relaxing in sickness absence reduction
  • However, there was moderate evidence that aerobic exercise was not more effective than relation or stretching

The review found evidence to support the use of telephone outreach management programs (stern Matron optional).

Discussion

This review evaluated a number of RCTs investigating work or clinical interventions. However, in each category, there was a large amount of variation between the studies and very few studies per category making comparisons difficult.

There was moderate evidence that work-directed interventions combined with a clinical intervention reduced sick leave, and that primary or occupational care combined with CBT also reduced absences. Additionally, there was evidence that a telephone outreach management program with medication reduced absences from work compared to care as usual.

This suggests the need for more research on work-directed interventions to be paired with clinical care, as they have the potential to reduce illness-related absences, but there are currently limited studies evaluating these interventions (Nieuwenhuijsen et al, 2014).

primary or occupational care combined with CBT also reduced absences.

Links

Nieuwenhuijsen K, Faber B, Verbeek JH, Neumeyer-Gromen A, Hees HL, Verhoeven AC, van der Feltz-Cornelis CM, Bültmann U. Interventions to improve return to work in depressed people. Cochrane Database of Systematic Reviews 2014, Issue 12. Art. No.: CD006237. DOI: 10.1002/14651858.CD006237.pub3.

American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition. Arlington, VA: American Psychiatric Association, 2013.

Hirschfeld RM, Montgomery SA, Keller MB, Kasper S, Schatzberg AF, Moller HJ, et al. Social functioning in depression: a review. Journal of Clinical Psychiatry 2000; 61 (4):268–75. [PubMed abstract]

Lerner D, Henke RM. What does research tell us about depression, job performance, and work productivity? (PDF) Journal of Occupational and Environmental Medicine 2008; 50(4):401–10.

Kessler RC, Akiskal HS, Ames M, Birnbaum H, Greenberg P, Hirschfeld RM, et al. Prevalence and effects of mood disorders on work performance in a nationally representative sample of U.S. workers. American Journal of Psychiatry 2006; 163(9):1561–8.

Department of Health (2012). Advice for employers on workplace adjustments for mental health conditions (PDF). Department of Health, May 2012.

– See more at: http://www.thementalelf.net/mental-health-conditions/depression/helping-people-with-depression-return-to-work/#sthash.7fnmUfRX.dpuf