Author Archives: annablackwell

Why I took part in the “Preregistration Challenge”

By Sarah Peters

The preregistration of study protocols has a long history in clinical trials, but is a more recent innovation in many other areas. The hope is that it will help counter the “reproducibility crisis” in psychological science – the failure of many published findings to replicate reliably. Here I discuss my experience with the Open Science Framework “Preregistration challenge”, and argue for more widespread adoption of preregistering reports.

There is an ongoing methodological crisis in psychological science – the reproducibility crisis refers to the failure of many scientific findings to be replicated. The Reproducibility Project, a recent initiative led by Professor Brian Nosek at the University of Virginia, aimed to identify the scale of this crisis. A large collaboration between 270 project members reran 100 published psychological experiments, and found that just 36% of the initial findings were replicated. Similarly, some classic textbook experiments have proven difficult to replicate, and publication bias – whereby positive findings are more likely to be published and negative findings to be dismissed – plagues the field.

Given this, scientists are exploring how to improve the way we conduct research and thereby improve the quality of what we produce. One suggestion is to preregister our research question, methods and analysis plan in advance of data collection. It is hoped that public preregistration will limit analytical flexibility and post hoc hypothesising, thereby improving the transparency and robustness of research findings.

Curious about the benefits of preregistration, and to see how it differed from the way I’d previously conducted my research, my colleagues and I published a preregistration for a recent study on Open Science Framework (OSF). We were interested in whether Cognitive Bias Modification, a psychological intervention designed to shift the emotional interpretation of faces, would impact clinically-relevant outcomes. We also entered the study into OSF’s (ongoing!) Preregistration Challenge, which offers the chance to win a $1,000 prize to 1,000 researchers who go from preregistration to publication.

Preregistering our study did require a greater time commitment prior to running it, but thinking about our predictions, design, and analyses meant that we could spot any potential issues and improve our experimental design before we collected data (i.e., before it was too late!). As a preregistration is public and cannot be changed after it’s published, it forced us to think more carefully about our decisions. For example, thinking more carefully about whether our data would truly answer our question made us wonder whether the emotional biases we wanted to study might be more prominent when an individual is under stress, so we decided to include another task to measure this. Also, by knowing which statistical analyses we would conduct before recruiting participants we could ensure that our study was adequately powered and would meet the assumptions of the planned analyses.

Initially I was concerned that this approach could be limiting. What if we found something interesting that we hadn’t expected and wanted to run additional analyses to probe it? But a preregistered report doesn’t prevent that – it simply means that you would (honestly and transparently!) report those analyses as exploratory. This protection against HARKing (hypothesising after the results are known) is important; separating analyses as planned versus exploratory can prevent overconfidence in weaker findings and the publication of attractive, but uncertain, positive findings.

Following data collection, we went back to our preregistration. It was here that our earlier time investment paid off; once our data were cleaned we could immediately run our planned analyses, and much of the manuscript writing (introduction and methods) was already done. We also ran a number of exploratory analyses, such as whether our results were moderated by participants’ anxiety scores. We subsequently published our findings in the academic journal Royal Society Open Science, and were thrilled to receive one of the latest $1,000 Preregistration Challenge prizes for bringing our study from preregistration to publication!

While interpreting findings and making discoveries is an important aim of scientific research, it is just as important to continuously scrutinise the scientific method. As a scientist, there is no question that seeing data can influence my decisions and interpretations. However, the adoption of preregistration can eliminate this, make the process easier in the long term, and improve research quality overall.

Professor Nosek and other members of the Reproducibility Project argue that, “Progress in science is marked by reducing uncertainty about nature”. But, if scientific findings have not or cannot be replicated, we can’t be certain that they exist. Preregistration is a simple change to the way we do research that can help to halt the reproducibility crisis and produce effective and credible science.

Read more about how to take part in the Preregistration Challenge here.

See Peters et al.’s preregistration here, and the published study here.

Sarah Peters can be contacted via email at: s.peters@bristol.ac.uk.

Can cognitive interventions change our perception from negative to positive, and might that be useful in treating depression?

By Sarah Peters

Have you ever walked away from a social interaction feeling uncomfortable or anxious? Maybe you felt the person you were talking to disliked you, or perhaps they said something negative and it was all you could remember about the interaction. We all occasionally focus on the negative rather than the positive, and sometimes ruminate over a negative event, but a consistent tendency to perceive even ambiguous or neutral words, faces, and interactions as negative (a negative bias), may play a causal role in the onset and rate of relapse in depression.

A growing field of psychological interventions known as cognitive bias modification (CBM) propose that by modifying these negative biases it may be possible to intervene prior to the onset of depression, or prevent the risk of subsequent depressive episodes for individuals in remission. Given that worldwide access to proven psychological and pharmacological treatments for mood disorders is limited, and that in countries like the UK public treatment for depression is plagued by long waiting lists, high costs, side effects, and low overall response rates, there is a need for effective treatments which are inexpensive, and both quick and easy to deliver. We thought that CBM might hold promise here, so we ran a proof of principle trial for a newly developed CBM intervention that shifts the interpretation of faces from negative to positive (a demonstration version of the training procedure can be seen here). Proof of principle trials test an intervention in a non-patient sample, which is important to help us understand a technique’s potential prior to testing it in a clinical population – we need to have a good idea that an intervention is going to work before we give it to people seeking treatment!

In this study, we had two specific aims. Firstly, we aimed to replicate previous findings to confirm that this task could indeed shift the emotional interpretation of faces. Secondly, we were interested in whether this shift in interpretation would impact on clinically-relevant outcomes: a) self-reported mood symptoms, and b) a battery of mood-relevant cognitive tasks. Among these were self-report questionnaires of depressive and anxious symptoms, the interpretation of ambiguous scenarios, and an inventory of daily stressful events (e.g., did you “wait too long in a queue,” and “how much stress did this cause you on a scale of 0 to 7”). The cognitive tasks included a dot probe task to measure selective attention towards negative (versus neutral) emotional words, a motivation for rewards task which has been shown to measure anhedonia (the loss of pleasure in previously enjoyed activities), and a measure of stress-reactivity (whereby individuals complete a simple task under two conditions: safe and under stress). This final task was included because it is thought that the negative biases we were interested in modifying are more pronounced when an individual is under stress.

We collected all of our self-report and cognitive measures at baseline (prior to CBM), after which participants underwent eight sessions (in one week) of either CBM or a control version of the task (which does not shift emotional interpretation). We then collected all of our measures again (after CBM). In order to be as sure of our results as possible, there were a number of critical study design features we used. Our design, hypotheses, and statistical analyses were pre-registered online prior to collecting data (this meant that we couldn’t fish around in our data until we found something promising, then re-write our hypotheses to make that result seem stronger). We also powered our study to be able to detect an effect of our CBM procedure. This meant running a statistical calculation to ensure we had enough participants to be convinced by any significant findings, and their potential to be clinically useful. This told us we needed 104 individuals split evenly between groups. Finally, our study was randomised (participants were randomly allocated to the intervention group or the control group), controlled (one group underwent an identical “placebo” procedure), and double-blind (only an individual who played no role in recruitment or participant contact knew which group any one participant was in).

So, what did we actually find? While the intervention successfully shifted the interpretation of facial expressions (from negative to positive), there was only inconclusive evidence of improved mood and the CBM procedure failed to impact most measures. There was some evidence in our predicted direction that daily stressful events were perceived as less stressful by those in the intervention group post-CBM, and weaker evidence for decreased anhedonia in the intervention group. In an exploratory analysis, we also found some evidence that results in the stress-reactivity task were moderated by baseline anxiety scores – for this task, the effects of CBM were only seen in individuals who had higher baseline anxiety scores. However, exploratory findings like this need to be treated with caution.

Therefore, as is often the case in scientific research, our results were not entirely clear. However, there are a few limitations and directions for future research that might explain and help us to interpret our findings. Our proof of principle study only considered effects in healthy individuals. Although these individuals were clearly amenable to training, and may indeed have symptoms of depression or anxiety without a clinical diagnosis, our observation that more anxious individuals appeared to be more affected by the intervention warrants research in clinical populations. In fact, a reasonable parallel to the effects observed in this study may be working memory training, which does not transfer well to other cognitive operations in healthy samples, but shows promise as a tool for general cognitive improvement in impaired populations.

Future research is also needed to disambiguate the tentative self-report stress and cognitive anhedonia effects observed here. One possibility, for example, is that the 104 participants we recruited were not enough to detect an effect of transference from CBM training to other measures (the size of which is unknown). Given the complexity of any mechanism through which a computerised task could shift the perception of faces and then influence behaviour, it is likely that a larger sample is necessary. While it could be argued that if such a large group of individuals is warranted to detect an effect, that effect is likely too small to be clinically useful, we would argue that even tiny effects can indeed be meaningful (e.g., cancer intervention studies often identify very small effects which can have a meaningful impact at a population level).

Another explanation for our small effects is that while one week was long enough to induce a change in bias, it may not have been long enough to observe corresponding changes in mood. For instance, positive interpretation alone may not be enough – it may be that individuals need to go out into the world and use this new framework to have personal, positive experiences that gradually improve mood, and this process may take longer than one week.

Overall, this CBM procedure may have limited impact on clinically-relevant symptoms. However, the small effects observed still warrant future study in larger and clinical samples. Given the large impact and cost of mood disorders on the one hand, and the relatively low cost of providing CBM training on the other, clarifying whether even small effects exist is likely worthwhile. Even if this procedure fails to result in clinical improvement, documenting and understanding the different steps in going from basic scientific experimentation to intervening in clinical samples is crucial for both the scientific field and the general public to know. The current study is part of a body of research which should encourage all individuals who are directly or indirectly impacted by depression or other mood disorders. Novel approaches towards understanding, preventing, and treating these disorders are constantly being investigated, meaning that we can be hopeful for a reduction in the devastating impact they currently have in the not so distant future.

Read the published study here

Sarah Peters can be contacted via email at: s.peters@bristol.ac.uk 

Does calorie and unit information influence our drinking behaviour?

By Olivia Maynard

Over the past two years we’ve invited hundreds of people into the lab to drink beer. Unfortunately, we weren’t there to socialise; this was in the name of science. We wanted to know whether giving people information about the number of units and or calories in their beer influenced how much they drank and their perceptions of drinking.

There are strong arguments for including this information: providing unit information may increase knowledge about alcohol consumption and calorie information may help drinkers choose lower calorie (and as a result lower unit) beverages. However, we also wondered whether there might be some unintended consequences of providing this information, particularly for those who are highly motivated to drink. What if unit information simply allows these drinkers to choose higher strength drinks and calorie information only discourages them from eating more, not drinking less? What if discussion around mandatory unit and calorie labelling is distracting us from the bigger issues: health warnings, minimum unit pricing, improving treatment for alcohol dependence and stopping alcohol advertising to young people, to name a few?

So, with this healthy level of scepticism, we set about inviting 264 regular alcohol consumers (mostly undergraduate students) to attend a lab session where they were given some beer and completed some taste ratings. What participants didn’t know was that they had been randomly assigned to one of four conditions. One group had information about the calorie and unit content of the beers, one group just got calorie information, another had just unit information, and the final group got no information at all. As well as measuring how much beer they drank, we also asked participants to reflect on the likely impact of unit and calorie information on their drinking behaviour.

You can read all the results in our (open access) paper that was published this week in the journal Alcohol and Alcoholism. If you want the concise version: we found no evidence that either unit or calorie information influenced how much beer people consumed and we found a lot of variation in the amount people drank.

However, it was our analysis of participants’ thoughts on unit and calorie information that proved vital to understanding what was going on here. Our participants told us that their main motivation for drinking alcohol was usually to get drunk; where unit information was perceived as being helpful, this was to help them choose the highest strength drink. Unit and calorie information was seen as distracting from the social aspect of drinking, and although some participants felt that calorie information might reduce consumption, most thought it would affect others, not themselves. Some people thought that calorie information could be misused by encouraging people to eat less (to compensate), rather than drink less.

It’s interesting that even though the unit and calorie information was very visible in our study (on a piece of paper, presented for 10 minutes), those who had received this information were still very inaccurate when it came to reporting how many units and calories were in their drinks. They basically didn’t seem to have read or engaged with it. If they’re not reading it in this context, is it likely that drinkers will read this information when it’s printed in tiny font on the back of the bottle?

So, what does this all mean for any plans to introduce unit and calorie information? Our study only really tells us about the potential impact of unit and calorie information among young adults (many of whom were students) who tend to drink to get drunk. However, our findings do call into question whether mandatory unit and calorie labelling on its own would reduce how much people drink, and also highlights potential negative unintended consequences of providing this information.

Despite some of these potential unintended consequences, there are still reasons to include unit and calorie information, if only because it’s a consumer right (you know how many calories are in just about everything else you consume). However, perhaps more effort needs to be placed on making this information more engaging and embedding it into public understanding of recommended drinking levels. Coincidentally, an analysis of the public’s awareness of new national alcohol guidelines was also published yesterday. This report argues that although the public have a relatively high awareness of what the guidelines are, they should be put into context by increasing the public’s awareness of the links between alcohol and cancer. Perhaps using health messages such as ‘Drinking alcohol regularly is linked to long-term risks such as cancer’, alongside unit and calorie information, might result in more meaningful changes in attitudes and behaviours around drinking. I feel another study coming on….

Olivia Maynard can be found on Twitter at @OliviaMaynard17

A Summary of the E-cigarette Summit US 2017

By Jasmine Khouja

The first E-cigarette Summit US was held in Washington DC on the 8th May 2017. The one-day event brought together researchers, medical professionals and members of industry from all over the US as well as many from the UK (where the organisers have held E-cigarette Summits successfully for the past four years). A review of the safety of e-cigarettes was followed by a review of the regulations that have been proposed in the US. Throughout the day, comparisons were made between the UK and US, particularly in the approaches taken to health messages and regulation of e-cigarettes. In Professor Kenneth Warner’s opening address, he suggested that there are two types of researcher in the field of e-cigarette research: sceptics, who are focussed on potential harm and protecting children regardless of the potential harm reduction for adult smokers, and enthusiasts, who are focussed on potential benefits to public health due to smoking cessation which could outweigh the potential risk to children. By this definition, the majority of researchers who presented evidence appeared to be enthusiasts.

Here are some highlights from the summit:

Evidence Updates

The majority of presentations suggested that previous research has overestimated the health risks of e-cigarettes by using inappropriate methods such as testing the toxicants produced from vaping using temperatures which are not used by vapers. Recently, Dr Konstantinos Farsalinos and his team have attempted to replicate such findings with maximum temperatures used by vapers and are yet to find evidence that supports the previous findings.

Dual use was also a common theme in the presentations; dual use is the use of e-cigarettes alongside smoking (or other tobacco product use depending on the definition used). However, as Dr Andrea Villanti pointed out, context is key when researching dual use; two people defined as dual users may be extremely different. For example, one dual user may smoke one cigarette a week and vape daily and another may vape once a week and smoke 20 cigarettes a day. With this in mind, Dr Robin Mermelstein’s research focussed on dual users and found that common reasons for using e-cigarettes were using e-cigarettes as a substitute for cigarettes, to cut down their cigarette consumption, to curb their cravings in places they were not allowed to smoke and because they were trying to quit smoking.

Public health

Professor Linda Bauld provided evidence that public health messages can impact the effectiveness of e-cigarettes as a smoking cessation tool. In the UK, there is generally a positive stance taken towards the use of e-cigarettes for smoking cessation among the public health community, however this stance has not been adopted in the US. It was suggested numerous times that consensus among the public health community could help smokers to quit and could help the medical community to provide accurate advice.

Regulations

New regulations for e-cigarettes are being proposed for the US in the Cole-Bishop proposal. Under these regulations, the e-cigarette market would essentially be frozen, preventing improvements to devices in safety and efficacy according to Deborah Arnott. However, Matthew Myers would disagree and sees flexibility in the FDA regulations which he believes are absolutely necessary.

Overall, the summit was extremely informative and highlighted the need to clearly communicate the findings of well-designed research to the public in order to maximise the potential for reducing smoking rates with use of e-cigarettes.

Action for Brain Injury Week

By Eleanor Kennedy

It’s Action for Brain Injury Week this week (8th – 14th May), a campaign run by the non-profit brain injury association Headway.  This year the campaign is all about “A New Me”, giving a platform to survivors and their families to discuss how life-changing a brain injury can be. In honour of the campaign, I’m writing a summary about my PhD research on mild traumatic brain injury.

Traumatic brain injury (TBI) is an injury to the head that results in an alteration in consciousness. My work focuses on mild TBI, which injury involves symptoms such as confusion/disorientation, loss of consciousness of less than 30 minutes and/or memory loss around the event that led to the injury.

I’m interested in how mild TBI in youth may be associated with later behaviour. Initially I conducted a systematic review of the literature and found that there was evidence for an association between childhood mild TBI and behaviours such as substance use, committing crimes and behavioural issues. However, this was based on a small number of studies and there were some limitations to be addressed.

A key issue was the use of appropriate control participants. In this kind of research, the behaviour of participants with mild TBI has been compared to that of participants with no injuries. These control participants are usually similar to the mild TBI group in terms of demographic factors such as age, gender and socioeconomic background. However, these similarities do not consider injury factors that could also have an impact on behaviour, for example pain, absence from school, and the trauma of having an injury. A second control group that includes participants with a non-head-related injury addresses this issue.

In my own research, I use data from the Avon Longitudinal Study of Children and Adolescents (ALSPAC). This is a birth cohort that began in the early nineties when over 14, 000 pregnant women were recruited; biological, genetic, environmental and psychological information has been gathered on participating families ever since. Participants and their parents have answered questions relating to head injury and fractures at many time points across the children’s life time. It is possible to have a group with mild TBI, a group with broken bone history and a group with neither injury.

So far, we have explored the association between mild TBI from birth to age 16 years and risk behaviour at age 17 years. We found that participants with a mild TBI were more likely to use alcohol to a hazardous level than participants with a broken bone and participants with no injury. This is in line with previous research, and has important implications for recurrent TBI and recovery from TBI. Another finding was that participants with either a mild TBI or a broken bone were more likely to commit offences – suggesting that there may be common risk factors for acquiring an injury and criminal behaviour. For example, an individual who has the personality trait of sensation seeking could potentially be more likely to get into risky situations leading to injuries and to commit offences.

I recently presented these findings at the International Brain Injury Association’s 12th World Congress in New Orleans. At the conference, there was an exhibition of masks created as part of a project called ‘Unmasking Brain Injury’. Each mask was designed and decorated by a survivor of brain injury to share their experience; each mask was as unique as the individuals’ story. Projects that give a voice to people living with a brain injury, such as ‘A New Me’ campaign, are a reminder of the challenges that are faced when dealing with a brain injury. It’s a privilege to contribute research to this field and to listen to the voices of those experiencing it to promote awareness and compassion.