Published 23 July 2020
As the novel coronavirus spread like wildfire around the globe in early 2020, some 1,700 studies and trials have been registered in a race to find treatments, according . Randomised controlled double blind, open-label, single-arm; what does it all mean?
Jenny Devenport, a Roche biostatistician, gives some basics on what happens in rigorous, scientific clinical trials, and why not all studies and trials provide the same quality of information.
Isn’t all information good information in times of crisis?
In previous crises, like the West African Ebola virus epidemic, the World Health Organization (WHO) noted that out of sheer desperation, people will start trying off-label use of medicines to see if they work. It feels like such a race, and it’s not always done in a systematic way. If you test a drug in a group of patients – just the next five patients who walk through the door – it's very difficult to interpret what the results from those five patients mean, or the effect of the drug.
Part of the reason is that we don't know if there was something special about those patients or other environmental conditions that would have made them get better or get worse anyway, separate from the treatment. They may have had different characteristics or a different prognosis to begin with. And with COVID specifically, we are still learning about the disease and its progression. The bottom line? In some observational stories of a single cohort, whether it's five patients or 5,000 patients, we don't actually know the cause of the outcome.
Does that mean the patients weren’t properly vetted?
It's actually that the study design hasn't been vetted, because you cannot rule out that the patients would have gotten better anyway.
When the patients are randomly allocated to a treatment arm (they get the drug being studied) and a control arm (they receive “standard of care” or typical care for their condition), the characteristics of the patients are roughly equal. You've effectively removed the patient effect. Then, you're just looking at the difference in outcomes for A, if a patient was treated with the new drug, versus B, if the patient was treated with a standard of care.
Is the goal a perfect comparison with no bias?
Say you could have one patient come in with a headache. You'd give them a treatment and you'd measure their headache an hour later. Then, you'd roll back time and have the same patient to come in, and you'd not give them this treatment. You'd measure their headache an hour later and you'd compare the two outcomes. Everything else would be exactly the same: same patient, same condition, same starting point. The only thing that's different is the treatment. This experiment is perfect.
Of course, the perfect experiment is impossible in practice, but we want to remove as much bias as possible. So when we're talking about different levels of evidence, we're talking about what factors can we remove other than the treatment effect as an explanation for the outcome.
What keeps you up at night?
Nights and weekends for the past months have been a blur. Many of the trials we are seeing out there, especially early on, were not randomised, controlled trials. They were observational proof of concept studies. As a biostatistician, you look at the growing exposure and are curious to see the data. But to be honest, you need a study that's going to give you an answer – not one that is just going to generate more questions. For months on end, all you heard in the media was anecdotal evidence. There were all these single-arm, open-label trials talking about outcomes that were observed, but few able to answer the question, in an unbiased way, if certain treatments were helping.
Clinical trial terminology: Knowing the lingo
Standard of care
A treatment guideline based on scientific evidence and collaboration professionals involved in the treatment of a given condition.
In a single-arm study, everyone enrolled in the trial will be treated in the same way.
A treatment with no ‘active ingredient’ or real drug. It usually looks and feels like the real treatment to prevent bias or expectations, or to keep the trial “blind.”
Participants are split randomly into two or more groups. Sometimes one group is given the new drug and the other group is given standard treatment, or a placebo if no standard treatment exists. In other cases, several groups are given different doses of the new drug alongside a group being given the standard treatment or placebo. Groups are balanced based on age, gender and other patient features.
In a blind trial, patients don’t know if they are being given the drug that is being tested. In a double-blind study, the investigators/doctors also do not know until the end of the trial which patients receive the treatment. This helps prevent bias and expectations from the patients and their doctors and investigators.
Can you describe a rigorous or robust scientific trial for us?
You have a randomised trial where patients are allocated to treatment and control arms independent of their characteristics in order to achieve comparable groups at baseline. Allocation may be equal (1:1 randomisation). Sometimes allocation is actually 2:1, so twice as many patients are more likely to get the new treatment than not. You do that because it helps with recruitment for the trial by creating a better chance of getting new treatment. Randomisation removes potential bias due to imbalances in the patient characteristics, which may affect prognosis with or without treatment.
A double-blind trial means that neither the patients nor the investigators know what treatment the patient is on. The reason blinding is important is because some of the outcomes are fairly subjective. So if it's not an objective outcome, like death, where you can't really fudge that, it's really good to have blinding. That way, nobody is influenced by the knowledge of the treatment when they're making the assessments.
Above all, the goal is to provide an answer to the question: does the treatment provide a benefit for these patients relative to standard of care?
Are you able to distance yourself from the sheer numbers?
From the public health perspective, statisticians understand very well the idea of quarantine is to make sure that hospitals don't get overwhelmed. And we understand the idea that if you have a product that can reduce time in a hospital bed, that is going to help the healthcare systems and the population at large.
But as a statistician who’s used to doing randomised control trials and getting hard answers, it’s a shift in mindset to see so many observational trials and emergency treatment protocols generating so much patient exposure, but not necessarily generating conclusive evidence about the effectiveness of treatments. It’s a relief when somebody benefits, but rigorous and controlled testing remains the best way to confirm whether the outcome was caused by the treatment.
Was the pandemic a surprise to you?
In terms of what I think personally? No – we should have expected this. This is not the first pandemic of my career. And it's certainly not the last we’ll see. But the WHO has a lot of experience and Roche has a lot of experience. So it really makes me determined to see what can we do better to enable discoveries faster.
Why do clinical studies of treatments matter?
Drug development just takes a long time. There are so many variations and so many things that can go wrong. As much as we like to think as a society that we're developing cures, sometimes you just need a stop-gap to keep the low-tech things afloat. So I really think it's very important for researchers to constantly be learning and constantly be thinking about: what do we do next time? Because there is always a next time, and it doesn't necessarily matter what the disease is. There will be patients who need a drug right away. And there will be doctors and patients who are brave enough to try what's already out there when nothing else has been approved for that condition. So we need to be prepared to help them do that in a systematic way and get to an answer and approval as efficiently and effectively as possible.
What can introduce bias into a study?
- No control: It’s impossible to rule out that patient outcomes would have been the same/better/worse regardless of treatment.
- No randomisation: there’s no control or historical control, and it’s impossible to rule out that patient characteristics and other subjective assessments influenced treatment selection and outcomes.
- No blinding: study is open-label, and it is impossible to rule out that knowledge of the treatment influenced outcomes.
- Missing data: if people are absent or drop out of the study, and that’s reflected in the treatment outcome, that could introduce bias.