Rapid critical appraisal of an RCT Conclusion

The study reported in the article by Spink, M.J. et al., was a well conducted, but possibly under-powered, randomised controlled trial. The methodology is well explained in the article. The findings of this study demonstrated a 36% reduction in the falls rate during the trial for the intervention group compared to the control group, suggesting that the multifaceted podiatry intervention is an effective falls prevention strategy in people aged over 65 years with disabling foot pain.

But can you apply these results to your own clinical question? This is explored in the next part of Section 2.

Calculating measures of effect

The objective of this study was to compare rates of local reactions associated with two needle sizes (25mm and 16mm) used to administer routine immunisations to infants. We will consider the results of this study to demonstrate how to calculate some measures of effect.

In this study, the researchers collected data on the outcomes of redness, swelling, and tenderness at 6 hours, 1 day, 2 days, and 3 days after immunisation. You can find the results presented in a table on page 932 of the article. We will focus on just one of these outcomes - redness 6 hours after injection.

Relative Risk (RR) tells us how many times more likely it is that an outcome (for example redness) will occur in the intervention group compared to the control group. Results are often presented in a 2 X 2 table that makes it easy to calculate relative risk. The results for this outcome can be presented in a 2 X 2 table as follows:

	Redness	No redness
Intervention - long needle (53)	21 (a)	32 (b)
Control - short needle (57)	34 (c)	23 (d)

The formula for calculating RR is:
RR = a / a+b divided by c / c+d
RR = 21/53 divided by 34/57
RR = 0.4 / 0.6
RR = 0.66

This means that the babies who received their immunisation via the long needle (intervention) were 0.66 times more likely to experience redness 6 hours afterwards than the babies who received the short needle (control), that is, the intervention was protective.

To summarise:
RR = 1 means that there is no difference between the 2 groups
RR < 1 means that the intervention reduces the risk of the outcome
RR > 1 means that the intervention increases the risk of the outcome
RR = 0.66 means that the 25mm needle reduces the risk of redness at 6 hrs

Once we know the RR, we can work out other common measures of effect.

Absolute Risk Reduction (ARR) tells us the absolute difference in the outcome between the intervention and control groups.

ARR = risk in control group (short needle) minus risk in intervention group (long needle)
ARR = 0.6 – 0.4
ARR = 0.2 (20%)

Therefore the absolute benefit of the long needle is a 20% reduction in redness at 6 hours.

Relative Risk Reduction (RRR) tells us the reduction in the rate of the outcome in the intervention group relative to the rate in the control group.

RRR = 1 – RR
RRR = 1 – 0.66
RRR = 0.34 (34%)

Number Needed to Treat (NNT) tells us the number of babies we need to immunise using the long needle in order to prevent one case of redness at 6 hours after injection (compared to the short needle).

NNT = 1 / ARR
NNT = 1 / 0.2
NNT = 5

In other words, we would need to immunise 5 babies with the long needle to prevent one case of redness at 6 hours after injection, compared to using the short needle.

You will see these terms used to report measures of effect of intervention studies, with relative risk and number needed to treat most commonly used.

Calculating measures of effect

A patient asks if taking a fermented milk product containing lactobacillus or some other probiotic would prevent antibiotic-associated diarrhoea.

The patient provides the paper by Beausoleil et al 2007 which discusses a randomised controlled trial of a product containing Lactobacillus acidophilus and Lactobacillus casei for this indication. Any benefits or harms of the treatment can be measured in a number of different ways and we discuss these below.

ARR or Absolute Risk Reduction

The absolute amount by which the intervention reduces the risk (of death for example). The ARR is calculated by: the risk of the event in the control group minus the risk of the event in the treatment group. For a trial that looks at treatment designed to prevent a bad outcome (e.g. death or a stroke) we can say the following:

ARR = 0 means that there is no difference between the two groups
ARR = positive (>0) means that the treatment is beneficial
ARR = negative (<0) means that the treatment is harmful

Example: The trial explored the effect of taking a probiotic containing Lactobacillus acidophilus and Lactobacillus casei on the prevention of antibiotic-associated diarrhoea. From the figures in the paper we can see that antibiotic-associated diarrhoea occurred in 7/44 patients taking the Lactobacilli probiotic (15.9%) and in 16/45 patients in the placebo group (35.6%). The difference between these two risks is the absolute risk reduction (ARR). In this trial the absolute risk reduction for developing antibiotic-associated diarrhoea if you take a probiotic containing Lactobacillus acidophilus and Lactobacillus casei was found to be 35.6% minus 15.9% = 19.7%. Put another way the probiotic reduced the risk of developing diarrhoea by almost 20%.

RR or Relative Risk/Risk Ratio

The risk of outcome in the treatment group divided by the risk of the event (such as death) in the control group. Unlike the ARR, the RR therefore expresses the difference between the groups in a relative way. Again, for a trial measuring a bad outcome:

RR = 1 means that there is no difference between the two groups
RR <1 means that the treatment reduces the risk of the event
RR >1 means that the treatment increases the risk of the event

Example: In the statin study we worked out above that the event rate (death in this case) was 2.58% in the statin group and 3.06% in the controls. The RR is therefore 2.58/3.06 = 0.84. This means that any individual was 0.84 times as likely to die if they took statins (that is, those on statins were better off).

RRR or Relative Risk Reduction

This is a measure calculated by dividing the absolute risk reduction by the control event rate. Much beloved of pharmaceutical companies, this measure sometimes gives the impression that the effect of an intervention is more striking than when expressed as ARR or RR, particularly when the incidence of the event is small.

The RRR is calculated by subtracting the event rate in the intervention group from the event rate in the control group, and then dividing by the control group rate (CER − EER) / CER’ or ARR/CER.

RRR can also be determined by calculating 1 - RR.

Example: In the probiotic study we worked out that the ARR was 19.7%, and we know the control group diarrhoea rate was 35.6%. Therefore, the RRR is 19.7/35.6 = 0.55 - that is to say the RRR is 55%. Note that we would get the same result by calculating RRR using the formula 1- RR (1 – 0.45 = 0.55). Compare this with the ARR - which number sounds more impressive - and which would you use if you were trying to sell probiotics?

NNT or Number Needed to Treat

The NNT is the number of patients with a particular condition who must receive a treatment in order to prevent the occurrence of one adverse outcome. NNT is therefore the inverse of the Absolute Risk Reduction. Similarly, ‘number needed to harm’ (NNH) refers to harmful outcomes. The NNT is often used when considering the ‘real world’ impact of an intervention.

Example: Coming back again to the probiotic study, the ARR is 19.7%, or 19.7/100. The NNT is simply calculated by turning over the fraction: 100/19.7 = 5. We can say, therefore, that over the same time course as the study, we would need to treat 5 patients with probiotics in order to save one extra individual from developing antibiotic-associated diarrhoea.

OR or Odds Ratio

While calculation of the RR above relies on comparing the probability of an event in the two groups in the study, the OR estimates the strength of an effect by comparing the odds of the event occurring in each group. It is very important not to confuse odds and probability (or risk). We are not very good at intuitively understanding odds - and this is a fact well-known to bookmakers the world over.

The OR is simply the ratio of the odds (those with the outcome divided by those without it) in the treatment group to the corresponding odds in the control group. An odds ratio of ‘1’ implies that the outcome is equally likely in both groups.

Example: Back to our probiotic paper: In the probiotic group, 7 people developed diarrhoea out of a total of 44. Thus, the odds of developing diarrhoea are 7 to 44 - 7, that is 7 to 37. Odds are usually written as 7:37 or as some number ‘to 1’ (like the odds of a horse winning a race being ‘10 to 1’). In this case, the odds of developing diarrhoea if you take probiotics are 0.189:1. Similarly, the odds of developing diarrhoea if you are in the control group are 16 out of 45-16, or 16: 29 or 0.55: 1. The odds of developing diarrhoea if you take probiotics compared to the odds if you do not (the OR) are therefore 0.189/0.55, or 0.34:1. Often, the ‘1’ is only implied, so the OR is simply expressed as ‘0.34’.

Note that in this example, the OR and the RR are different (0.34 versus 0.45). In cases where the event rate is very low in both groups the OR and RR may be very similar. The OR and RR tend to be widely divergent when outcomes are common (that is, when the outcome occurs frequently).

For more detail on OR versus RR, including when each should be used, please consult an EBM text such as Strauss SE et al. Evidence-based medicine. How to practice and teach EBM. Fifth Edition, Churchill Livingstone, Edinburgh 2018.

The p Value: The probability that any particular outcome would have arisen by chance. A lower number indicates a lower probability that any observed differences are in fact due to chance. It has become common practice that when this probability is less than 1 in 20, we accept that the difference is ‘real’. That is, we accept a chance of 19 out of 20 (p = 0.95 or 95%) that we are correct, as ‘good enough’ for us to believe the result. Therefore, standard scientific practice usually agrees that a p value of less than 0.05 (5%) (expressed as p = <0.05) is ‘statistically significant’, or less than 5% likely to be due to chance.

A p value of less than one in 100 (p=<0.01) is considered ‘statistically highly significant’. A higher p value (in the non-significant range) tells you that there is either no difference between groups or there were too few subjects to demonstrate a difference.

CI or Confidence Interval

The CI gives a measure of the limits to our confidence about the true effect we have estimated in a study (we can calculate this confidence for whatever outcome measure we use). It puts some boundaries to our uncertainty and helps us to know if we should feel confident enough to apply the results of a study to the population we are interested in (usually our patients). If the CI crosses the value that represents no effect (e.g. an RR of 1.0), then by definition, there is no statistically significant effect. The CI is more important than simply knowing the p value because it contains both a measure of statistical significance and an indication of the likely size of the effect. Depending upon the values at each end of the CI, we can characterise the strength of the evidence (strong or weak, or perhaps even definitive).

The CI will always contain our best estimate of the size of the treatment effect and may be defined as the range of values that is 95% likely to include the real value (sometimes authors use 90% or 99%). A more accurate way of saying this is that if we did the same study 100 times we would expect that 95 of the 100 estimates of effect would fall between our confidence limits. The CI is therefore based on the idea that, while the same study carried out on different patients would not give identical results, the result is likely to be within the spread of the CI. All these measures of effect explained here can be expressed with 95% confidence intervals.

Because small studies have low power, they tend to have wider intervals and are therefore less reliable. The more subjects that there are in the study, the narrower the CIs are likely to be and therefore larger studies give more reliable results.

Example: Back to the probiotics paper - the authors calculated the OR for developing diarrhoea on probiotics to be 0.343 and the 95% CI was 0.125 to 0.944 (that is, between as small an effect as only a 12.5% chance of developing diarrhoea, or as good an effect as 94% chance, relative to someone not taking probiotics). The 95% CI was close to but did not cross 1, and the associated p value was 0.05.

While this result is statistically significant the numbers in the trial were very small. You might hesitate to base a judgement on whether or not to use probiotics on this paper. The paper also discussed the effect of probiotics on the prevention of Clostridium difficile-associated diarrhoea. If you search Cochrane you will find a recent systematic review on using probiotics to prevent Clostridium difficile-associated diarrhoea which includes this paper (Goldenberg  JZ, Yap  C, Lytvyn  L, Lo  CKF, Beardsley  J, Mertz  D, Johnston  BC. Probiotics for the prevention of Clostridium difficile‐associated diarrhea in adults and children. Cochrane Database of Systematic Reviews 2017, Issue 12. Art. No.: CD006095. DOI: 10.1002/14651858.CD006095.pub4.). It looked at the results of 31 studies with 8672 participants. This may be a better source of evidence on which to make a clinical decision.

Note: The CI does not take into account badly designed or poorly conducted studies. CIs therefore always underestimate the total amount of uncertainty.