If we confine ourselves to the predefined hypotheses and the part of the analysis that is indeed comparative, the conclusion should be that quality of homeopathic trials is better than o
Trang 1PROOF
ORIGINAL PAPER
The 2005 meta-analysis of homeopathy:
the importance of post-publication data
ALB Rutten1,*and CF Stolper2
1
Homeopathic physician, former general practitioner, Breda, The Netherlands
2
General practitioner, homeopathic physician, Machteldskamp 19, 8181 ZN Heerde, The Netherlands
Abstract: Background: There is a discrepancy between the outcome of a meta-analy-sis published in 1997 of 89 trials of homeopathy by Linde et al and an analymeta-analy-sis of 110 trials
by Shang et al published in 2005, these reached opposite conclusions Important data were not mentioned in Shang et al’s paper, but only provided subsequently
Questions: What was the outcome of Shang et al’s predefined hypotheses? Were the homeopathic and conventional trials comparable? Was subgroup selection justified?
The possible role of ineffective treatments Was conclusion about effect justified? Were essential data missing in the original article?
Methods: Analysis of post-publication data Re-extraction and analysis of 21 higher quality trials selected by Shang et al with sensitivity analysis for the influence of single indications Analysis of comparability Sensitivity analysis of influence of subjective choices, like quality of single indications and of cut-off values for ‘larger samples’
Results: Quality of trials of homeopathy was better than of conventional trials Regard-ing smaller trials, homeopathy accounted for 14 out of 83 and conventional medicine
2 out of 78 good quality trials with n < 100 There was selective inclusion of unpublished trials only for homeopathy Quality was assessed differently from previous analyses
Selecting subgroups on sample size and quality caused incomplete matching of homeop-athy and conventional trials Cut-off values for larger trials differed between homeophomeop-athy and conventional medicine without plausible reason Sensitivity analyses for the influence of heterogeneity and the cut-off value for ‘larger higher quality studies’ were missing Homeopathy is not effective for muscle soreness after long distance running,
OR = 1.30 (95% CI 0.96–1
was based was heterogeneous, comprising 8 trials on 8 different indications, and was not matched on indication with those of conventional medicine Essential data were missing in the original paper
Conclusion: Re-analysis of Shang’s post-publication data did not support the conclu-sion that homeopathy is a placebo effect The concluconclu-sion that homeopathy is and that conventional is not a placebo effect was not based on comparative analysis and not justified because of heterogeneity and lack of sensitivity analysis If we confine ourselves
to the predefined hypotheses and the part of the analysis that is indeed comparative, the conclusion should be that quality of homeopathic trials is better than of conventional trials, for all trials (p = 0.03) as well as for smaller trials (p = 0.003) Homeopathy (2008)
-, 1–9
Keywords: homeopathy; meta-analysis; comparative analysis; quality bias; selection bias; cut-off value; adverse effects
Q2
Introduction
The discussion about proof for homeopathy is in part,
a meta-discussion about proof Several meta-analyses of randomised controlled trials (RCT) – in 1991, 1997 and
*Corresponding author Lex Rutten, Aard 10, 4813 NN Breda, The
Netherlands Tel.: +31 (0) 765227340; Fax: +31 (0) 765227487.
E-mail: lexrtn@concepts.nl
Received 11 January 2008; revised 6 August 2008; accepted 11
September 2008
doi: 10.1016/j.homp.2008.09.008 , available online at http://www.sciencedirect.com
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142
Trang 2PROOF
2000 – indicate a specific effect of homeopathy.1–3 Both
homeopathic and conventional meta-analyses have been
criticised.4–6Some authors suggest that there is no
differ-ence between proof for homeopathy and for conventional
methods.1,7 However, the implausibility of homeopathy’s
mechanism of action seems to have led to an amalgamation
of bias Sterne, Egger and Smith concluded that the role of
low quality in small studies was neglected in Linde’s
meta-analysis.8Commenting on the analysis of homeopathy by
Shang et al published in August 2005, and which referred
to the ‘small low quality study’ hypothesis, the editor of
the Lancet advised ‘‘doctors need to be bold and honest
with their patients about homeopathy’s lack of benefit’’.9
Vandenbroucke concluded that this meta-analysis showed
higher sensitivity for potential bias for homeopathic than
for allopathic trials.10
The Cochrane Handbook for Systematic Reviews states
‘‘Reliable conclusions can only be drawn from analyses
that are truly pre-specified before inspecting the trials’
results’’.11Such pre-specification is more difficult because
most homeopathy trials have been analysed in earlier
meta-analyses The Cochrane Handbook further
recom-mends ‘‘Meta-analysis should only be considered when
a group of trials is sufficiently homogeneous in terms of
participants, interventions and outcomes to provide a
mean-ingful summary’’ Pooling of results is thus questionable if
homeopathy works for some conditions and not for
others.12 Egger stated ‘‘If subgroup analyses are to be
done, they need to be as complete as possible and should
involve commonly defined subgroups and outcomes across
all the trials in the subgroup’’.13Meta-analysis is a
subjec-tive procedure, Boden warns that it can easily become
a weapon instead of a tool.14
The hypotheses predefined mentioned in the introduction
of Shang et al’s paper were: ‘‘Bias in conduct and reporting
of trials is a possible explanation for positive findings of
placebo-controlled trials of both homeopathy and allopathy
(conventional medicine)’’; and: ‘‘These biases are more
likely to affect small than large studies; the smaller a study,
the larger the treatment effect necessary for the results to be
statistically significant, whereas large studies are more
likely to be of high methodological quality and published
even if their results are negative’’
Shang et al’s analysis was criticised because the authors
failed to include essential data to support their
conclu-sion.15–17Four months later the missing data were revealed
(www.ispm.ch) The missing data were 1 Excluded trials
2 The trials regarded as of higher quality 3 The trials
(8 homeopathy, 6 conventional medicine) that led to the
final conclusion
Questions
More or less the same set of homeopathy trials has been
re-analysed several times The contradiction between
Linde’s conclusion based on 89 trials, and Shang et al’s
conclusion, based on 110 trials seems odd Shang et al’s
analysis was presented as a comparative analysis matching
110 homeopathy trials with 110 conventional trials by
indications The conclusion was based on 8 homeopathy trials and 6 conventional trials
The post-publication data enabled us to reconstruct the analysis, although data were presented as graphs, not as raw numbers In our recent paper ‘The conclusions on the effectiveness of homeopathy highly depend on the set of analysed trials’ we re-analysed the data from the original articles, did sensitivity analyses and estimated the influence
of heterogeneity.18 The large amount of heterogeneity suggests that this factor was not considered at all We found
no reasonable explanation for the choice of cut-off value for
‘larger trials’
After these basic conclusions several questions remain:
1 What was the outcome of the pre-specified hypotheses?
2 Were the two methods comparable?
3 Was subgroup selection rectified?
4 What is the influence of ineffective treatments?
5 Was the final conclusion rectified?
6 Were essential data missing in the original article?
Methods
We analysed the subsequently disclosed data and inves-tigated which hypotheses were tested The ISPM website presented graphs, but no data about effect sizes and confidence intervals were given We reconstructed the odds ratios and confidence intervals of the 21 higher quality homeopathy studies from the original articles Data were processed and analysed with methods identical or equiva-lent to those of Shang et al’s analysis We checked the results with Shang et al’s data, then focused on these 21 higher quality studies because the conclusion was based
on larger higher quality studies For these trials a random effects meta-analysis was performed and the pooled odds ratio was estimated We estimated odds ratios and confi-dence intervals for some of the trials excluded by Shang
et al, but regarded as good quality by Linde et al We performed meta-analyses for other eligible sets of trials
We tested comparability and matching of trials We com-pared this analysis with referenced publications to check predefinition of hypotheses We assessed the influence of some subjective choices, like quality and cut-off values for sample size and performed sensitivity analysis to check for the influence of separate indications SAS/StatÒ, release 9.1 statistical software was used
Results
Shang et al presented their study and their conclusion as
a comparison of homeopathy and conventional medicine
To reconstruct their work we had to make several hypothe-ses that were not predefined by Shang et al, to arrive at their conclusions In this process comparibility of the homeo-pathic and conventional groups was lost
The predefined hypotheses The first predefined hypothesis (quality in homeopathy is worse than in conventional medicine) was falsified by
2
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284
Trang 3PROOF
Shang et al Median sample sizes were the same: 65.5 in
homeopathy, 65 in conventional medicine Effects of
home-opathy and conventional medicine were similar; 95% of the
odds ratios were from 0.12 to 1.65 for homeopathy and
from 0.13 to 1.52 for conventional medicine According
to Shang et al ‘‘Most odds ratios indicated a beneficial effect
of the intervention’’ In the homeopathy group (including
unpublished trials) 21 (19%) of the trials were of higher
quality, in the conventional group 9 (8%) Overall quality
in homeopathy studies was better than for conventional
medicine (p = 0.03)
Quality in small studies.Shang et al referred, for their
second predefined hypothesis, to Sterne, Egger and Smith
stating that quality bias is mainly influenced by quality of
small studies.8Effects of treatment could in their view truly
be larger in high quality smaller trials because of better
selection of patients On the other hand effects are
over-es-timated if quality is low In both cases we see asymmetry in
the funnel plot, but in the first case this does not indicate
bias and if larger trials with poorer patient selection then
indicate no effect the conclusion that the therapy is placebo
is not justified
Post-publication data showed which studies were
regarded as of higher quality We chosen < 100 as cut-off
value for smaller studies, Shang chosen < 98 for
homeopa-thy and n < 146 for conventional medicine (see below)
There were 14 homeopathy studies of higher quality out
of 83 trials (16.9%) withn < 100 There were two
conven-tional studies of higher quality out of 78 trials (2.6%) with
sample size <100.19,20 The hypothesis that low quality
small studies are therefore responsible for the positive
findings in homeopathy is mostly falsified (p = 0.003,
Fisher exact probability test) There is statistically
signifi-cant difference in quality of smaller studies in favour of
homeopathy
Since the quality of conventional and homeopathic
stud-ies was not comparable, comparison of effects of the two
methods was not valid The underlying hypothesis for
Shang et al’s analysis was that results cannot be compared
if quality is different As so much emphasis was laid on the
relation between quality and result we will nevertheless
continue with our observations concerning this relation,
although we did not compare effects of homeopathy and
conventional medicine
Comparability
The comparison of the two methods was somewhat
flawed by publication bias The 110 homeopathy trials
were matched on indication with 110 conventional trials
But all conventional trials were published as journal articles
while 16 (15%) of the trials in the homeopathy group were
unpublished According to Chan et al the odds of publishing
results in conventional medicine are greater if results were
significant (pooled odds ratio 2.4, 95% CI 1.4–4.0).21So,
in comparing effects homeopathy is disadvantaged by the
selective inclusion of unpublished trials But it also affects
comparison of quality Shang et al reported (in
post-publi-cation data) that none of the 16 unpublished homeopathy
trials were of higher quality The ratio of higher quality trials in published trials was 22% (21 out of 94) instead of 19% in the original paper
We did not further investigate possible selection bias by excluding trials, but we were surprised by the exclusion of Wiesenauer’s trial on chronic polyarthritis.22 This was
a larger trial (n = 176), of good quality according to Linde, with positive results.2 This trial would have contributed positively to the outcome of the larger higher quality trials
Shang excluded this trial because no matching trial could be found
Subgroups were selected on quality This selection further influenced matching on indication, and therefore comparability The homeopathy group contained 21 ‘higher quality’ studies, the conventional group 9 At this point only
4 homeopathy studies were matched on indication by con-ventional studies (19%) From this point onward Shang’s study consisted in fact of two incomparable meta-analyses
of effects, one about homeopathy, one about conventional medicine
Differences in effect between methods can no longer be evaluated if the matching is disrupted This can be shown
by comparing results for muscle soreness The post-publi-cation data show that neither homeopathy nor conventional medicine is effective for this indication, seeFigure 1 But the homeopathy studies are of higher quality while the conventional studies are not This difference was of fundamental importance in the subset that led to the final conclusion
The indication ‘muscle soreness’ has the largest influence
on the results of homeopathy and on the comparison between homeopathy and conventional medicine because four homeopathy studies were classified as higher quality against none for conventional medicine One of the home-opathy trials was also large and therefore higher in the funnel plot This trial inclines the funnel plot to the right (towards OR = 1.0), while the smaller trials for this indica-tion inclined it to the left because the pivot point is above these trials There is a strong influence of chance in such
a limited number of indications
We did not consider clinical relevance, but one could wonder about the inclusion of treatments that may not be used because of serious adverse effects Shang et al men-tioned in the discussion that a limitation of their study was its disregard of adverse effects They highly valued larger studies as a measure of quality and extrapolated effects towards the largest studies This extrapolation is questionable if the largest studies involve treatments that are not available because of serious adverse effects In
a larger trial of higher quality on weight loss homeopathy had no effect.23 The matched conventional study showed
a considerable positive effect of Dexfenfluramine,24 but Dexfenfluramine for weight loss was withdrawn by the American Food and Drug Administration in 1997 because
of serious cardiac complications.25 Two other larger stud-ies, Deladumone (androgen–estrogen) in breastfeeding and Piroxicam for soft tissue injury suffered from the same problem.26,27These two treatments were also with-drawn because of adverse effects.28,29There might be other
3 285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426
Trang 4PROOF
treatments which are hard to compare because of safety,
such as Tamoxifen for pre-menstrual syndrome.30
Possible bias in subgroup selection
Shang et al’s conclusion was based on comparison of
‘larger higher quality trials’ Possible pitfalls here are
in-comparability, heterogeneity and subjective criteria for
quality and sample size
The subjectivity of interpreting quality is demonstrated
by differences between authors of meta-analyses The
following studies were not classified as good quality by
Shang et al, although they are among the quality top 10
of Linde’s meta-analysis2: de Lange-de Klerk,31 Reilly
1986,32Hofmeyr,33Reilly 1994,34seeTable 1
If we add these four studies to Shang et al’s pooled Odd
Ratio (OR) of 25 trials becomes 0.74 (95% CI: 0.59–0.94)
Why should these studies not be valued as of higher
qual-ity? Schultz et al showed that inadequate concealment of
treatment allocation is the most important quality factor,
as-sociated with 41% (95%CI: 27–52%) exaggeration of
ef-fect.35 Other quality factors have less effect; sequence
generation 15%, (95%CI: 12–19%), double blinding 17%
(95%CI: 4–29%) The Jadad score for quality used by
Linde, does not consider allocation concealment These 4
trials in Linde’s meta-analysis had maximum Jadad scores,
and as far as we can tell also had adequate allocation
con-cealment We cannot estimate the influence of effect of
choices regarding quality because some new trials were
published after Linde’s study and some of Linde’s lower quality trials were regarded as of higher quality But, in combination with an unclear definition of ‘larger’ sample size, this subjectivity in defining quality opens a variety
of possible subgroups which could be considered ‘larger higher quality’ trials
Pooled odds ratio of all higher quality studies We reconstructed the ORs and confidence intervals of the 21 higher quality studies selected by Shang et al 36–55The pooled OR using random effects analysis for all 21 higher quality studies in homeopathy is 0.76 (95% CI 0.59– 0.99), which is not compatible with the placebo hypothe-sis.18
Cut-off value for sample size.Cut-off values for sample size were not mentioned or explained in Shang el al’s analysis Why were eight homeopathy trials compared with six conventional trials? Was this choice predefined
or post-hoc? Post-publication data showed that cut-off values for larger higher quality studies differed between
Figure 1 The effects of homeopathy and conventional medicine on ‘muscle soreness’ compared The other trials in the group ‘Musculo-skeletal complaints’ are disregarded The four studies concerning muscle soreness for both methods are indicated by author names.
N = trial size Source www.ispm.ch
Table 1 The four best studies according to Linde et al, arranged
by sample size First author Indication Sample size OR 95% CI of OR
de Lange-de Klerk 31 Upper respiratory
tract infection
170 0.85 0.47–1.53 Reilly32 Pollinosis 144 0.43 0.22–0.85 Hofmeyr 33 Childbirth 122 1.03 0.40–2.64 Reilly34 Asthma 24 0.08 0.02–0.40
4
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568
Trang 5PROOF
the two groups In the homeopathy group the cut-off value
wasn = 98, including eight trials (38% of the higher quality
trials) The cut-off value for larger conventional studies in
this analysis wasn = 146, including six trials (66% of the
higher quality trials) These cut-off values were
consider-ably above the median sample size of 65 There were 31
homeopathy trials larger than the homeopathy cut-off value
and 24 conventional trials larger than the conventional
cut-off value We can think of no criterion that could be
common to the two cut-off values This suggests that this
choice was post-hoc
Effect of larger higher quality trials Shang et al
decided that, based on this subset, homeopathy is a placebo
response The studies that constitute the evidence for the
conclusion of the authors are listed inTable 2
The two sets of trials are incomparable and
heteroge-neous with a pooled OR = 0.88 (95% CI: 0.65–1.19) for
homeopathy Only two homeopathy studies (ref.38,40) could
be matched with conventional studies (56–58) The
homeop-athy group consisted of 8 trials on 8 different indications
Egger warned ‘‘Opinions will often diverge on the correct
method for performing a particular meta-analysis The
robustness of the findings to different assumptions should
therefore always be examined in a thorough sensitivity
anal-ysis’’.59Our sensitivity analysis showed that if Vickers´ trial
on muscle soreness is omitted from the eight largest higher
quality homeopathy trials the overall odds ratio reduces
from 0.88 to 0.80, but remains statistically not significant
(95% CI: 0.61–1.05)
Ineffective unusual treatment
Sensitivity analysis of the higher quality studies showed
one indication with four studies: homeopathic Arnica for
muscle soreness after long distance running.37,49,50,52The
pooled effect of those studies was in favour of placebo,
OR = 1.30 (95% CI: 0.96–1.76) As treatment of healthy individuals is very rare in homeopathic practice this outcome has low external validity to judge the effect of homeopathy as a method The fact that conventional medi-cine is also ineffective for this indication (seeFigure 1) is omitted due to disrupting of matching on indication
The final conclusion The final conclusion that homeopathy is a placebo response (and conventional medicine is not) was flawed
on several grounds:
1 Homeopathy and conventional trials were not comparable
2 Heterogeneity disallows conclusions about effect
3 Sensitivity analysis was missing
4 The cut-off value for larger trials was decisive
How essential were the missing data?
In theBox 1we summarise the conclusions that could only be drawn from post-publication data These data pro-vided all answers to the questions we mentioned above except one: the fact that overall quality was better in homeopathy trials
Another possible outcome Often subjective choices must be made in meta-analy-ses We evaluated the influence of some such choices in this case: the indication ‘muscle soreness’, the cut-off value and the interpretation of quality We did not con-sider exclusion of trials, publication bias, quality bias or other possible bias Table 3andFigure 2show the influ-ence of cut-off values after exclusion of the trials on mus-cle soreness In Table 3some pooled OR and confidence intervals for ‘larger trials’ are given, with and without Linde’s trials discarded by Shang et al If we choose the overall median sample size (n = 65) as cut-off we disre-gard half of all trials This seems a reasonable cut-off value for larger studies
Figure 2shows the cumulated-pooled OR (including the Linde trials omitted by Shang et al) if we increase step by step the number of included higher quality studies, starting with the largest two
Adding the four ‘Linde trials’ does not change effects, but shifts the most unfavourable cut-off value from the 7th to the 10th trial Discarding the indication ‘muscle sore-ness’ lowers the pooled OR from 0.88 to 0.80 Depending
on the choice of cut-off value the OR varies between 0.72 and 0.80 Cochrane reviews are typically based on 8–10 studies60and are homogenous as to indication Linde con-cluded that there was insufficient evidence for only single condition.2But Shang et al’s 110 trials included 8 of home-opathy for acute upper respiratory tract infection, with no evidence of quality bias and a considerable effect size,
OR = 0.36, 95% CI: 0.26–0.50 For muscle soreness after marathon running homeopathic Arnica is clearly not effective
Table 2 Larger higher quality studies, according to Shang et al
Indication Homeopathy Conventional
medicine Diarrhoea Jacobs 40
N = 116
Kaplan 56
N = 256 Treatment of
influenza
Papp.38N = 334 Nicholson.57
N = 319
de Flora.38
N = 248 Prevention of
influenza
Rottey.36
N = 501 Plantar warts Labrecque.39
N = 162 Weight loss Schmidt.23
N = 208 Muscle
soreness
Vickers 37
N = 400 Headaches Walach 42
N = 98 Sinusitis Weiser 41
N = 104
Post operative
infection
Crowley.
N = 273
5 569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710
Trang 6PROOF
Discussion
We found indications that Shang et al’s hypothesis and
hence its conclusion was sensitive to subjective choices
and the influence of one indication and that the subsets on
which the conclusions were based were not comparable
The missing data were of crucial importance, exploring
these data seriously undermines the conclusion that
home-opathy is a placebo response
We calculated a number of possible pooled odds ratios
for the effect of homeopathy as a method, excluding one
indication for which homeopathy is ineffective, to show
the isolated position of Shang et al’s hypothesis Taking
all pooled odds ratios indicates an effect, but in some cases
the confidence interval includes 1.0, depending on the
definition of ‘larger trial’ The quality of the whole set
(p = 0.03) and quality in small studies (p =0.003, Fisher
exact test) are better in homeopathy than in conventional
medicine The placebo hypothesis is also falsified if only
higher quality studies are considered The comparison of
homeopathy and conventional medicine was flawed by
the inclusion of unpublished trials only in the homeopathy group and possibly by excluding trials The conclusion of Shang et al is based on one subgroup of 8 trials on 8 differ-ent indications, not on a comparative analysis Our sensitiv-ity analysis showed one indication and a specific cut-off value play a decisive role in the final conclusion Our addition of Linde’s best trials did not alter Shang’s overall results, but increased the number of (larger) higher quality trials, and the effect of the eight largest higher quality trials became significant (OR = 0.73; 95%CI: 0.59–0.91)
Small effects can be clinically relevant In a meta-analy-sis of statin treatment and the occurrence of haemorrhagic stroke Vergouwen et al found an effect of OR = 0.88 This is the same OR as in Shang’s final subset.61 In the case of statins the 95% confidence interval is below 1.0 because the pooled sample size is large Insufficient sample size and heterogeneity could easily lead to type II error (false negative) if OR = 0.88
Shang et al compared homeopathy and conventional medicine in terms of quality and effect, but originally matched trials on indication Quality and effect are
Table 3 The influence of cut-off values for ‘larger studies’, excluding ‘muscle soreness’, with or without Linde’s best studies, using random
effects analysis
Largest higher quality trials,
Shang’s quality criteria,
without muscle soreness
6 trials, cut-off n = 104 OR = 0.73 (0.59–0.91) 0.0051
7 trials, cut-off n = 98 OR = 0.80 (0.61–1.05) 0.1087
8 trials, cut-off n = 81 OR = 0.75 (0.58–0.96) 0.0246
13 trials, cut-off overall median OR = 0.66 (0.49–0.89) 0.0058 Largest higher quality trials,
without muscle soreness, + 3
of Linde’s best quality studies
8 trials, cut-off n = 116 OR = 0.73 (0.54–0.98) 0.0336
9 trials, cut-off n = 104 OR = 0.72 (0.56–0.94) 0.0158
10 trials, cut-off n = 98 OR = 0.78 (0.59–1.03) 0.0776
11 trials, cut-off n = 81 OR = 0.75 (0.57–0.98) 0.0340
16 trials, cut-off overall median OR = 0.78 (0.57–0.97) 0.0273
Box 1 Conclusions that could only follow from the subsequently disclosed data
Comparison of quality and effect is flawed by inclusion of unpublished trials only for homeopathy Restraining to
published trials, quality was higher in 22% instead of 19% as mentioned by Shang
Judgement of quality is different from other analyses in at least four trials
The predefined hypothesis that positive results of homeopathy could be explained by quality bias in smaller trials was
falsified (p = 0.003)
If only higher quality trials are considered, the placebo hypothesis for homeopathy is falsified, OR = 0.76 (95%
CI 0.59–0.99)
The final conclusion was not based on comparative analysis, there was no matching on indication between homeopathy
and conventional medicine
The conclusive subgroup analysis was not rectified because of heterogeneity, it considered 8 trials for 8 different
indications
Cut-off values for larger trials were unexplainably different for homeopathy (n = 98) and conventional medicine
(n = 146) This suggests post-hoc hypothesizing
Sensitivity analysis showed that one indication and the chosen cut-off value for larger trials explained the final
conclusion of statistically non-significant effect
At least one larger higher quality homeopathy trial with positive result was excluded on unclear grounds
Comparative extrapolation of effects was questionable because of publication bias, selection bias, difference in quality
and sample size and difference in safety
6
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852
Trang 7PROOF
interrelated; better quality trials show less effect
Compar-ing effects when quality is not matched is thus questionable
They subsequently selected subgroups matched for quality,
but that disrupts matching by indication In the end the
conclusion about the effect of homeopathy was based on
meta-analysis of a selection of trials Discarding trials for
some indication because no comparable trials could be
found causes selection bias This selection could have had
more influence on the final conclusion (positive or negative)
than our re-analysis of 21 trials could detect The
insuffi-cient matching on indication in the final subsets of 8
home-opathy and 6 conventional trials did not allow any
comparison of effects We assumed matching on indication
when we tested the hypothesis of quality bias in smaller
trials The subgroups of 78 (homeopathy) and 83
(conven-tional medicine) smaller trials are largely but not fully
(82%) matched on indication
Shang et al made the choice to disregard safety This
decreased the relevance of the comparison of effects of
homeopathy and conventional medicine Some
conven-tional treatments in this analysis are not available because
of serious adverse effects
We also performed meta-regression analysis on the 21
good quality trials and found asymmetry in the funnel
plot If we extrapolate the odds-ratios by meta-regression
we see no difference between homeopathy and placebo at
extreme sample numbers We think that this is irrelevant
to this discussion Sterne, Egger and Smith stated that
asym-metry in good quality trials is not caused by bias but by
stronger effects in smaller trials.8This could be interpreted
as proof that the asymmetry in the set of 110 homeopathy
trials is not caused by bias However, from mathematical
statistics it is well known that such meta-regressions are
imprecise, especially when the number of observations is
small Asymmetry of homeopathy trials and conventional
trials cannot be compared, because there is a significant
dif-ference in the number of smaller good quality trials between
homeopathy and conventional medicine Different size in
matched trials also plays a role in asymmetry of the funnel
plot: both homeopathy and conventional medicine are
inef-fective for muscle soreness, but homeopathy is higher in the
funnel plot because the trials are larger The influence of this indication on asymmetry is opposite for homeopathy and conventional medicine One could also question the role
of drugs with strong effects but with serious adverse effects
on asymmetry Three conventional treatments, which have been withdrawn because of serious adverse effects, had large effect sizes and small standard errors and therefore considerable positive influence on the asymmetry and the extrapolated effect of the funnel plot of conventional med-icine Difference in publication bias has also influence on the position of the funnel plot
We did not investigate the influence of subjective choices
on the OR of conventional trials We think that there are methodological objections against comparing effects in this analysis Moreover, the clinical relevance of such
a comparison is low Homeopathy is mostly used after con-ventional medicine failed, so the indication for use is differ-ent Homeopathy is highly valued for its safety The scientific relevance of Shang’s comparative analysis lies
in the comparison of quality Quality of trials was an impor-tant issue in the discussion about proof and implausibility
The fact that the pooled effect of homeopathy excluding the indication ‘muscle soreness’ is positive does not mean that homeopathy is effective for all other indications If the trials on influenza or Jacobs’ trials on diarrhoea are excluded results become statistically not significant As our re-analysis is post-hoc we cannot draw conclusions regarding efficacy
The clinical relevance of trials is not considered in this analysis, but doctors must be interested in Shang’s finding that eight trials showed a substantial effect of homeopathy
in acute upper respiratory tract infections (OR = 0.36, 95% CI: 0.26–0.50), without indications of bias
Conclusion
A review of data provided after publication of Shang et al’s analysis did not support the conclusion that homeopa-thy is a placebo effect There was intermingling of compar-ison of quality and comparcompar-ison of effects, and thus matching was lost The comparison of effects was also flawed by subjective choices and heterogeneity The result
in the subgroup from which the conclusion was drawn was further influenced by the choice of cut-off value for ‘larger’
trials If we confine ourselves to the predefined hypotheses and the part of this analysis that is consistent with the comparative design, the only legitimate conclusion is that quality of homeopathy trials is better than of conventional trials, for all trials (p = 0.03) as well as for smaller trials withn < 100 (p = 0.003)
Acknowledgements
We thank Rainer Lu¨dtke for assisting with the statistics
References
1 Kleijnen J, Knipschild P, ter Riet G Clinical trials of homeopathy.Q3
BMJ 1991; 302: 316–323.
Figure 2 Indicates that all but 3 of 20 possible cut-off values lead
to a significant effect if we consider all higher quality trials The
ex-ceptions are n = 144 (including 6 trials), n = 122 (7 trials) and
n = 98 (10 trials) If the cut-off point is below n = 98 the pooled
results of higher quality studies is consistently statistically
significant.
7 853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962
Trang 8PROOF
2 Linde K, Clausius N, Ramirez G, et al Are the clinical effects of
homeopathy placebo effects? A meta-analysis of placebo-controlled
trials Lancet 1997; 350: 834–843.
3 Cucherat M, Haugh M, Gooch M, Boissel J Evidence of clinical
efficacy of homeopathy - A meta-analysis of clinical trials Eur J
Clin Pharmacol 2000; 56: 27–33.
4 Morrison B, Lilford RJ, Ernst E Methodological rigour and results
of clinical trials of homeopathic medicines Perfusion 2000; 13:
132–138.
5 Ernst E A systematic review of systematic reviews of homeopathy.
J Clin Pharmacol 2002; 54: 577–582.
6 Ezzo J, Bausell B, Moerman DE, Berman B, Hadhazy V Reviewing
the reviews How strong is the evidence? How clear are the
conclu-sions? Int J Technol Assess Health Care 2001 Fall; 17(4): 457–466.
7 Vandenbroucke JP Medical journals and the shaping of medical
knowledge Lancet 1998; 352: 2001–2006.
8 Sterne JAC, Egger M, Smith GD Investigating and dealing with
publication and other biases in meta-analysis BMJ 2001; 323:
101–105.
9 Shang A, Huwiler-Mu¨ntener K, Nartey L, et al Are the clinical
ef-fects of homeopathy placebo efef-fects? Comparative study of
placebo-controlled trials of homeopathy and allopathy Lancet
2005; 366: 726–732.
10 Vandenbroucke JP Homeopathy and the growth of truth Lancet
2005; 366: 691–692.
11 Higgins JPT, Green S (eds) Cochrane Handbook for Systematic
Re-views of Interventions 4.2.5 The Cochrane Library 2005;(Issue 3).
Chichester, UK: John Wiley & Sons, Ltd [updated May 2005].
12 Jonas WB, Kaptchuk TJ, Linde K A critical overview of
homeopa-thy Ann Intern Med 2003; 138: 393–399.
13 Egger M, Smith GD, Altman DG Systematic reviews in health care:
meta-analysis in context 2nd edition BMJ Books, 2001.
14 Boden WE Meta-analysis in clinical trials reporting: has a tool
become a weapon? Am J Cardiol 1992; 69: 681–686.
15 Walach H, Jonas W, Lewith G Letter to the editor Lancet 2005;
366: 2081.
16 Linde K, Jonas WB Letter to the editor Lancet 2005; 366:
2081–2082.
17 Fisher P, Berman B, Davidson J, Reilly D, Thompson T, et al Letter
to the editor Lancet 2005; 366: 2082–2083.
18 Lu¨dtke R, Rutten ALB The conclusions on the effectiveness of
ho-meopathy highly depend on the set of analysed trials J Clin Epid (To
be filled in).
19 Gade J, Thorn P Paraghurt for patients with irritable bowel
syndrome: a controlled clinical investigation from general practice.
Scand J Prim Health Care 1989; 7: 23–26.
20 Humphrey RG, Bartfield MC, Carlan SJ, O’Brien WF, O’Leary TD,
Triana T Sulindac to prevent recurrent preterm labor: a randomized
controlled trial Obstet Gynecol 2001; 98: 555–562.
21 Chan AW, Hjobartsson A, Haahr MT, Gotzsche PC, Altman DG.
Emperical evidence for selective reporting of outcomes in
random-ized trials: comparison of protocols to published articles JAMA
2004; 291: 2457–2465.
22 Wiesenauer M, Gaus W Wirksamkeitsnachweis eines
Homo¨opathi-kums bei chronischer Polyarthritis Eine randomisierte
Doppelblindstu-die bei niedergelassenen A ¨ rzten Aktuelle Rheumatologie 1991; 16: 1–9.
23 Schmidt JM, Ostermayr B Does a homeopathic ultramolecular
dilution of Thyroidinum 30CH affect the decrease of body weight
re-duction in fasting patients? - A randomised Placebo-controlled
dou-ble-blind trial Homeopathy 2002; 91(4): 197–206.
24 Enzi G, Crepaldi G, Inelmen EM, Bruni R, Baggio B Efficacy and
safety of dexfenfluramine in obese patients: amulticenter study Clin
Neuropharmacology 1995; 12: S173–S178.
25 Available from: <http://www.fda.gov/CDER//news/phen/
fenphenpr81597.htm
26 Louviere RL, Upton RT Evaluation of Deladumone OB in the
suppres-sion of postpartum lactation Am J Obstet Gynecol 1975; 121: 641–642.
27 Lacey PH, Dodd GD, Shannon DJ A double blind, placebo con-trolled study of piroxicam in the management of acute muscoloske-letal disorders Eur J Rheum Inflam 1984; 7: 95–104.
28 Available from: <http://www.fda.gov/ohrms/dockets/98fr/ 102998b.pdf>
29 EMEA Press release London: European Medicines Agency Rec-ommends Restricted Use for Piroxicam, 25 June 2007.
30 Grio R, Cellura A, Geranio R, Porpiglia M, Piacentino R Efficacia clinica del tamoxifene nel trattamento della mastodynia premenstru-ale Min Ginecol 1998; 50: 101–103.
31 de Lange-de Klerk ESM Effects of homeopathic medicines on chil-dren with recurrent upper respiratory tract infections BMJ 1994; 309: 1329–1332.
32 Reilly DT, Taylor MA, McSharry C, Aitchison T Is homeopathy
a placebo response? Controlled trial of homeopathic potency with pollen in hay fever as model Lancet 1986;881–886.
33 Hofmeyr GJ, Picconi V, Blauhof P Postpartum homeopathic Arnica montana: a potency-finding pilot study Br J Clin Pract 1990; 44: 619–621.
34 Reilly D, Taylor MA, Beattie NG, et al Is evidence for homeopathy reproducible? Lancet 1994; 344: 1601–1606.
35 Schulz KF, Chalmers I, Hayes RJ, Altman D Emperical evidence of bias Dimensions of methodological quality associated with esti-mates of treatment effects in controlled trials JAMA 1995; 273: 408–412.
36 Rottey EED, Verleye GB, Liagre RLP Het effect van een homeopa-thische bereiding van micro-organismen bij de preventie von griep-symtomen - Een gerandomiseerd dubbel-blind onderzoek in de huisartspraktijk Tijdschr IntegGeneeskunde 1995; 11: 54–58. Q7
37 Vickers AJ, Fisher P, Wyllie SE, Rees R Homeopathic Arnica 30X
Is Ineffective for Muscle Soreness After Long-Distance Running - A randomized, double-blind, Placebo-controlled trial Clin J Pain 1998; 14(3): 227–231.
38 Papp R, Schuback G, Beck E, et al Oscillococcinum in patients with influenza-like syndroms: A placebo controlled double-blind evalua-tion Brit Homeopath J 1998; 87(2): 69–76.
39 Labrecque M, Audet D, Latulippe L, Drouin J Homeopathic treat-ment of plantar warts Can Med Assoc J 1992; 146(10): 1749–1753.
40 Jacobs J, Jime´nez LM, Malthouse S, et al Homeopathic Treatment
of Acute Childhood Diarrhea - Results from a Clinical Trial in Nepal.
J Alternat Complement Med 2000; 6(2): 131–139.
41 Weiser M, Clasen BPE Randomisierte plazebokontrollierte Doppel-blindstudie zur Untersuchung der klinischen Wirksamkeit der homo¨-opathischen Euphorbium compositum-Nasentropfen S bei chronischer Sinusitis Forsch Komplementarmed 1994; 1: 251–259.
42 Walach H, Haeusler W, Lowes T, et al Classical homeopathic treat-ment of chronic headaches Cephalalgia 1997; 17: 119–126.
43 Jacobs J, Jime´nez LM, Gloyd SS, Gale JL, Crothers D Treatment of Acute Childhood Diarrhea With Homoeopathic Medicine: A Ran-domized Clinical Trial in Nicaragua Pediatrics 1994; 93(5): 719–725.
44 Jacobs J, Springer DA, Crothers D Homeopathic treatment of acute otitis media in children: a preliminary randomized placebo-controlled trial Pediatr Infect Dis J 2001;177–183.
45 Hart O, Mullee MA, Lewith G, Miller J Double-blind, placebo-con-trolled, randomized clinical trial of homoeopathic arnica C30 for pain and infection after total abdominal hysterectomy J R Soc Med 1997; 90: 73–78.
46 Wiesenauer M, Ha¨ussler S, Gaus W Pollinosis-Therapie mit Galphi-mia glauca MMW Fortschr Med 1983; 101(17): 811–814.
47 Zell J, Connert WD, Mau J, Feuerstake G Behandlung von akuten Sprunggelenksdistorsionen - Doppelblindstudie zum Wirksamkeits-nachweis eines homo¨opathischen Salbenpra¨parats MMW Fortschr Med 1988; 106(5): 96–100.
48 Bo¨hmer D, Ambrus P Behandlung von Sportverletzungen mit Trau-meel-Salbe - Kontrollierte Doppelblindstudie Biol Medizin 1992; 21(4): 260–268.
8
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051
Trang 9PROOF
49 Vickers AJ, Fisher P, Smith C, Wyllie SE, Lewith GT
Homoeopa-thy for delayed onset muscle soreness - A randomised double blind
placebo controlled trial Brit J Sports Med 1997; 31: 304–307.
50 Jawara N, Lewith GT, Vickers Aj, Mullee MA, Smith C
Homoeo-pathic Arnica and Rhus toxicodendron for delayed onset muscle
soreness - A pilot for a randomized, double-blind,
placebo-con-trolled trial Brit Hom J 1997; 86(1): 10–15.
51 Chapman EH, Weintraub RJ, Milburn MA Homeopathic Treatment of
Mild Traumatic Brain Injury - A Randomised, Double-Blind,
Placebo-Controlled Clinical Trial J Head Trauma Rehabil 1999; 14: 521–542.
52 Tveiten D, Bruset S, Borchgrevink CF, Norseth J Effects of the
homeopathic remedy Arnica D30 on marathon runners: A
random-ized, double-blind study during the 1995 Oslo Marathon
Comple-ment Ther Med 1998; 6(2): 71–74.
53 Stevinson C, Devaraj VS, Fountain-Barber A homeopathic arnica
for prevention of pain and bruising: randomized placebo-controlled
trial in hand surgery J R Soc Med 2003; 96: 60–65.
54 Lepaisant C Essai the´rapeutique en home´opathie: traitement des
tensions mammaires et mastodynies du syndrome pre´menstruel.
Rev Fr Gyne´col Obste´t 1995; 90(2): 94–97.
55 Chapman EH, Angelica J, Spitalny G, Strauss M Results of a study
of the homoeopathic treatment of PMS J Am Inst Hom 1994; 87:
14–21.
56 Kaplan MA, Prior MJ, McKonly KI, Du Pont HL, Temple AR, Nelson EB A multicenter randomized controlled trial of a liquid lo-peramide product versus placebo in the treatment of acute diarrhea in children Clin Pediatr 1999; 38: 579–591.
57 Nicholson KG, Aoki FY, Osterhaus ADME, et al Efficacy and safety of oseltamivir in treatment of acute influenza: a randomised controlled trial Lancet 2000; 355: 1845–1850.
58 de Flora S, Grassi C, Carati L Attenuation of influenza-like symp-tomatology and improvement of cell-mediated immunity with long-term N-acetylcysteine treatment Eur Respir J 1997; 10:
1535–1541.
59 Egger M, Smith GD, Philips AN Meta-analysis: principles and procedures BMJ 1997; 315: 1533–1537.
60 Shang A, et al Author’s reply Lancet 2005; 366: 2083–2085.
61 Vergouwen MD, de Haan RJ, Vermeulen M, Roos YB Statin treatment of hemorrhagic stroke in patients with a history of cerebro-vascular disease Stroke 2008; 39: 497–502.
9 1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094