Alternative Medicine comes with many claims of great success. Some therapies offered are good, some do not do much and others are dangerous. How do we know if a therapy is good and really works? As we have learned, testimonials, often used to promote therapies, are really not proof of effectiveness. The fact that many problems naturally improve over time, even when no therapy is applied, and the fact that just doing something can lead to improvement even when the therapy physically has no way to improve things, can make it difficult to determine if a therapy actually brings about improvement. So, how can we know we are not being fooled by the appearance of a therapy’s effectiveness?
What makes for good evidence? This is not a simple question to answer. Here is my attempt. Often what people are attempting to achieve is to basically feel better. Although feeling better is a reasonable goal, as presented in the last blog posting, Alternative Medicine – Evidence: Subjective vs Objective, evidence based on feelings is not a very strong kind of evidence. Whether, the evidence of the usefulness of a therapy is subjective and based on feelings or objective and measured independently of how a persons feels, to say a therapy is good or bad, better or worse, beneficial or not beneficial, there always needs to be some kind of comparison.
Questions to ask as you evaluate the evidence presented for the effectiveness of a therapy include,
– Is the use of the therapy compared to not using the therapy?
– Has the use of the therapy been studied in a reasonably large enough number of people?
– Are natural history and the placebo effect taken into account?
– Are comparisons done in a way as to minimize biases?
1. Good evidence compares using the therapy to not using the therapy, rather than just reporting on the use of the therapy. A comparison to the effects of using to not using a therapy is a called a controlled trial. The human body has an amazing ability to heal. The natural history of most illnesses is that they improve over time. For example, since most throat infections get better over time, when the use of a therapy is associated with improvement it is possible that the improvement is due to time and has nothing to do with the therapy. To demonstrate effectiveness of a therapy there needs to be some kind of comparison trial, comparing the use of the therapy to not using it. Claims that promote the effectiveness of a therapy that present no such comparison trial are suspect.
2. Good evidence acknowledges the fact that trials comparing using to not using a therapy can still be misleading and attempt to minimize the possibility of presenting misleading or false results.
3. Good evidence attempts account for all who receive the therapy, including reporting on both good and bad outcomes. Usually, the more people involved in a trial the more likely it is that the results of the trial will report the truth about a therapy. Evidence that does not attempt to provide this whole picture may be interesting but is not proof and may even be deceptive. Consider a therapy given to 1000 people, where 10 have a great outcome, 100 find no difference, and 890 get worse. Would you think it wise to promote such a therapy? What if a report on the same therapy provides 10 glowing testimonials on the great outcomes achieved? Would you be tempted to consider the therapy? Evidence that only reports on a small group of people can be misleading.
4. Good evidence attempts to take into account the fact that the placebo effect can bias results. As discussed in … , among those who believe a therapy is likely to work, one third will report improvement even when the therapy physically does nothing. To account for the placebo effect comparisons, or controlled trials. often include a group of people who receive the therapy and a group who receive a placebo (something that looks and feels like the therapy and has not physical effect on the body). Such comparisons are called placebo controlled trials.
A placebo is something that looks and feels like the therapy but is unable to physically do anything. Because the placebo effect is linked to believing that the treatment will work, it is important that either the placebo group be led to believe that they are getting the real therapy (such a study would be unethical) or that neither the treatment group nor the placebo group know which they are receiving. Hiding the fact of whether one is receiving the real therapy or the placebo is called blinding. It is believed that it is best to blind both subjects of the trial (those receiving either the therapy or the placebo) and those running the trial. This is called double blinding. This is where the name double blind placebo controlled trial comes from. Although, such a trial may be the most likely to determine if a therapy is truly effective, to set up a large double blind placebo controlled study is time consuming and expensive. So, we often find we are left to depend on less rigorous studies. Some comparisons without a placebo control group may still be useful. Since the placebo effect gives a positive effect about 1/3 of the time, if a study is done without using a placebo control group, and the therapy group gets a response much greater than 33%, one could conclude that the therapy is likely effective.
5. Good evidence tries to include a large number of people in any comparisons. Looking at a therapy’s effect on only a small group of people can give misleading results. Consider the flipping of a coin where there is a 50% chance that the result is heads any time the coin is flipped. If you flip 100 coins you should not be surprised if around half of the flips result in a head and half in a tail. It would be shocking if most of the flips came up heads. But, if you only flip 3 or 4 coins it would not be shocking if all of the flips resulted in heads? Likewise, if a therapy’s claim is supported by a study of only a small number people, the results could lead to false claims.
6. Good evidence tries to compare similar groups of people when testing using or not using a therapy. Consider a therapy that claims to promote facial hair growth, where 90% of those treated grew facial hair while only 10% did not. This appears impressive until you find out that the treatment group were all men and the comparison group are all women. One way to try to solve this problem is by profiling all the people in the treatment group and trying to choose people with similar profiles for a matched comparison group. (The matched group is know as the control group and such comparisons are known as case-controlled trials). Another way is by taking a large group of people who have not yet been treated and then trying to randomly divide them into a treatment group and a control group. Hopefully, by using a random process with a large enough group of people will result in the treatment group having a similar profile to the control group. It can be a problem to let the treatment group and the control group chose themselves. Those who believe a therapy is likely to work are also more likely to join the treatment group. Because they believe it will work, they will be more likely to have a positive placebo effect. This can lead to what is called selection bias..
-this blog posting is part of a series – the next in the series, [This blog posting is part of a series – the next in the series will hopefully be posted in the near future.]
-the first posting in this series – Alternative Medicine – Important Questions
-for all postings on Alternative Medicine – Alternative Medicine Postings Page