We are in a new world today compared to where we were only a few decades ago. Due to the availability of the World Wide Web, Facebook, Twitter, and other online sources, you can gather information about just about anything. You can read blogs, scientific journal articles, regular journal articles, and people’s first hand accounts with whatever topic or problem that you may have.
There are many people both online and in the world itself who tout the success of their treatment for a variety of issues. These people could be researchers, doctors, life coaches, treatment centers, educators, politicians, counselors, and more. In addition, with the rise of Facebook and other online forums, it is easy to ask for advice from others and hear about other people’s experiences with issues that you’ve been having, especially when it comes to things like medications or treatments for a certain condition. But how valid is the information that you are being given?
Let’s take, for example, a few situations that I have heard about concerning the validity of reporting success rates for chemical dependency treatment centers. Chemical Dependency Treatment Centers commonly run statistics and data analyses in order to test the success of their treatment program and to promote their company. Some treatment centers, though, have performed shady studies with a skewed sample, but due to most people’s inability to understand the importance of analyzing populations, they get away with it. Let me give you a hypothetical scenario to help you understand this problem.
Imagine that your adult son or daughter is considering going to chemical dependency treatment. You naturally want to find a program with a high success rate but only seem to find programs in your area with success rates of about 40 or 50%. You get discouraged. Then you discover a treatment center that has a 75 % success rate, and that this success rate has been discovered not just by the center itself but by actual scientific researchers from a nearby institution! Sounds promising, right?
Think again. You are absolutely sure that you want your child to go to this treatment center, and tell a friend of yours about it. You then sign up your child to attend in about a week. However, your friend is well versed in scientific research and decides to check on the scientific study in question. What she finds is that although the program’s calculation of success rate is accurate, it is based off of patients who actually completed the entire two year program. But wait a second, what about the number of people who did not complete the study? Your friend then looks through the methodology of the journal article to find out the percentage of people that actually complete the entire program but cannot find any information about it. She calls the chemical dependency center who reports to her that only 50 % of people actually complete the program and that they don’t have any statistics as to how the people fared who only partially completed their treatment program or dropped out right away. She reports this back to you, and explains that this study is using a skewed sample to misreport the success rates of this treatment program. She explains to you that this is basically shady business of both the treatment center and the researchers involved. You wonder why researchers who should be well versed in statistics would overlook such a basic problem. Weird, huh? You then call up the treatment center angrily and let them know that your child will not be going there. Whew. It sure is nice to have a friend that can interpret research studies!
The truth is, though, that this kind of skewed sampling happens all of the time. It happens within treatment centers, with drug trials, with doctors trying to tout a new kind of treatment, with places that tout an alternative treatment for a particular disease, and even with counselors or psychologist who are proponents of certain methods of counseling.
It’s actually not that hard to read and interpret scientific journals or to simply consider success states success rates. The main thing that you want to look for is whether or not the entire population of what is being measured is actually represented in the study or in the success rate that a person or organization claims. Another thing that you will want to look for when interpreting is whether there are possible factors that would influence a study, theory, or idea that have not been taken into account. These influencing factors are called confounding factors, and when they are not controlled for statistically, they threaten the entire validity of a study, the claim of success of a particular program or support group, or even of how valid it is that everyone in your support group has had problems with XYZ drugs. Confounding factors can also invalidate theories within science itself.
Let’s take, for example, an online support group. Maybe you have depression, and are about to start Prozac. You’ve heard that it can cause weight gain, sedation and other side effects and are especially worried about weight gain. So you browse the web and go to a couple of Facebook support groups where most people tell you that the drug itself causes weight gain and that they themselves experienced this. You decide, then, that you don’t want to take Prozac or any other SSRI.
Before you do, though, you might want to consider who you are talking to and possible confounds that they did not mention to you. First of all, how do we know that it is the drug itself that caused the weight gain? Does it increase hunger? And if so, did most of these people give into these hunger cravings, or did they continue to maintain their current diet despite it? Researchers have found that there is a possibility that food cravings and increased hunger from SSRIs like Prozac can result in people eating more once they take it, but that this doesn’t happen for everyone. In fact, people who are able to control their diets gain less or no weight after taking Prozac than those who do not control their diets.
If you were to look back at the answers that you got in the support group, then, how many of them mentioned this possible connection? Did they eat more, or did they try to diet despite increased hunger? And which ones had increased hunger versus which ones simply had weight gain? It turns out that most of these people didn’t specify whether or not they had effectively controlled their diet or even if they’d experienced an increase in hunger at all. You yourself have always been good about keeping your diet in check even if you had hunger cravings due to depression, but because you did not consider these possible variables and how they fit in with you as an individual, you still decide not take any SSRIs (drugs similar to Prozac) even though your doctor thinks that it will be a good fit for you. You start looking for alternatives for depression treatment, but continue to stay depressed for a year until you try Zoloft. Despite an increase in hunger, you only gain about five pounds due to your stellar diet and wish that you had considered SSRIs earlier.
The other problem with online forums and groups is that they are not necessarily representative of the overall population in question. For example, the question posed about whether or not Prozac causes weight gain tends to attract people who have experienced weight gain from Prozac. But what about the people who haven’t experienced it, or the ones who just experienced it a little? These people are less likely to be searching for this topic on google or potentially even notice posts about this topic, so you might hear much from them or at all. If a person has a problem with a treatment or medication, they are usually more likely to share about it and ask for advice as well, and because of this sometimes certain issues or problems might appear to be more common simply because they are discussed more often in a certain support group.
You also must consider that there is a certain population of people who tend to not just join support groups, be on Facebook, or use the internet in general and/or to seek out information from others. People who tend to join support groups could be more extroverted, could be isolated, may need more social support than they currently have, may like to give advice to others, or may just like to participate in communities or groups in general. But what about everyone else? What could they have to contribute to your question or issue? Could this the population in your support group or forum be skewed in some way? These are all possible factors when deciding on the validity of what you hear in support groups, especially ones that are on the internet.
The same, though, is true for in person support groups. Let’s address one of my favorite topics, which is looking at confounding variables within the claims that are made by members of the support group Alcoholics Anonymous. It is believed among most people who find success in AA that working the steps has led them to get and stay sober. But wait… how do we know this to be true? Everyone says it, right, so it must be true! The truth is that there may be other factors that are present before or after a person works the steps that are predictive of sobriety. For example, what if people who work the steps are the ones who are highly motivated to get sober in the first place? What if people who stay in AA and work the steps want to get sober more badly than the ones who stay in AA and don’t work the steps? And for the person who relapsed four times, finally worked the steps, and says that this saved their life: What if that fourth time you came into AA was the first time that you were actually motivated to stay sober? What if your sobriety is due to motivational factors that were present before you even worked the steps? Wouldn’t it make sense that a person who goes to meetings a few times a week, works the steps, and is of service might be more motivated to stay sober than the person in a meeting who does not? There are many other variables, too, that could preclude a person to work the steps that also may be associated with whether or not to stay sober. So the question here really is, what came first, the chicken or the egg?
When analyzing whether or not the steps result in a person staying sober, it is also important to factor in things like group think and group dynamics. Have you ever noticed that a person who works the steps gets more praise than one who does not? How do we know that this reward and conditioning of praise and the social support that comes with it is not the driving force as to why a person who works the steps stay sober rather than working the steps itself? And have you ever noticed that people who do not work the steps are treated quite differently than those who do? We must recognize that in AA people who work the steps versus those who don’t are treated differently, which means that group dynamics within Alcoholics Anonymous itself may be a primary factor in whether or not a person stays sober.
Furthermore, if you are in AA you probably hear accounts of people who relapse due to not working the program and then come back to AA and report to you on this. Thus you conclude that working the program leads to sobriety. But what about the people who leave the program and stay sober? Most of them don’t come back to AA meetings to report to you that they are staying sober. Thus there are many sampling biases when it comes to discerning the information that you hear in any support group including AA.
I tend have skepticism when I read people’s informational blogs. I wonder ask myself questions like: Are they getting their information from other people’s blogs, from articles on the internet, or from scientific journals? Or is this person just reinterpreting another person’s words? I sometimes wonder if information can get passed from blogger to blogger to blogger to the point where the original source becomes completely forgotten or overlooked. It is always important to consider where the source of your information comes from, because even professional online writers may not be getting their information from primary sources. Always be alert when reading anything on the internet and always consider how or if the person is a credible source of information and where they got their information from.
While it is always beneficial to do your own research on any given treatment and to look for things like success rates and other statistics, it is important too to consider the methods used to calculate the numbers that you are being given. Furthermore, it is important to consider all the possible factors influencing any given situation or information before you make a decision or conclusion about a treatment, support group, a new drug or medication, a doctor or provider, and many other things. You can even consider situations like what types of people are inclined to write those google reviews about a clinician or a restaurant before you go out to eat or see that five star doctor. Remember, too, that even though everyone says that something is true, this doesn’t mean that it is actually true. Think critically and don’t be afraid to be a skeptic. Most successful researchers are.