Benchmarking the Effectiveness of Psychotherapy Treatment for Adult Depression in a Managed Care Environment: A Preliminary Study

Takuya Minami University of Utah

Bruce E. Wampold and Ronald C. Serlin University of Wisconsin–Madison

Eric G. Hamilton PacifiCare Behavioral Health

George S. (Jeb) Brown Center for Clinical Informatics

John C. Kircher University of Utah

This preliminary study evaluated the effectiveness of psychotherapy treatment for adult clinical depres- sion provided in a natural setting by benchmarking the clinical outcomes in a managed care environment against effect size estimates observed in published clinical trials. Overall results suggest that effect size estimates of effectiveness in a managed care context were comparable to effect size estimates of efficacy observed in clinical trials. Relative to the 1-tailed 95th-percentile critical effect size estimates, effec- tiveness of treatment provided in this setting was observed to be between 80% (patients with comorbidity and without antidepressants) and 112% (patients without comorbidity concurrently on antidepressants) as compared to the benchmarks. Because the nature of the treatments delivered in the managed care environment were unknown, it was not possible to make conclusions about treatments. However, while replications are warranted, concerns that psychotherapy delivered in a naturalistic setting is inferior to treatments delivered in clinical trials appear unjustified.

Keywords: benchmarking, effectiveness, managed care, clinical trials, depression

More than a decade has passed since estimating the effect of psychotherapy as it is delivered in natural settings was identified as a critical issue in psychotherapy research (e.g., Barlow, 1981; Cohen, 1965; Luborsky, 1972; Seligman, 1995; Strupp, 1989; Weisz, Donenberg, Han, & Weiss, 1995). Although the benefits of psychotherapy have been investigated in laboratory environments with randomized clinical trials (RCTs) and found to be substantial as early as the late 1970s (Smith & Glass, 1977; also Smith, Glass, & Miller, 1980), surprisingly little is known about the effects of psychotherapy in natural settings. The dichotomy of laboratory and natural settings was emphasized by Seligman (1995), who discrim- inated between efficacy, which is now used to denote the effects of

psychotherapy in RCTs, and effectiveness, which is used to denote the effects of psychotherapy in clinical practice.

The few studies that have investigated effectiveness over the years have provided mixed results, attributed in part to a variety of methodologies used to investigate effectiveness because of diffi- culty in using a randomized control group design in natural set- tings. Notably, three methods have been used to estimate the effects of psychotherapy in natural settings: clinical representa- tiveness, direct comparison, and benchmarking. Clinical represen- tativeness studies, including some of the analyses conducted by Smith et al. (1980), statistically estimate effectiveness from effi- cacy studies, which are based on factors that distinguish natural

Takuya Minami and John C. Kircher, Department of Educational Psy- chology, University of Utah; Bruce E. Wampold, Department of Counsel- ing Psychology, University of Wisconsin–Madison; Ronald C. Serlin, Department of Educational Psychology, University of Wisconsin– Madison; Eric G. Hamilton, PacifiCare Behavioral Health, San Francisco; George S. (Jeb) Brown, Center for Clinical Informatics, Salt Lake City, UT.

Eric G. Hamilton is currently employed by United Behavioral Health, which took over PacifiCare Behavioral Health in a merger.

Part of this article is based on a doctoral dissertation, in partial fulfill- ment of the requirements for a doctorate in counseling psychology from the University of Wisconsin–Madison, completed by Takuya Minami under the guidance of Bruce E. Wampold and Ronald C. Serlin. An earlier version of this article was presented at the 36th annual international

meeting of the Society for Psychotherapy Research, Montreal, Quebec, Canada. Partial funding for this study was provided by the Department of Counseling Psychology, University of Wisconsin–Madison, as a doctoral research award to Takuya Minami. Bruce E. Wampold and George S. (Jeb) Brown have periodically consulted with PacifiCare Behavioral Health (PBH) and, subsequent to its merger with United Behavioral Health (UBH), with UBH. All analyses in this article were conducted indepen- dently of PBH and UBH, and no editorial oversight was exercised by either organization, as per an agreement between PBH and the authors prior to undertaking this project.

Correspondence concerning this article should be addressed to Takuya Minami, Department of Educational Psychology, University of Utah, 1705 East Campus Center Drive, Room 327, Salt Lake City, UT 84112. E-mail: takuya.minami@ed.utah.edu

Journal of Consulting and Clinical Psychology Copyright 2008 by the American Psychological Association 2008, Vol. 76, No. 1, 116 –124 0022-006X/08/$12.00 DOI: 10.1037/0022-006X.76.1.116

116

 

 

settings from clinical trials (e.g., random assignment of patients, use of treatment manuals). Most comprehensive clinical represen- tativeness studies are meta-analyses conducted by Shadish and colleagues (Shadish, Matt, Navarro, & Phillips, 2000; Shadish et al., 1997) that estimated effectiveness of psychotherapy from an original pool of approximately 1,000 independent clinical trials. Although their investigations led them to conclude that the benefits of psychotherapy provided in clinically representative environ- ments are similar to those attained in treatments delivered in clinical trials, their conclusions must be taken tentatively because only about 5% of treatments in these meta-analyses met even minimal criteria for clinical representativeness (Shadish et al., 1997). Estimates of effectiveness were based on statistical estima- tions rather than actual clinical outcomes in natural settings.

The second means of estimating effectiveness involves direct comparisons of treatments tested in RCTs with treatments deliv- ered in natural settings. In these studies, empirically supported treatments (ESTs) or other manualized treatments are transported into natural settings and their pre- and posttreatment effects are compared with treatments that are already being offered in these natural settings, which are known as treatments-as-usual (TAU). Numerous studies have investigated the feasibility of ESTs in natural settings for various disorders, including panic disorder, depression, and substance abuse (e.g., Addis et al., 2004; Merrill, Tolbert, & Wade, 2003; Morgenstern, Blanchard, Morgan, Labou- vie, & Hayaki, 2001).

The results of direct comparison studies have provided mixed results. For example, TAUs conducted in a community-based substance abuse treatment program showed benefits comparable to those of cognitive– behavioral therapy, which was implemented in the same setting (Morgenstern et al., 2001). However, Addis et al. (2004) reported that the delivery of an EST in a managed care environment, notably panic control therapy, attained significantly better outcomes for some variables than TAU did. The mixed results may be due to significant methodological issues. In many studies, training and supervision is provided to therapists in the EST condition, whereas therapists in the TAU condition do not receive any additional training relative to treatments being deliv- ered or the disorder being treated (e.g., Addis et al., 2004). In some studies, there are also significant differences in the dose of treat- ment between ESTs and TAUs (e.g., Verheul et al., 2003). In addition, it is possible that implementation of ESTs produces stronger allegiance and expectancy effects in the EST condition as compared with the TAU condition, because therapists are deliver- ing the EST as part of an experimental arrangement (e.g., Addis et al., 2004). In the youth literature, Weisz, Jensen-Doss, and Hawley (2006) conducted a meta-analysis of comparisons of evidence- based treatments to usual care and found that no study adequately controlled for confounding variables such as setting, therapists, training, and dose of treatment. It may well be that conducting direct comparisons in natural settings by implementing an alter- native treatment produces biased estimates of TAU effectiveness.

One promising method for evaluating psychotherapy effective- ness, without altering any aspect of TAUs and obviating compar- ison to a treatment delivered with favorable conditions (e.g., extra therapist training and supervision), is to use benchmarks created from clinical trials. Specifically, benchmarking allows pre- and posttreatment data in natural settings to be compared with pre- and posttreatment data from clinical trials. For example, in assessing

the effectiveness of TAUs for children and adolescents, Weersing and Weisz (2002) conducted a benchmarking study by comparing the symptom trajectory of depressed youths who were provided TAUs in community mental health centers with aggregated symp- tom trajectories derived from clinical trials. Their study signifi- cantly improved on previous benchmarking studies (e.g., Merrill et al., 2003; Wade, Treat, & Stuart, 1998) by constructing bench- marks on the basis of meta-analysis rather than by using a select number of clinical trials. Benchmarking thus allows for statistical evaluation of TAUs against rigorous standards established in clin- ical trials without altering any aspects of the TAUs delivered in natural settings and without delivering an established treatment with augmentations, such as supervision and training of therapists, in the established treatment condition.

The purpose of the current study was to evaluate the effective- ness of TAUs delivered in a managed health care organization (i.e., HMO) by means of a benchmarking strategy. Specifically, data on adult patients diagnosed with major depressive disorder (American Psychiatric Association, 1994) were statistically compared with benchmarks derived from clinical trials for adult major depression treatment (Minami, Serlin, Wampold, Kircher, & Brown, 2006; Minami, Wampold, Serlin, Kircher, & Brown, 2007). Benchmark- ing was conducted with samples that were progressively matched to the clinical population most commonly observed in clinical trials on the basis of their inclusion– exclusion criteria and antide- pressant medication use. Additionally, we provided indices of relative strength of the observed treatment effect size estimates as compared with treatment efficacy observed in clinical trials.

Method

Participants