Researchers have a host of methods at their disposal with which to assess the efficacy of service interventions and to establish the foundations on which evidenced-based practices emerge. Some of these methods, such as the Randomized Controlled Trial (RCT) (arguably the most potent of instruments in the empiricist’s toolbox), have rarely been applied outside the rarefied realm of medical research and were heretofore unavailable to practitioners within the social sciences. In recent years, however, more robust and rigorous analyses of social service interventions have emerged in the research literature. These provide stakeholders with the information needed to render objective appraisals of publicly-funded programs and to direct scarce resources accordingly. Permanent Supportive Housing (PSH) is one of many targets of increasingly sophisticated investigations within the realm of social services, as it is presumed to hold great promise for many of society’s most vulnerable individuals. A survey of research to date suggests this presumption is largely correct, although additional studies are needed to elucidate the mechanisms through which PSH exerts its salutary effects and to identify the populations for whom it may provide the greatest benefit.
It is generally impractical, and often unethical, to apply RCTs or similarly rigorous analyses to interventions that affect human subjects in natural settings, as subjects assigned to “control” groups in RCTs are necessarily denied potentially beneficial interventions available to “treatment” group participants. Nevertheless, some researchers have managed to overcome this challenge and to administer RCTs of PSH through which subjects were assigned to “treatment” groups (i.e., placement in PSH with associated support services) or “usual care” groups (i.e., placement in other temporary or transitional residential settings as are customarily available to homeless and vulnerable persons). One study examined a sample of 405 homeless individuals, 201 of whom were assigned to a treatment group and 204 of whom were assigned to a control group, in order to determine the impact of PSH on treatment group members’ utilization of inpatient hospital and emergency department services (Sadowski et al., 2009). These researchers applied a variety of controls characteristic of RCTs and necessary to mitigate the impact of potentially confounding variables. For instance, subjects were randomly assigned to treatment and control groups, practitioners associated with the project remained “blind” to subjects’ group designations and a variety of demographic variables were evaluated in order to ensure relative uniformity among groups. This study found the provision of PSH and support services resulted in fewer hospital days and emergency department visits among its recipients during an 18-month survey period (Sadowski et al., 2009).
Other investigators offered a qualified endorsement of these findings through a comprehensive review and meta-analysis of the research literature, but they suggest more rigorous examinations are needed to bolster the evidence base for PSH (Rog et al., 2014). These authors cited certain deficiencies in the literature due to methodological limitations associated with naturalistic observation (a mode of investigation commonly used in the social sciences that cannot control for potentially confounding variables as rigorously as RCT or other “laboratory-based” approaches). The authors cited additional concerns about the current state of research on PSH, including inconsistencies in the operational definitions of housing models and their associated service interventions, small sample sizes and ill-defined subject selection criteria, among others. These factors compromise the validity of any conclusions that may be drawn and suggest a need for greater methodological rigor. These deficiencies notwithstanding, the authors concluded a “moderate” level of evidence indicates PSH promotes housing stability and reduces homelessness among its recipients. These findings were consistent among studies they surveyed irrespective of sampling and procedural variations and other methodological differences (Rog et al., 2014).
Another team of investigators employed a sophisticated epidemiological analysis in the development of a predictive modeling tool designed to boost the efficiency with which PSH services are provided (Toros & Flaming, 2018). These authors acknowledge the prevalence of homelessness among individuals with disabilities and a paucity of PSH resources available to them, and they suggest predictive modeling would enable stakeholders to identify individuals for whom PSH might provide the greatest benefit and to allocate scarce resources accordingly. They surveyed demographic data and health and social service utilization records for 57,259 individuals during a two-year period and developed an algorithm that identified 1,000 members of this cohort for whom health and social service expenditures would be greatest in the following year. Inasmuch as predictive modeling is inexact its application invariably entails a tradeoff between “sensitivity” and “specificity.” That is, modeling aims to maximize the percentage of “true positives” (i.e., those who fulfill modeling criteria) and to minimize the percentage of “false positives” (i.e., those who fail to fulfill modeling criteria but are erroneously included in the former category due to natural variations in the data that inform the predictive algorithm). The model developed by these authors achieved a very strong “Concordance Statistic” (C-statistic) of .83. (C-statistics higher than .7 suggest considerable predictive power.) The potential value of predictive modeling becomes clear when considered against the economic toll of chronic homelessness among the most vulnerable individuals. One study of a chronically homeless population in northern California determined only 10% of its population incurred 61% of health and social service costs (Economic Roundtable, 2015). Another study found 20% of shelter users consumed the largest share of health, social and criminal justice services at a similarly exorbitant cost (Ly & Latimer, 2015). In 2012, the New York State Medicaid Redesign Team (MRT) instituted numerous reforms within its Medicaid program, many of which entailed the identification of individuals who incurred the greatest costs and for whom PSH and other community-based interventions would produce favorable outcomes. MRT reforms have been largely successful in reducing expenditures among the most costly cohort of Medicaid recipients, but these reforms would surely have benefitted from sophisticated predictive modeling technologies that accurately identify the subsets of chronically homeless individuals that incur the greatest costs.
As a variety of transformative efforts presently underway aim to replace fee-for-service reimbursement systems with value-based alternatives, payers, policymakers and other stakeholders will seek sophisticated technologies with which to allocate scarce resources to those for whom the need (and potential savings) is greatest. The tools of the empiricists’ trade, including systematic observation, experimentation and predictive modeling, will surely prove invaluable to this endeavor. We should expect them to play an increasingly prominent role in cultivating the evidence basis for PSH and other social service interventions.
Located in Valhalla, New York, Search for Change is dedicated to improving the quality of life and increasing the self–sufficiency of individuals with emotional, social, and economic barriers. We teach the skills needed to choose, obtain and maintain desirable housing, meaningful employment, higher education and productive relationships with family and friends. Our programs and services are focused on individual choices, needs, interests and abilities. The author may be reached at (914) 428-5600 (x9228) or by email at abrody@searchforchange.org.
