Ilana Nossel1, Sarah Piscitelli,1, Susan Essock2, Franco Mascayano2, Lisa Dixon1; 1New Yoirk State Psychiatric Institute, 2Columbia University
OnTrackNY, New York State’s coordinated specialty care (CSC) program, has grown rapidly to 22 sites, highlighting the importance of scalable fidelity assessments. Our approach evolved in phases, using three information sources. We first relied on findings from high-touch training activities. Second, we used client- and program-level data on care processes and outcomes solicited quarterly from teams which are tabulated centrally, generating reports that teams receive. Third, building on Addington’s FEP Fidelity Scale, we added site visits, including chart reviews, team meeting observation and staff, participant and family member interviews. Our scale has 25 domains, comprised of 94 sub-items, 48 from data and 46 from site visit. Each domain includes one “critical sub-item” essential for fidelity. We completed a pilot study (2 sites) focusing on assessing the incremental value of site visits. Both programs had high fidelity, meeting expectations in 23/25 domains. Data items addressed team functioning and outcomes over the past year. Raters identified items that were sub-threshold, required clarification, or had gaps; these were validated and supplemented with information from the site visit. Ten domains were heavily or solely scored based on site visit including assertive outreach, treatment planning, client preferences/ shared decision making, and trauma. Eligibility and enrollment, crisis services, care processes, metabolic monitoring, and peer services had critical items that relied on site visit. Final scores reflect all information sources; raters discussed discrepancies between sources and reached consensus. Conclusion: Site visits can be a useful adjunct to regular summaries of administrative data used to monitor implementation fidelity.
