A structured approach for selecting outcomes in the evaluation of whole-system reforms
What were we trying to do?
We wanted to develop a clear and practical approach to assess the complete impact of large-scale health and care reforms.
Traditional evaluations often rely on a narrow set of outcome measures, which can overlook wider system changes and unintended consequences. Our aim was to design a method that could help policymakers and researchers capture the broader effects of system-wide reforms in a transparent and consistent way.
Why was this important?
Health systems are complex, and reforms often target multiple parts of the system at once.
To evaluate these changes, we need to look at more than just one or two measures. However, outcome selection can be inconsistent, open to bias, and fail to reflect the changes being made without a structured approach.
So, there was a clear need for a generalisable method that could support whole-system evaluations across different settings and policies.
How did we do it?
We proposed a five-step process for selecting a comprehensive, relevant, and policy-aligned set of outcome measures. This approach is based on the World Health Organization’s (WHO) Health System Performance Assessment framework and was designed to be repeatable across settings.
We then used our approach to select outcomes to evaluate the devolution of healthcare services in Greater Manchester to show how the method could be applied in practice.
In this example, we reviewed over 500 potential outcome measures from national outcome frameworks and mapped information on 466 local interventions using a documentary analysis. In total, we identified 99 outcomes that were suitable for evaluating the whole-system impact of our case study.
Findings
The method supported a more balanced and transparent approach to outcome selection.
The process demonstrated how complex reforms affected many parts of the system, from access and quality of care to broader social outcomes.
Only around half of local interventions had clearly specified outcomes in the application of the method, highlighting a gap in how system-level initiatives are planned and evaluated.
Our approach provides an alternative application of the WHO Health System Performance Assessment framework to assist the evaluation of future whole-system initiatives.
Resources
Who did we work with?
This work was led by the Health Organisation, Policy and Economics (HOPE) group at The University of Manchester.
We collaborated closely with NHS Greater Manchester Integrated Care and the Greater Manchester Combined Authority throughout.
Funding information
This research was funded by The Health Foundation.
More information
Senior Programme Lead
Mike Spence
Michael.spence@manchester.ac.uk