Evaluation

By 2028, the vision is for the NIHR Applied Research Collaborations will be recognised as international leaders in knowledge mobilisation KM ensuring evidence is meaningfully integrated into public health, health, and social care decision-making. The work of the ARCs will drive policy, practice, and system change, improving health and social care.

To understand if this vision has been achieved, and to provide valuable insights and learning from this NIHR funded programme a full evalution synthesising regional learnings across NIHR ARCs and translating them into nationally actionable insights has been designed, and include the following key elements.

1. Working Accross ARCs to Synthesise Regional Evidence

The team  are taking multi‑level case study design approach to enable cross‑regional synthesis to examine:

• Broker‑level activity (what KM Fellows do), 
• ARC‑level capability and infrastructure, and 
• National‑level KM coordination and shared resources. 

This allows for systematic comparison of regional activity and identification of transferable learning across contexts. Key to the cross-ARC evaluation is creating a structured data collection processes and metrics.

We are collating insights from all ARCs using four complementary methods: 

• Self‑reported impact narratives – KM Fellows across ARCs will document project aims, knowledge being mobilised, engagement actions, contextual factors, and observed outcomes using a shared national template. This standardisation supports aggregation and comparison across ARCs.
• Semi‑structured interviews and focus groups - Interviews with KM Fellows, ARC Leads, partner organisations, and national stakeholders capture nuanced regional variation and illuminate mechanisms that explain why certain approaches succeed in specific contexts. 
• Documentary analysis of logic models – We will use three iterations (from year 1, year 2 and year 3) of focussed logic models from each ARC to track how regional KM theories of change mature over time. Their common structure means the team can synthesise patterns in mobilisation strategies, engagement actions, enabling conditions, and impacts. During  the initial data collection  in year 1, ~40 completed logic models were submitted which has enabled the evalution team to develop a taxonomy of approaches. 
• National stakeholder workshops - Interactive workshops bringing all ARCs together to sense‑check interim findings, sharing regional insights, surfacing tacit mechanisms, and co‑producing a coherent set of national lessons.

We are using a template analysis, followed by matrix analysis, to compare KM Fellow behaviour, ARC‑level capability, and outcomes across regions, to produce a structured synthesis showing: 

• Common facilitators of impact across ARCs
• Patterns of variation linked to organisational context
• Mechanisms that can be generalised for national application
• Specific contextual dependencies that inform scale‑up strategies.

2. Evaluating Outcomes and Impacts 

The  evaluation is guided by the SPIRIT‑ME [1] framework, which provides a multi‑domain structure that captures: 

• Catalysts (triggers for evidence use) 
• Capacity (individual, organisational, and system-level) 
• Engagement actions (push, pull, linkage & exchange, and system-level) 
• Outcomes (instrumental, tactical, conceptual, imposed, capability, connectivity, culture) 
• Impacts & sustainability (realisation of outcomes, enduring effects, maintenance work, unintended consequences) 

This ensure that both proximal, and distal outcomes are captured,  to look at developing National‑level learning and impact. 

• More info about PROXIMAL OUTCOMES

  Using the SPIRIT‑ME framwoek, the team are looking to evaluate changes in: 

  • Skills, expertise, and KM capability
  • Quality and density of regional and national KM relationships
  • Influence on decision‑making (instrumental/tactical/conceptual)
  • Changes in culture and attitudes toward evidence use.

  These outcomes are captured through interviews, narratives, and documentary analysis. 

• More info about DISTAL OUTCOMES

  Over time, team are looking to evaluate: 

  • Improvements to services or pathways 
  • Increased sustainability of evidence‑informed practices 
  • System‑level changes such as strengthened cross‑ARC networks 
  • Unintended or adverse consequences (e.g., political tensions, role ambiguity).
• More info about NATIONAL LEVEL LEARNING & IMPACT

  The evaluation team are synthesissing ARC‑level insights into:

  • National best practice principles for knowledge brokering
  • Recommendations for commissioning and supporting multi‑level brokering
  • Evidence of mechanisms that produce transferable impact
  • Lessons on sustaining KM capability beyond individual projects.