Implementation outcomes

Image
Evaluate outcomes

Why is it important to evaluate how the interventions were implemented and what impact this had on the outcomes?

Given that a practice change or an intervention will not be effective if it is not implemented well, it is important to measure the implementation outcomes that were taken to achieve the desired practice change or interventions (Proctor, 2020). An evaluation of the implementation outcomes can give you information about whether the intervention or practice change was implemented as intended.

Distinguishing implementation effectiveness from intervention effectiveness is critical for implementation success. When a practice or intervention does not get properly used, or not used at all, it is important to determine why the change attempt failed. This can be due to two key factors:

  1. Intervention failure – i.e., when the implementation intervention was ineffective. For example, the depth and breadth of the content of an education series were insufficient to support the development of knowledge to achieve the practice change, or the sessions included only minimal time for hands-on practice to develop the skills to achieve the practice change.  
  2. Implementation failure: The implementation intervention was effective but was not deployed adequately. For example, the education series effectively addressed the competencies needed to achieve the practice change however the sessions were poorly attended as they were held during vacation season. Or the session was effective but staff were not accepting of the intervention or felt it was not sufficiently applicable to their setting.

Alternatively, when an implementation intervention successfully supports knowledge use and a practice change, it can be helpful to evaluate the strategies that were used and how they were used effectively – i.e., what steps were taken to support the practice change and how they were provided or deployed.  

The evaluation of implementation outcomes answers the following questions:

  • Why does or doesn’t the practice change or intervention work in my setting?
  • Did we implement what we intended?
  • What worked? What didn't? 

Implementation outcomes serve three functions (Khadjesari et al., 2017). They:

  • indicate implementation success (e.g., a practice change has been achieved).
  • serve as indicators of implementation processes.
  • serve as important intermediate outcomes for service and clinical outcomes.

Identifying and measuring implementation outcomes will allow change teams to understand the implementation processes. You can use it to determine the following regarding an implementation intervention:

  • whether it occurred or not,
  • whether it did or did not get used to support the practice change, or
  • whether it did or did not provide the intended benefit to the person/patient.

Remember: It is difficult to measure the effectiveness of an intervention if your implementation didn’t go as planned! In these cases, understanding the implementation outcomes may assist in explaining why things did or did not work.

  • There are seven key implementation outcomes that serve as effective indicators of the success of an implementation intervention (Proctor et al., 2011). These seven implementation outcomes are described in the table below and describe how other variables such as staff, the local context, or the setting impact outcomes.   
  • Implementation outcomes can be interrelated, and occur in a logical, but not necessarily linear sequence. Outcomes earlier in the sequence can contribute to the implementation strategies that will be used later (Glasgow et al., 2019).

Types of implementation outcomes:  

SOURCE: Proctor et al., 2011. 

Acceptability
Implementation outcome Related terms Description Strategies for data collection
Acceptability Satisfaction The extent to which persons/patients perceive a treatment, service, practice, or innovation to be agreeable, palatable, or satisfactory.

Surveys

Interviews – Semi-Structured

Feasibility
Implementation outcome Related terms Description Strategies for data collection
Feasibility Actual fit or utility, suitability for everyday use, practicability The extent to which the practice change can be successfully used or carried out within a given setting.

Surveys

Chart audits

Appropriateness
Implementation outcome Related terms Description Strategies for data collection
Appropriateness Perceived fit, relevance, compatibility, suitability, usefulness, practicability The extent to which the practice change can be successfully used or carried out within a given setting.

Surveys

Chart audits

Cost
Implementation outcome Related terms Description Strategies for data collection
Cost Marginal cost, cost-effectiveness, cost-benefit The financial impact of the practice change. May include costs of treatment delivery, cost of the implementation strategy (e.g. an education session, new equipment), cost to the person/patient, and cost of using the service setting.

Administration data

Adoption
Implementation outcome Related terms Description Strategies for data collection
Adoption Uptake, utilization The intention, initial decision, or action to try or employ the practice change. Adoption may also be called “uptake.”

Surveys

Interviews – Semi-Structured

Observation

Fidelity
Implementation outcome Related terms Description Strategies for data collection
Fidelity Delivered as intended, adherence, integrity, quality of program delivery The degree to which an implementation strategy was delivered as prescribed in the original protocol or as intended by program developers. May include multiple dimensions such as content, process, exposure, and dosage.

Observation

Checklists

Self-reports

Reach
Implementation outcome Related terms Description Strategies for data collection
Reach Level of institutionalization, spread, service access The extent to which the practice change is integrated within a setting. 

Case audits

Checklists

Sustainability
Implementation outcome Related terms Description Strategies for data collection
Sustainability Maintenance, continuation, durability, incorporation, integration, institutionalization, sustained use, routinization

The extent to which a recently implemented practice change is maintained and/or institutionalized within a setting’s ongoing, stable operations.

Checklists

Questionnaires

Case audits

Interviews – Semi-Structured

CLICK HERE TO DOWNLOAD THE "IMPLEMENTATION OUTCOMES PLANNING WORKSHEET"

This worksheet includes guiding questions to support your change team to start thinking about how you can evaluate your implementation plan according to the seven implementation outcomes listed in the table above.

Remember: The more of these implementation outcomes that are being met, the more likely the practice change will be implemented successfully (Barwick et al., 2020).

Image
linkages

Accelerate Your Success: When the intervention or practice is successfully implemented and achieves its intended outcomes, consider building momentum for this change! Momentum is the energy needed to initiate and sustain change by change agents, such as champions and others. These change agents commit to work together to make and sustain the change. Check out the Social Movement Action Framework’s ‘Momentum’  to learn more about how you can build momentum so that the new intervention or practice can be better integrated in routine practice.