Evaluating engagement

You and your team may consider collecting continuous feedback from persons with lived experience to get input on how you can improve their experience during the partnership. Obtaining continuous feedback, and reflecting and taking action on ways to improve, can help you maintain your focus on engaging persons with lived experience (RNAO, 2015). Before seeking feedback from persons, however, you and your team must ask if they wish to participate in evaluating their experiences in partnership with you on your project.

Methods to get feedback:

  • questionnaires
  • interviews
  • small group discussions
  • observations

Guiding questions to help you think about how to evaluate the engagement experience (Alberta Health Services, 2014):

  • How did the contributions of the persons with lived experience impact the project, and have their contributions been communicated to the persons?
  • How were the initial expectations of persons being met at the end of the project?
  • How collaborative was the process?

Steps for designing an evaluation plan

Evaluating the engagement process and its outcomes is an important component to engaging persons with lived experience. Evaluation determines whether the objectives of your engagement initiative are being met.  The perspectives of the persons and of your team should be equal in the design of this evaluation plan. Here are some steps for designing this evaluation plan, provided by Alberta Health Services (2014):

  1. Work with the persons to create an evaluation outline: Design an evaluation timeline early in the engagement initiative that includes routine check-ins (e.g., before and after meetings). Routine check-ins allow you to see if the project is on track for meeting the goals and objectives mutually determined by the team and the persons. These check-ins also allow you to make adjustments to the plan as issues arise.
  2. Determine what needs to be evaluated and how to evaluate it: Decide what questions need to be asked to determine if the objectives of both the engagement process and the project are being met. Determine the best method for evaluation of the data you need (e.g., a survey or an interview).  Ensure the evaluation questions and process are mutually determined by all stakeholders and meet the needs of all members.
  3. Be clear about the data needed to evaluate: Specify the exact data you will need to complete the evaluation (e.g., time-keeping records, expense reports and/or meeting notes) so those involved in the evaluation can be sure to track and record data throughout the project. 
  4. Develop an evaluation plan: Identify key milestones and timelines, as well as who will be involved in the evaluation process and what they will do (e.g., who will conduct the interview; report the outcomes of the evaluation to supervisors, etc.)
  5. Determine what will be done with the evaluation information: Will the results be sent to managers or other stakeholders? If so, clarify which stakeholders, and who will share the information.

Evaluating the engagement process and outcomes

You may wish to evaluate the process of engagement (how the activity unfolded), the outcomes of the engagement (what was achieved), or both.

To evaluate the engagement process, you can look at:

  • Representativeness of the population of interest
  • Early involvement of persons with lived experience
  • Number and level of clearly defined task(s) for persons with lived experience
  • Transparency of decision-making process
  • Clear articulation of roles and responsibilities
  • Satisfaction of persons and team with process
  • Timeliness, participation rate and costs incurred

SOURCES: Alberta Health Services (2014); Capital Health (2011).

Engagement process questions

Below is a table of sample questions to generate feedback from persons and team members on the engagement process:

For persons with lived experience For team members
  • How detailed, complete and easy to understand was the background
    information provided to you?
  • Do you feel you had the right information to take part in the discussion?
  • Overall, how is your experience as a partner?
  • How could your experience as a partner be improved?
  • What would you do differently next time?
  • Any additional comments?
  • How many persons with lived experience participated in the engagement activity?
  • Were the right persons at the table?
  • Did you use any incentives to encourage participation? If so, what incentives did you use?
  • Were engagement costs within budget?

SOURCES: Alberta Health Services (2014); Capital Health (2011); McMaster University (2018).

To evaluate the outcomes of engagement, you can look at:

  • Influence/contribution of persons with lived experience
  • Persons’ experience of being heard and understood
  • Effect on team members’ attitudes towards the engagement experience
  • Engagement goals

SOURCES: Alberta Health Services (2014); Capital Health (2011).

Engagement outcomes questions

Sample questions to generate feedback from persons and team members on the engagement outcomes.

For persons with lived experience For team members
  • Overall, how satisfied are you that your opinions were heard and understood?
  • Overall, how confident are you that your opinions will influence the final decision/outcome?
  • How satisfied are you with the decision/outcome?
  • How satisfied are you with the communication of the decision or outcome?
  • Any additional comments?
  • Did the patient engagement activity or the participation of the persons with lived experience … missing something here…?
  • Were the decision and rationale communicated to the persons?
  • Was input from the persons included in the decision-making process?
  • Was the organizational goal and promise back to persons achieved?
  • What would you do differently next time?

SOURCES: Alberta Health Services (2014); Capital Health (2011); McMaster University (2018).