Appendix C: Development process

The aim of the Leading Change ToolkitTM is to support practice change through the use of two complementary frameworks that apply the concepts of the SMA Framework with the action cycle phases of the KTA Framework, to enhance and accelerate change. The toolkit was developed as follows:

1. Needs assessment

A needs assessment was conducted with six external focus groups, two key informant interviews, and a survey with the expert panel. The needs assessment aimed to explore the needs of end-users of the Leading Change Toolkit™ and to gather names of knowledge experts as potential panel members. A total of 67 individuals participated in the needs assessment. These individuals worked in hospitals, academic institutions, public health agencies, or other health community centers (Appendix 1: Needs Assessment Methods). These processes were followed by a full-day in-person discussion where a summary of key themes was shared and discussed.

2. Concept analysis of the Social Movement Action Framework

A concept analysis is ‘a formal, rigorous process by which a concept is explored, clarified, validated, defined and differentiated from similar concepts to inform theory development’ (Xyrichis & Ream, 2008). A concept analysis helps gain a deeper understanding of the concept’s definitions, including all of the uses of the concept: key characteristics, preconditions, outcomes, and measurement tools (Walker & Avant, 2005).

The two Toolkit Developers generated a list of eligibility criteria for the literature search, in consultation with the expert panel co-chairs and expert panel members of a working group on the concept analysis (see Acknowledgements). In addition, the results were shared with the full expert panel at a panel meeting for discussion and feedback (Appendix 2: Table of Eligibility Criteria for Concept Analysis of Social Movement Actions). The Toolkit Developers conducted a preliminary focused literature search to determine key terms in the topic area, with feedback from the expert panel co-chairs, members of the working group and the full expert panel. Information specialists developed a search strategy (Appendix 3: MEDLINE Search Strategy for the Social Movement Action Concept Analysis). A second team of information specialists peer-reviewed the search strategy, as per the Peer Review of Electronic Search Strategies (PRESS) Guideline, to ensure sensitivity and specificity (McGowan et al., 2016). The Toolkit Developers systematically searched indexed and grey literature databases in the English language, from inception to June 2019 with input by the co-chairs, members of the working group and the full expert panel. Peer-reviewed and grey literature databases, as well as relevant scientific journals, were searched (Appendix 4: List of Databases, Journals, and Websites Searched for Concept Analysis).

Titles and abstracts screening, and full-text screening were completed independently and in duplicate, by the Toolkit Developers using a set of predetermined eligibility criteria (Appendix 2: Table of Eligibility Criteria for Concept Analysis of Social Movement Actions). Please refer to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA; Moher et al., 2009) diagram for details on the search and screening process (Appendix 5: PRISMA diagram for Concept Analysis). Disagreements were resolved through consensus; when required, a third team member made the final decision. The Toolkit Developers manually searched the references of the included publications to uncover additional articles. Expert panel members also suggested relevant articles for the Toolkit Developers to consider. The suggested articles followed the title and abstracts, and the full-text screening process as described above.

To determine the commonalities across each description of a social movement in the context of knowledge uptake and sustainability, the Toolkit Developers extracted all instances of the concept in detail independently and in duplicate. Extracted study characteristics included: the discipline of the published article, the authors’ discipline(s), setting, the purpose (as stated in the publication), the study design, and stated definition(s) of a social movement (if available). Data on preconditions, key characteristics, and outcomes were also extracted, context of knowledge uptake and sustainability, the Toolkit Developers extracted all instances of the concept in detail independently and in duplicate. Extracted study characteristics included: the discipline of the published article, the authors’ discipline(s), setting, the purpose (as stated in the publication), the study design, and stated definition(s) of a social movement (if available). Data on preconditions, key characteristics, and outcomes were also extracted, with disagreements resolved through consensus.

Results were reviewed, discussed and shaped by the co-chairs, members of the working group and the full expert panel; together, the expert panel and Toolkit Developers engaged in the creation of a framework of social movement in the context of knowledge uptake and sustainability, over several sessions involving energized discussions and mapping exercises. The insights and lived experiences of social movement approaches shared by the co-chairs, members of the working group and the full expert pacontext of knowledge uptake and sustainability, over several sessions involving energized discussions and mapping exercises. The insights and lived experiences of social movement approaches shared by the co-chairs, members of the working group and the full expert panel were incorporated in the concept analysis.

3. Literature review of the Knowledge-to-Action Framework

A systematic search of peer-reviewed literature was conducted by the Toolkit Developers to collect information on implementation science according to the Action Cycle components of the Knowledge-to-Action Framework (KTA; Graham et al., 2006). Informational specialists developed a search strategy according to the aims of the literature review (Appendix 6: MEDLINE Search Strategy for KTA Framework Literature Review). The Toolkit Developers searched CINAHL, MEDLINE, PsycINFO, and AMED databases from database inception to September 2019 for relevant reviews of the KTA Framework, with feedback from the co-chairs and expert panel (Appendix 7: PRISMA Flow Diagram for KTA Literature Review for the included studies of the screening process). Because the KTA framework is a longstanding and widely used framework, researchers have published reviews, and reviews of reviews, of the KTA Framework. The reference lists of these reviews were used to identify relevant citations that helped to inform the Toolkit Developers of each phase of the KTA Framework. Title and abstract, and full-text screening were conducted independently and in duplicate to screen for citations that met the eligibility criteria (Appendix 8: Table of Eligibility Criteria for KTA Literature Review). Eligibility criteria were established based on feedback from the expert panel. Disagreements were resolved via consensus. The Toolkit Developers also manually searched the reference list of included publications, sought input from the expert panel (to uncover other relevant publications).

4. Systematic search of tools that operationalize the KTA Framework

The Toolkit Developers conducted a systematic search of peer-reviewed publications on tools that mapped on to each phase of the Action Cycle in the KTA Framework. The search was performed using a search strategy (Appendix 9: MEDLINE Search Strategy for KTA Tools) developed by the Information Specialists and subsequently reviewed by the co-chairs, members of a working group focused on KTA Tools, and the full expert panel with feedback provided. The Toolkit Developers searched CINAHL, MEDLINE, COCHRANE, EMBASE, and PsycINFO. Screening of the tools (hereafter called ‘KTA tools’) followed the same systematic process as the methods of conducting the literature review of the KTA Framework, listed above. To ensure that the search was comprehensive, the Toolkit Developers screened for titles and abstracts of peer-reviewed articles that mentioned the use of a tool, in addition to the development publications of a tool. Please refer to the PRISMA flow diagram (Appendix 10: PRISMA Flow Diagram for KTA Tools) for the number of includes for each stage of screening. The final list of tools that met the eligibility criteria were circulated to the co-chairs, members of the KTA Tools working group, and the full expert panel to obtain feedback about the inclusion of each tool (Appendix 11: Table of Eligibility Criteria for Selection of KTA Tools). Expert panel meetings were held to assess the eligibility criteria to the list of tools, discuss any gaps, and provide suggestions for additional tools. A citation analysis of the tools was completed to identify publications authored by individuals who are not developers of the tool, and that cited the development paper of each tool. Tools were eliminated if less than two publications used the tool. An exception was made when the tool was developed within the past five years. In total, 54 KTA tools were identified.

5. Assessment of KTA tools

A KTA tools working group consisting of five expert panel members, one of the co-chairs, and two research assistants from the University of Ottawa was developed. An online search was conducted to locate the 54 KTA tools. If the tool was not found in the initial online search, the research assistants contacted the authors of tools’ development papers. The authors were contacted a second time if they did not respond to the initial email. In total, 42 of the 54 tools were located.

The 42 KTA tools were first assessed for their pragmatic properties using the PAPERS tool developed by Stanick and colleagues (2021). According to Stanick and colleagues (2021), tools are pragmatic if they are deemed acceptable, easy to use, compatible, and useful. The PAPERS tool has two parts: 1) objective pragmatic assessment and 2) stakeholder-facing pragmatic assessments. The objective pragmatic assessments were completed independently by the two research assistants and involved assessing each tool on criteria such as cost and use of accessible language. The stakeholder facing pragmatic assessments were completed by two or more stakeholders, who worked in health or education settings and who had clinical experience with implementing guidelines. The stakeholder facing pragmatic assessments involved evaluating the tools on criteria such as the tool’s perceived usefulness and ease of completion. After the pragmatic assessments, nine KTA tools were eliminated; three tools were eliminated because they were associated with an unknown cost, five tools were eliminated because they were given low pragmatic scores by stakeholders, and one tool was eliminated due to not being directly applicable to KTA. Content validity, defined as the extent that the content of a tool reflects the construct that is intended to be measured, of each of the 33 KTA tools remaining after the pragmatic assessments were then evaluated. Using an adapted version of a checklist by Mokkink et al. (2010), a research assistant from the University of Ottawa and a research coordinator from the Ottawa Hospital Research Institute assessed the content validity for each of the KTA tools. This assessment was verified by the Chair of the KTA tools working group. The final phase of the KTA tools project involved mapping the 33 KTA tools to the phases of the KTA framework (Graham et al., 2006). This process was carried out by a research assistant, an expert panel member, and the chair of the working group. Summaries of the pragmatic and content validity properties of the final sample of 33 KTA tools are included in the Leading Change Toolkit™.  Appendix 12: PRISMA Flow Diagram for the pragmatic and psychometric testing

6. Literature review on ‘Engaging persons with lived experience

Four databases were searched (MEDLINE, Cochrane, CINAHL, EMBASE) for peer-reviewed literature using a search strategy provided by the Information Specialist. The eligibility criteria are summarized in Appendix 13: Eligibility Criteria of Engaging Persons with Lived Experience Literature Review

All retrieved records were independently screened according to the eligibility criteria. Title and abstract screening were performed first, followed by full-text. Please see Appendix 14: PRISMA Flow Diagram for Patients/Persons and Families Literature Review. The Former Project Lead of the Leading Change Toolkit™ performed 10 per cent of the title and abstract screening and full-text screening, using the same criteria. Both reviewers met to resolve any discrepancies after the title and abstract screening and again after the full-text screening. A data extraction table was used to summarize the pertinent information from the included records. The information collected included: author, publication date, location, study design, purpose, target population, description of the engagement tool/strategy/approach, and description of the implementation of best practice. All data extraction was done independently and examined by a second reviewer. Examples and types of engagement were categorized across a continuum of engagement, adapted from the patient and family engagement framework developed by the American Institutes for Research (2019).

7. User testing

As part of the development process of the Leading Change Toolkit™, internal (mainly RNAO Implementation Science managers, guideline development methodologists) and external (mainly BPSO® leads, nurses, individuals in academic institutions) user testing were conducted with 48 stakeholders across a wide range of health-service organizations, academic institutions, practice areas and sectors. Stakeholders included nurses, educators, students and persons with lived experience and administrators. Stakeholders were given access to navigate the functionality and asked to assess the usability of the drafted sections of the Leading Change Toolkit™. They were asked to fill out a survey on the ease of use, ease of understanding, the flow of navigation, and appearance of the toolkit. Input from stakeholders was incorporated by the Toolkit Developers and subsequently, the Web Editor and Web Developers. The process and outcomes of the internal and external user testing were reviewed and guided throughout by members of the Evaluation Working Group with feedback also from the full expert panel.

An impact survey of the Leading Change Toolkit™ was developed by members of the Leading Change Toolkit™ Development Team with input by the co-chairs and members of the team from the co-sponsor Healthcare Excellence Canada. Using a pragmatic approach, the impact survey assesses the users’ experience of the online resource and the outcomes that have been achieved as a result of its implementation.      

8. Expert Panel feedback

Nineteen expert panel members and the two co-chairs provided feedback on the written content of the Leading Change Toolkit™. Each expert panel member was randomly assigned at least two written sections of the toolkit. Each section of the toolkit represents an action cycle phase (for Knowledge-to-Action), an element (for Social Movement Action Framework), or a segment (for patients/persons and families). In the feedback survey, expert panel members were asked to assess the ease of understanding, clarity, appropriateness, and usefulness of the toolkit. Written feedback was also sought from expert panel members about content that was most and least helpful, and any additional suggestions to improve the content. The two Toolkit Developers summarized and presented the feedback to the expert panel, and sought any clarification in a subsequent expert panel meeting. Both Toolkit Developers incorporated the feedback and addressed any questions in the subsequent draft of the Leading Change Toolkit™. This version of the toolkit was submitted to the co-chairs for a review of all sections. Co-chairs’ feedback for the full draft of the Leading Change Toolkit™ was incorporated prior to publication. 

9. Website development

Website development comprised of five phases, as shown below:

Phase 1: Systems analysis

The Leading Change Toolkit™ team met with RNAO’s Information Management and Technology (IM&T) team to gain a deeper understanding of the website functionality, requirements and specifications. Both teams reviewed the Leading Change Toolkit™ scope against the website functionality to ensure a seamless end-user experience. The IM&T team advised the best way to implement and host the Leading Change Toolkit™ website within the RNAO web infrastructure, and advised on the preferred method to implement the website.

Phase 2: Planning and design

The Leading Change Toolkit™  Development Team worked with the graphic designers to create a logo, hero images and other graphics, branding, templates, and colours for the website. The IM&T team worked with the graphic designers to develop mock-up pages, ensuring the creative concepts and branding were maintained. The IM&T team scoped the requirements and components required to develop and display the content and provided the designer with input and feedback for the visual elements required.

Phase 3: Development and upload of content

The IM&T team integrated the design provided by the designer to ensure fit with the current RNAO web infrastructure, developed elements and components to allow the easy navigation and display of the content as per the design creation of the navigation structure; developed, integrated and modified modules to enhance website functionality; and provided demos and training to web editor and toolkit developers. Once the content was finalized by the expert panel and the RNAO Leading Change Toolkit™  Development Team, it was shared with the web editor for upload. The web editor worked closely with the IM&T team and the graphic designers to ensure the content was uploaded in a timely manner, and that the site functionality and design were maintained. Each page was uploaded, reviewed, and approved by the Leading Change Toolkit™ Development Team and the IM&T team.

Phase 4: Integrating feedback from user testing

The IM&T team made revisions to the navigation, functions and flow of the Leading Change Toolkit™ website after receiving feedback from user testing (as described in #7. User Testing). The Leading Change Toolkit™ Development Team, the IM&T team, and graphic designers held meetings to review all feedback from user testing to inform improvements that were required prior to the launch of the website.

Phase 5: Monitoring, updating, and sustainment

In addition to regular monitoring and sustainment, the Leading Change Toolkit™, as a living resource, will continue to be modified and updated as new content becomes available. The Leading Change Toolkit™  Development Team welcomes continued feedback and suggestions from end-users to continuously enhance the user experience and engagement.