Appendix C: Development process

We conducted a needs assessment with six external focus groups, two key informant interviews, and a survey with the expert panel. Our goals were to explore the needs of end users and to gather names of knowledge experts as potential panel members. 67 people working in hospitals, academic institutions, public health institutions, public health agencies, and other health community centres participated.

After concluding the survey, focus groups and interviews, we hosted a full-day in-person discussion, sharing and discussing a summary of key themes that had emerged. See Appendix 1: Methods of the needs assessment of the Leading Change Toolkit™ for more information. 

A concept analysis is a "formal, rigorous process by which a concept is explored, clarified, validated, defined and differentiated from similar concepts to inform theory development" (Xyrichis and Ream, 2008). A concept analysis helps gain a deeper understanding of the concept’s definitions, including all of the uses of the concept: key characteristics, preconditions, outcomes, and measurement tools (Walker and Avant, 2005).

The two toolkit developers generated a list of eligibility criteria for the literature search, in consultation with the expert panel co-chairs and expert panel members of a working group on the concept analysis (see Acknowledgements). We shared the results with the full expert panel for discussion and feedback: see Appendix 2: Table of Eligibility Criteria for Concept Analysis of Social Movement Actions for more information.

The toolkit developers then conducted a preliminary focused literature search to determine key terms in the topic area, with feedback from the expert panel co-chairs, members of the working group and the full expert panel. Information specialists developed a search strategy (see Appendix 3: MEDLINE Search Strategy for the Social Movement Action Concept Analysis).

A second team of information specialists peer-reviewed the search strategy per the Peer Review of Electronic Search Strategies (PRESS) Guideline, to ensure sensitivity and specificity (McGowan et al., 2016). The toolkit developers systematically searched indexed and grey literature databases in the English language, from inception to June 2019 with input by the co-chairs, members of the working group and the full expert panel. Peer-reviewed and grey literature databases, as well as relevant scientific journals, were searched (see Appendix 4: List of Databases, Journals, and Websites Searched for Concept Analysis).

Titles and abstracts screening, and full-text screening were completed independently and in duplicate, by the Toolkit Developers using a set of predetermined eligibility criteria (see Appendix 2: Table of Eligibility Criteria for Concept Analysis of Social Movement Actions). Please refer to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA; Moher et al., 2009) diagram for details on the search and screening process (see Appendix 5: PRISMA diagram for Concept Analysis). We resolved disagreements through consensus; when required, a third team member made the final decision.

The toolkit developers manually searched the references of the included publications to uncover additional articles. Expert panel members also suggested relevant articles for the toolkit developers to consider. The suggested articles followed the title and abstracts, and the full-text screening process  described above.

To determine the commonalities across each description of a social movement in the context of knowledge uptake and sustainability, the toolkit developers extracted all instances of the concept in detail independently and in duplicate. Extracted study characteristics included: the discipline of the published article, the authors’ discipline(s), setting, the purpose (as stated in the publication), the study design, and stated definition(s) of a social movement (if available). We also extracted data on preconditions, key characteristics, and outcomes were also extracted, and resolved any disagreements through consensus.

Results were reviewed, discussed and shaped by the co-chairs, members of the working group and the full expert panel. The expert panel and toolkit developers then engaged in creating a framework of social movement in the context of knowledge uptake and sustainability, over several sessions involving energized discussions and mapping exercises. The insights and lived experiences of social movement approaches shared by the co-chairs, members of the working group and the full expert panel were incorporated in the concept analysis.

The toolkit developers conducted a systematic search of peer-reviewed literature to collect information on implementation science according to the Action Cycle components of the Knowledge-to-Action Framework (KTA; Graham et al., 2006). Informational specialists developed a search strategy according to the aims of the literature review (see Appendix 6: MEDLINE Search Strategy for KTA Framework Literature Review).

The toolkit developers searched CINAHL, MEDLINE, PsycINFO, and AMED databases from inception to September 2019 for relevant reviews of the KTA Framework, with feedback from the co-chairs and expert panel (see Appendix 7: PRISMA Flow Diagram for KTA Literature Review for the included studies of the screening process). The KTA Framework is long-standing and widely used; researchers have published reviews, and reviews of reviews, of it. The reference lists of these reviews were used to identify relevant citations that helped to inform the toolkit developers of each phase of the KTA Framework. Title and abstract, and full-text screening were conducted independently and in duplicate to screen for citations that met the eligibility criteria (see Appendix 8: Table of Eligibility Criteria for KTA Literature Review). The toolkit developers established eligibility criteria based on feedback from the expert panel, resolving any disagreements through consensus. The toolkit developers also manually searched the reference list of included publications and sought input from the expert panel to uncover other relevant publications.

The toolkit developers conducted a systematic search of peer-reviewed publications on tools that mapped on to each phase of the Action Cycle in the KTA Framework. The search was performed using a search strategy developed by the information specialists and reviewed by the co-chairs, members of a working group focused on KTA tools, and the full expert panel (see Appendix 9: MEDLINE Search Strategy for KTA Tools). The databases searched were: CINAHL, MEDLINE, COCHRANE, EMBASE and PsycINFO.

Screening of the KTA tools followed the systematic process described above. To make sure that the search was comprehensive, the toolkit developers screened for titles and abstracts of peer-reviewed articles that mentioned the use of a tool, as well as the development publications of a tool. The number of includes for each stage of screening are reflected in our PRISMA flow diagram: see Appendix 10: PRISMA Flow Diagram for KTA Tools for more information. 

A final list of tools that met the eligibility criteria was circulated to the co-chairs, members of the KTA Tools working group, and the full expert panel for feedback: see Appendix 11: Table of Eligibility Criteria for Selection of KTA Tools for more information. The expert panel also convened meetings to assess the eligibility criteria to the list of tools, discuss any gaps and provide suggestions for additional tools. A citation analysis of the tools was completed to identify publications authored by individuals who are not developers of the tool, and that cited the development paper of each tool. Tools were  typically eliminated if less than two publications used the tool, although there were exceptions made for tools developed within the previous five years. In total, 54 KTA tools were identified through this process.

We developed a KTA tools working group which included one of the co-chairs, five expert panel members and two research assistants from the University of Ottawa. An online search was conducted to locate the 54 KTA tools. If the KTA tool was not found in the initial online search, the research assistants tried to contact the authors of KTA tool development papers, attempting a second contact if the first was not successful. In total, the working group located 42 of the 54 KTA tools.

These 42 KTA tools were first assessed for their pragmatic properties using the PAPERS tool developed by Stanick and colleagues (2021). The PAPERS tool has two parts: objective pragmatic assessment and stakeholder-facing pragmatic assessments. According to Stanick and colleagues, tools are pragmatic if they are deemed acceptable, easy to use, compatible and useful.

The two research assistants completed the objective pragmatic assessments independently. These assessments involved assessing each tool on criteria such as cost and use of accessible language.

The stakeholder-facing pragmatic assessments were completed by two or more stakeholders who worked in health or education settings and who had clinical experience with implementing guidelines. These assessments involved evaluating the tools on criteria such as the tool’s perceived usefulness and ease of completion. After the pragmatic assessments, nine KTA tools were eliminated: three  because they were associated with an unknown cost, five because they were given low pragmatic scores by stakeholders, and one tool because it was found not to be directly applicable to KTA.

Content validity - defined as the extent that the content of a tool reflects the construct that is intended to be measured - of each of the remaining 33 KTA tools was then evaluated. Using an adapted version of a checklist by Mokkink et al. (2010), a research assistant from the University of Ottawa and a research coordinator from the Ottawa Hospital Research Institute assessed the content validity for each of the KTA tools. This assessment was verified by the chair of the KTA tools working group.

The final phase of the KTA tools project involved mapping the 33 KTA tools to the phases of the KTA framework (Graham et al., 2006). A research assistant, an expert panel member, and the chair of the working group conducted this process. Summaries of the pragmatic and content validity properties of the final sample of 33 KTA tools are included in the Leading Change Toolkit™: see Appendix 12: PRISMA Flow Diagram for the pragmatic and psychometric testing for more information. 

Four databases were searched (MEDLINE, Cochrane, CINAHL, EMBASE) for peer-reviewed literature using a search strategy provided by the Information Specialist. The eligibility criteria are summarized in Appendix 13: Eligibility Criteria of Engaging Persons with Lived Experience Literature Review. 

All records retrieved in this search were independently screened according to the eligibility criteria. Title and abstract screening were performed first, followed by full-text: see Appendix 14: PRISMA Flow Diagram for Patients/Persons and Families Literature Review for more information. The former project lead of the Leading Change Toolkit™ performed 10 per cent of the title and abstract screening and full-text screening, using the same criteria. The reviewers met to resolve any discrepancies after the title and abstract screening and again after the full-text screening. Relevant information from the included records was summarized using a data extraction table. This information included: author, publication date, location, study design, purpose, target population, description of the engagement tool, strategy or approach, and description of the implementation of best practice. All data extraction was done independently and examined by a second reviewer. Examples and types of engagement were categorized across a continuum of engagement, adapted from the patient and family engagement framework developed by the American Institutes for Research (2019).

As part of the development process, internal and external user testing was conducted with 48 stakeholders across a wide range of health-service organizations, academic institutions, practice areas and sectors. Stakeholders included nurses, educators, students and persons with lived experience and administrators.

They tested navigation and usability of online sections of the Leading Change Toolkit™, and were asked to fill out a survey on the ease of use, ease of understanding, the flow of navigation, and appearance of the toolkit. This impact survey was created by members of the development team, with input from the co-chairs and members of the co-sponsor team from Healthcare Excellence Canada. The survey used a pragmatic approach to assess users’ experience of the online resource and the outcomes that have been achieved as a result of its implementation.      

Input from stakeholders was incorporated by the toolkit developers and subsequently, RNAO's web editor and web developers. The process and outcomes of the internal and external user testing were reviewed and guided throughout by members of the Evaluation Working Group with feedback also from the full expert panel.

Nineteen expert panel members and the two co-chairs provided feedback on the written content of the Leading Change Toolkit™. Each section of the toolkit represents an action cycle phase (for Knowledge-to-Action), an element (for Social Movement Action Framework), or a segment (for patients/persons and families). Each expert panel member was randomly assigned at least two written sections of the toolkit.

In the feedback survey, expert panel members were asked to assess the ease of understanding, clarity, appropriateness and usefulness of the toolkit. Written feedback was also sought from expert panel members about content that was most and least helpful, and any additional suggestions to improve the content. The two toolkit developers summarized and presented the feedback to the expert panel, and sought clarification in a further meeting of the expert panel.

The toolkit developers then incorporated the feedback and addressed any questions in a new draft of the Leading Change Toolkit™. This version of the toolkit was submitted to the co-chairs for a review of all sections. Co-chairs’ feedback for the full draft of the toolkit was incorporated before publication. 

Website development was executed in these five phases:

Phase 1: Systems analysis

The Leading Change Toolkit™ team met with RNAO’s Information Management and Technology (IM&T) team to gain a deeper understanding of the website functionality, requirements and specifications. Both teams reviewed the scope of the toolkit against the website functionality to ensure a seamless end-user experience. The IM&T team gave advice on the best ways to implement and host the Leading Change Toolkit™  website within the RNAO main site structure.

Phase 2: Planning and design

The Leading Change Toolkit™ development team worked with an RNAO graphic designer to create a logo, hero images and other graphics, branding, templates and colours for the website. The IM&T team worked with the designer to develop mock-up pages incorporating these creative concepts,  scoping the requirements and components required to develop and display the content and providing input and feedback on the visual elements.

Phase 3: Development and upload of content

The IM&T team integrated the toolkit design to ensure fit with the current RNAO web infrastructure. They also developed elements and components to allow the easy navigation and display of the content, developed, integrated and modified modules to enhance website functionality, and provided demos and training to web editor and toolkit developers. Once the Leading Change Toolkit™  development team and expert panel finalized content, it was shared with the web editor for upload. The web editor worked closely with the IM&T team and the graphic designers to ensure that the site functionality and design were maintained. Each page was uploaded, reviewed and approved by the development team and the IM&T team.

Phase 4: Integrating feedback from user testing

The IM&T team made revisions to the navigation, functions and flow of the Leading Change Toolkit™ website after receiving feedback from user testing (see #7 "User Testing", above). The Leading Change Toolkit™ development team, the IM&T team, and graphic designers held meetings to review all feedback from user testing to inform improvements that were required prior to the launch of the website.

Phase 5: Monitoring, updating and maintenance

The Leading Change Toolkit™ is a living resource. In addition to regular monitoring and maintenance, it will continue to be modified and updated as new content becomes available. The Leading Change Toolkit™  Development Team welcomes continued feedback and suggestions from end users to improve user experience and engagement.