The Ontario Telemedicine Network (OTN) is a leader in innovative virtual care models that enhance access to, and quality of, health care. OTN is an independent, not-for-profit organization funded by the Ontario Ministry of Health and Long Term Care (MOHLTC). OTN manages one of the busiest and most comprehensive telemedicine programs in Canada. Using advanced information and communication technologies, and electronic medical devices, OTN supports the delivery of clinical care, professional education, and health-related administrative services across Ontario. OTN’s services include real-time video-based clinical consulting and education, and remote patient monitoring (RPM); this service is more commonly known as Telehomecare (THC).
OTN’s ongoing six-month cycle Telehomecare program has remotely linked more than 7,000 patients with chronic obstructive pulmonary disease and congestive heart failure with healthcare providers who provide remote monitoring and health coaching in 10 of the 14 LHINs. As a Best Practice Spotlight Organization, Telehomecare has implemented a number of best practice guidelines. As technological advances are quickly becoming a mainstay of healthcare, a gap in program evaluation and outcome measurements has been identified and this may hinder the program’s sustainability and future expansion. These efforts are required in order to ensure all Ontarians are equally able to access and benefit from the program, and can expand to other programs or chronic disease management strategies. With the aim of enhancing internal capacity to lead evaluation, Telehomecare’s Clinical Practice Manager – Julie Kwan was accepted to the RNAO Advanced Clinical Practice Fellowship (ACPF) program to learn evaluation research skills and develop an evaluation framework.

The ACPF program undergoes a rigorous application process and accepts only a limited number of applicants. It is designed to promote nursing knowledge and expertise through the knowledge translation and skills from a mentor. In turn, it is hoped that the benefits will improve patient care and outcomes in Ontario. This report will describe and outline the ACPF experience by Julie Kwan, the fellow and with collaboration from her mentor, Dr. Kerry-Anne Hogan who teaches research methods at the University of Ottawa, and is the Chief Research Officer leading evaluation at the Winchester District Memorial Hospital. We will also highlight the successes in having a mentor living miles away.

Small scale results to date have been promising for Telehomecare, with demonstrated improvements in metrics such as use of emergency department (ED) and hospital admissions. However, the value has not yet been established on a program evaluation level. The overall learning goal central to this fellowship was to acquire the knowledge and skill in evaluation research methodologies to empower myself to analyze, drive, and lead quality initiatives.

Evaluation research is a type of methodology that differs from other evaluation methodologies and does not require any ethics review. Evaluation research is a type of research that investigates if a program has met the identified goals and has a specific purpose. I was able to take this opportunity to use this learning to enhance my knowledge to evaluate daily operations, and to implement and drive quality improvement initiatives. Evaluation research involves analyzing and synthesizing knowledge; understanding, interpreting, and applying theory and frameworks. In this fellowship, I learned to activate, develop and refine my knowledge and skills through a number of planned activities with my mentor which will be described in the updated learning plan.

My fellowship with Dr. Kerry-Anne Hogan provided me the ability to comprehend, analyze, and synthesize knowledge; apply theory and research and develop an organizational evaluation framework to recommend to OTN; and be able to replicate these skills throughout the organization. With continuous changes in the health care environment, continuing education is required in establishing a solid foundation the advancement of the nursing profession. During this short time, I gained skills and knowledge to implement evaluation research methodologies, which I will be able to influence key stakeholders and policy makers in an evidence based manner to ensure programs and future expansions are sustainable, efficient, cost-effective, and of high quality which in turn will decrease the burden to the health care system on a macro level. The overarching learning focus was to solidify our internal evaluation infrastructure and processes. Having the opportunity of being a fellow in the ACPF assisted me in achieving this goal.

Although there were geographical challenges between myself and my mentor, the method of communication was not a barrier. Due to the distance, the mentoring relationship had to be flexible and creative. Using OTN’s personal videoconferencing system, email, telephone, texting, and face-to-face meetings we are able to effectively engage in knowledge translation activities, and uphold a project plan through our bi-weekly progress check points ensuring our efforts remained on track, and the sharing of materials was focused and targeted towards the identified learning objectives.
This opportunity has given me the foundation to continue to my leadership competency through translating the information I gained and providing recommendations to OTN. With close collaboration I had with my mentor, we developed an evaluation framework and logic model that will help evaluate and measure success of OTN programs corporately. This will aid in the long-term monitoring of various programs and set standards within the organization. In addition, a culture of quality and evaluation will be promoted across OTN. Creating a recommended evaluation framework and template tool will be valuable in advancing practice in nursing for the future and promote the use of evidence-based decision making.
With these new found skills and knowledge, I have the affinity to lead and develop evaluation research processes within programs in the future. The guidance and knowledge that I learned from

Dr. Hogan will be instrumental in process improvements for clinicians, aid in translation of evaluation content to senior leadership, and ultimately impact the province to provide sustainable and accessible care provincial wide.

Since OTN represents a new way of delivering health care, evaluation research is required to determine what evidence-based practice should be incorporated. For my fellowship, I conducted a literature review on research evaluation models and frameworks and used the information to create a set of questions to interview an external evaluation group, and develop a logic model and a framework. This competency augmented my leadership role in my quality portfolio and to provide valuable contributions to the senior management team producing a comprehensive corporate view of our patient care issues on a number of projects.

With the growing trend of using accurate, reliable, and relevant data to support key decision making, continuing education will allow me to gain the knowledge, leadership skills, and confidence to further the nursing profession in my preferred field of clinical informatics. The change and challenges will be great and expertise is required to facilitate correlations between best practices and policy development. A solid foundation in evaluation research is necessary to improving patient care delivery. My experience with the ACPF has provided me with the experience to mentor with an individual who is an expert in the field and also an advocate of lifelong learning with a passion in nursing care.

At Telehomecare we have learned many lessons since the beginning of the ACPF journey, and have greatly appreciated the expertise and support from Dr. Hogan, and both of our supporting organizations (OTN and WDMH). We look forward to more opportunities to enhance and scale our efforts of evaluation for the future.

Evaluation is crucial for the current COPD and CHF program where lessons learned and analysis can be leveraged to improve and sustain future program delivery. Thus patients with chronic diseases will be affected tremendously and Telehomecare has the opportunity to impact this population in an optimistic manner.

As health care becomes more complex, whether through resource and funding constraints or increased acuity of illness and rise in chronic diseases, the system needs more evidence-based information to make a greater contribution to health care solutions. This includes evaluation research. With more nurses having enhanced knowledge, skill, and expertise in a variety of areas, we can work together to improve the current health care system.

In order to improve ongoing quality within a program, the use of accurate data and analysis is required for evaluation. The leadership team and colleagues at OTN have been fully supportive and encouraged me to learn evaluation skills. The result is that I am now confident and I can support my colleagues in the role as a mentor. Building mentorship capacity at OTN, has great potential to translate into practice to positively affect our front line stakeholders such as nurses, primary care providers and the patients.

Although it is difficult to predict a project’s impact, evaluations are critical to the evolution of health care programs, allowing stakeholders to recognize what is effective and sustainable and what is not. The lessons we learned are able to be shared with others and have the potential to assist multiple stakeholders in the world of health care. At OTN, we recognize that health care technologies are incredibly expensive to implement and our resources are dwindling, thus today’s health care system requires more accountability for health care spending, and increasingly, stakeholders are demanding information about both the actual and future value of these projects. Evaluations are able to justify investment in projects and drive the evolution and uptake of telehealth care practices.

In summary, this opportunity has inspired me to meld my passion for nursing informatics, promote research evaluation, and advance this new specialized area of nursing. I am also able to contribute to increased awareness of how important evaluation plays a vital role in monitoring risk and quality issues and will seek mentorship opportunities through nursing leadership.