Implementation and evaluation

The goal of the process thus far has been to explore and embrace the unique character of your community. Regardless of which strategy you implement once you have completed your action plan, this guide assumes that it will be the first of many projects. A thorough evaluation of the action plan increases the likelihood that the plan (and the evaluation results) will influence future decision-making. This section highlights some helpful objectives to guide you through your evaluation process.

Among many evaluation frameworks, Patton’s Utilization-Focused Evaluation’ (U-FE) framework has a rich history within the field of evaluation and has been adopted by private companies and public organizations around the world.^{footnote 17[17]}

This guide highlights the U-FE framework because it focuses on creating a sense of ownership, credibility and technical confidence among its potential users.

Objectives

• Identify primary users
• Determine the purpose of your evaluation – summative vs. formative
• Identify methods and measurement
• Interpret findings and make judgments
• Develop future AFC plans
• Improve existing AFC action plan

Key skills

• Utilization-focused evaluation skills
• Research, data collection and data analysis
• Report writing

Key tasks

• Establish a direction for monitoring and evaluating the success of the plan
• Determine an appropriate monitoring mechanism
• Specific goals and objectives

Identify Primary Users

Begin by identifying and bringing together your primary users and stakeholders (staff, administrators, program participants) to share in decision-making about the evaluation. You may find that your primary user is an individual or group already involved in your AFC process, although this does not necessarily mean they will be responsible for implementing the project or program being evaluated. In fact, because implementation of your plan will involve a range of stakeholders, each may only be responsible for implementing one strategy. The body implementing a current initiative may therefore have less interest in evaluation results than a stakeholder responsible for a future project may.

Once you have a completed shortlist, you can narrow the scope of potential users and uses by discussing possible evaluation questions (see Appendix V). As this discussion unfolds, consider each candidate in light of the following points that have been adapted from Patton’s evaluation framework:

• Primary users want to answer evaluation questions and care about the answer.
• Primary users want to answer the evaluation questions for themselves, not just someone else.
• Primary users can indicate how to use answers to the evaluation; they can specify the relevance of answers for action.

Determine the Purpose of Your Evaluation — Summative vs. Formative

Before an assessment of the action plan can start, both the evaluator and affected stakeholders need to determine the purpose of the evaluation (for example, goals, program implementation).

Both summative and formative evaluations can be used.

Evaluation throughout Implementation (Formative)

The purpose of a formative evaluation is to assess initial plan goals and ongoing plan activities. The purpose of a summative evaluation is to assess the quality and effect of the plan after it has been fully implemented.^{footnote 18[18]}Perform a formative evaluation using the following:

• Checklists: A simple yet effective reporting strategy to municipal council to demonstrate that priority action items set out in the plan — such as developing resources to inform older adults and their families about housing options — have been fulfilled or are in progress according to the plan’s implementation schedule.
• Focus groups: Speak with older residents of your community in a small meeting to gauge their awareness of, and opinions about, age-friendly initiatives that result from your action plan.
• Needs assessment: Use the needs assessment again to gauge whether your community’s social and physical resources have changed since the first time you conducted the needs assessment.

Evaluation of Quality and Effect After Implementation (Summative)

A summative evaluation uses the same methods (focus groups, needs assessment) and asks the same questions as a formative evaluation, but it takes place after the plan’s time period has ended.^{footnote 19[19]}In addition to the questions you asked during focus group discussions and the needs assessment, use a summative evaluation to address the following general questions:

• Did the action plan meet its goals for change or effect?
• Which plan components were most effective? Which components need improving?
• Were the results worth the cost of particular age-friendly strategies and programs?
• Can those strategies and programs be sustained?

Your AFC movement will involve innovations in built and social environments and adaptations to projects that already exist. Whether you follow a summative or formative evaluation method, you must involve the potential user of the evaluation data at every stage of developing the evaluation. See Appendix V for more detailed information on conducting summative and formative evaluations.

Identify Methods and Measurement

The third part of the U-FE process involves identifying methods and measurements and making design decisions. Your AFC committee will consider a range of possibilities:

• Using qualitative versus quantitative data;
• Using selective or random sampling strategies;
• The degree to which the data collected provides insights into issues/problems (depth) or allows you to make generalizable statements (breadth);
• Threats to the validity and usefulness of the data.

Ultimately, the overriding concern will be the usefulness of the data you collected. In other words, will results obtained from alternative methods be accurate, cost effective and useful?

Interpret Findings and Make Judgments

Once you have collected and organized data for analysis, invite the intended users to actively and directly interpret results and provide recommendations.

Northern Ontario

Dryden’s Age-friendly Community Experience

Implementing Community-Supported Change

What has Dryden done?

Dryden’s Age-Friendly Network is continually looking for innovative ideas to expand upon services within the community. By forming partnerships to work beyond the boundaries of any one organization, the network has been able to creatively address service gaps, improving access to services and quality of life for seniors and caregivers in Dryden. Many improvements have already been made, including:

• The Creation of a Community Service Guide which supports seniors during health transitions, such as the move from hospital to home or vice versa. This innovation was recognized as a leading best practice by Accreditation Canada
• Hosting education sessions targeting caregivers and seniors
• The creation of an Anishenaabe Community Liaison who supports First Nation community residents through health transitions and with traditional health teaching
• The opening of Outreach Nurse Practitioner Clinics with evening hours, which service supportive housing units, long-term care facilities, and First Nations communities
• Expanding the use of Telehealth Ontario in supportive housing and long-term care facilities to reduce ER visits
• The establishment of a regional caregiver support network through Telehealth Ontario

Dryden has risen to the challenge of creating an age-friendly community by establishing true partnerships that have successfully advanced collaborative care with a senior/caregiver focus.

How did Dryden get there?

2009: Dryden Age Friendly System Integration Group/Network was initiated

Patricia Region Senior Services and Dryden Area Family Health Team formed a partnership under a Community Living IADL investment in Dryden

Senior and caregiver survey completed

First senior and caregiver learning series. Continued each year since its inception

2010: Developed a strategic plan

Released a Senior and Caregiver Community Directory

2011: Ontario Telehealth Network proposal submitted and funded

Patricia Region Senior Service Tub room submitted and funded as supportive housing expansion into the community

2012: Patricia Region Senior Service and Grace Haven Adult Day program service integration

2013: Supportive Housing Physiotherapy Program proposal submitted and funded for falls prevention

What are Dryden’s next steps?

The Age-Friendly Network and Steering Committee will continue to meet and concentrate on barriers and gaps in care, providing solution-focused programs and services. Community partners are currently working on senior isolation, palliative care/end of life care, and volunteerism initiatives.

To learn from other community experiences, visit http://healthy.uwaterloo.ca/~afc/community_stories.html

Northern Ontario

Thunder Bay’s Age-friendly Community Experience

Revisiting Plans to Stay On-Track

What has Thunder Bay done?

In their strategic planning framework, the Age-Friendly Thunder Bay Committee stated that one of their goals was “To ensure the sustainability of Age- Friendly Thunder Bay by continuing its organizational development process.” The goal is further explained to continually clarify policies, proposals and roles that are involved in the AFC process. This sustainable mentality has led to the ongoing success of Thunder Bay’s age-friendly plans such as establishing themselves as a member of the World Health Organization’s Global Network of Age-Friendly Cities and Communities. Thunder Bay has many age-friendly features, including a respectful attitude toward seniors, access to pleasant places for walking, numerous volunteer opportunities, and many cultural events and activities.

How did Thunder Bay get there?

2009: Age-Friendly Thunder Bay Committee established

2010: Thunder Bay City Council endorsed working toward the City of Thunder Bay becoming age-friendly

2011: The World Health Organization (WHO) accepted the City of Thunder Bay as a member of the Global Network of Age-Friendly Cities and Communities

2012: City Council endorsed the Thunder Bay Senior Charter

What are Thunder Bay’s next steps?

The Age-Friendly Thunder Bay Committee has received funding from the Ontario Trillium Foundation for a project that focuses on promoting and marketing the age-friendly concept in the community of Thunder Bay. This has resulted in the development and implementation of an age-friendly business toolkit, intergeneration education modules and a public awareness campaign.

For more information, please visit http://cerah.lakeheadu.ca/