Evaluation

Page content: Evaluation | Indicators for planning | Impact evaluation | Narrative evaluation | Dissemination | Evaluation skill development forums

Evaluation

Evaluation in its simplest form is the process of deciding the worth or value of something. This process involves measurement, observation and comparison with some criterion or standard.¹

An evaluation may be conducted for a number of reasons including:

• being accountable to key partners and funding bodies
• ascertaining if things went as expected
• determining whether the program has achieved its goal and objectives (and if not, why not?)
• considering whether something was worth the effort or resources
• future planning and identifying opportunities for improvement
• securing additional or future funding
• fulfilling accreditation requirements and making continuous quality improvements
• contributing to the evidence base for quality integrated health promotion practice.

Programs with a mix of interventions lead to multiple outcomes at varying levels. Different levels of change will occur according to different time scales, depending on the nature of the program and the type of social or health problem being addressed.²

Evaluation should be conducted throughout the life of the program. Planning of the overall program should involve the development of an evaluation plan. The evaluation plan sets out and links the goal, objectives and strategies for the program, along with the data collection methods for evaluation (see Table 1). Indicators are used to guide the collection of data that answer the evaluation questions. They show progress and health change and are used at all levels of evaluation. At the most basic level, there are three types of evaluation: process, impact and outcome evaluation (as illustrated in the PDF below).

For more information download program mangagement for integrated health promotion program linking to impacts and outcomes on page 32 of the IHP Resource Kit: A practice guide for service providers.

For full explanations of levels of evaluation download Section 6 of the IHP Resource Kit: A practice guide for service providers.

Indicators for planning

An indicator is a measure of something that matters, an important issue or policy. In Victoria there is a lot interested in the use of indicators for planning. For more information about community wellbeing indicators and frameworks see the Victorian Community Indicators Project.

top of page

Impact evaluation

This guide has been developed to assist agencies within primary care partnerships (PCPs) to design appropriate impact evaluation methods and develop impact indicators for health promotion programs.

Measuring health promotion impacts: A guide to impact evaluation in integrated health promotion (224kb, pdf)

What is narrative evaluation action in health promotion?

Narrative evaluation integrated within an action research framework is an invaluable adjunct to the statistical evaluation methodologies used to measure health promotion program impacts and outcomes. In combination, these two differing health promotion evaluation perspectives provide stakeholders with varied forms of ‘evidence’ about the whole of a health promotion program: the value of what is being done, why, how and its effects on population health.

The approach serves to illuminate health promotion practice through the creation of stories. These stories can be used to convey the complex realities of health promotion practice as told through the voices of multiple stakeholders. Embedding the stories within an action research cycle facilitates program and practice development.

A manual: ‘Writing narrative action evaluation reports in health promotion- guidelines, resource kit and case-studies’, has been produced for the use of Community and Women’s Health agencies. The manual is designed to assist agencies build their capacity to evaluate health promotion programs and practice using an approach that incorporates narrative evaluation and action research. The manual was first developed in collaboration with pilot Community and Women’s Health Services during the first phase of the Narrative Evaluation in Action Research (NEAR) Project undertaken by the Department of Human Services North and West Metropolitan Region Public Health. The manual was trialled in a broader range of agencies during 2004 and 2005. Following the trial a final version was published in 2007.It includes a ‘Quick Guide’ for ready reference to essential information.

For more information contact:
Karen Goltz
Telephone: (61 3) 9412 2665

top of page

Writing narrative action evaluation reports in health promotion
– guidelines, resource kit, case studies and QuickGuide

The manual: Writing narrative action evaluation reports in health promotion-guidelines, resource kit and case studies, has been produced for the use of Community and Women’s Health agencies. The manual is designed to assist agencies to build their capacity to evaluate health promotion programs and practice using an approach that incorporates narrative evaluation and action research.

The manual was first developed in collaboration with pilot Community Health Services during the first phase of the Narrative Evaluation Action Research (NEAR) project undertaken by the Department of Human Services North and West Metropolitan Region Public Health. The manual was trialled in a broader range of agencies during 2004 and 2005. Following the trial a final version (below) has been published in 2007 as a 2nd edition. It includes the addition of a short handy QuickGuide for ready reference to ‘the essentials’.

Below, you can either download the entire manual as a single document, or choose only the parts that you need. Each of these resources works best if printed double-sided. Please ask for a printed version if unable to do this on your agency printer.

Download entire manual - WARNING: the file below is 868kb.

Writing narrative action evaluation reports in health promotion – manual of guidelines, resources, case studies and QuickGuide (868kb, pdf)

top of page

Download sections of the manual

Introduction

Contents:

• Preface
• Dedication
• Thanks and acknowledgements
• Table of Contents
• Introduction
• What’s in this manual?

NEAR Manual – Introduction (74kb, pdf)

Section 1 - Full guidelines

Contents:

• The story so far…
• Now read on...Writing your own health promotion narrative
• Now you’re ready…Writing your own evaluative health promotion narrative
• OK… Drafting your narrative (or redrafting an earlier one)
• The next exciting instalment…(10kb, PDF)
• From NEAR to FAR to OAR

Section 1 - Manual of guidelines - for writing narrative action evaluation reports in health promotion (367kb, pdf)

top of page

Section 2 - Full resource kit

Contents:

• A - Initial Department of Human Services narrative guidelines 2002-2003
• B - Revised Department of Human Services narrative guidelines 2003–2004/2004–2006
• C - Project information resource (Project description)
• D - Annotated bibliography of readings
• E - An agency reflection on ‘first time’ narratives
• F - Some agencies’ questions to guide narrative evaluation writing
• G - What makes a good narrative?
• H - A planner or checklist of questions for narrative action evaluation writers
• I - A practitioner reflects on the value of narrative action evaluation
• J - Marjorie Oke’s hydrotherapy story (Vale)
• K - Map of levels, voices and styles
• L - Northcote Hydrotherapy Case Study (Extract CDIH collection 1st Edition 1988)
• M – Training your own trainers (DIY) – to extend skills in writing narrative action evaluation
• N – Example of consent form and plain language description (ethics)
• O – Supporting Narrative Evaluation in a Community Health Setting – conference paper

Section 2 - Full resource kit - for writing narrative action evaluation reports in health promotion (354kb, pdf)

top of page

Section 3 - Full case studies

Contents:

• ‘From obligation to inspiration’ – shifting the culture of health promotion in a community health centre
• ‘Will I ever get to see feet?’ - The challenges of integrating wholistic health promotion into the daily practice of allied health professionals in a community based setting
• ‘Health Promotion and Fish Sauce’
• 'It's not just the icing on the cake, it's the actual cake'
• ‘From cross about crossings to safely walking school bus’ - Community Health acting constructively and cooperatively for win-win solutions
• ‘The MIGs Story’ by Jaime Timmerman
• Less is more (‘Throw away that stick !’) by Michael Falloon
• A story of survival – the Collingwood Community Gym by Kristine Olaris, Lina la Guardia, and Melissa Bryan
• ‘So much more than tea and biscuits’ (the MM Young, Pregnant and Parenting group) by Katherine Goulliart
• ‘…there is never enough dancing’ by Sally Camilleri
• Young mums rock! by Joy Free
• The Children’s Art Space at Panch Health Service by Marianne Mahoney

Section 3 - Case studies - for writing narrative action evaluation reports in health promotion (388kb, pdf)

top of page

Section 4 - QuickGuide (summary of steps and key essentials)

Contents:

• Steps to follow
• Essential exercises
• Key resources

Section 4 - QuickGuide - for writing narrative action evaluation reports in health promotion (384kb, pdf)

PowerPoint resources accompanying Section 2 - Resource Kit

Accompanying Resource A

Karen Goltz’s PowerPoint presentation to the DHS Evaluation Skills Forum, May 2005

Narrative Evaluation Action Research in Health Promotion – The ‘NEAR’ Story so far in DHS Northern and Western Metropolitan Regions (323kb, MS PowerPoint)

Accompanying Resource O

Melanie Block and Jaime Timmerman’s PowerPoint presentation to the Health Promotion Annual Conference in Alice Springs, April 2006

Supporting Narrative Evaluation in a Community Health Setting (241kb, MS PowerPoint)

Accompanying Resource M

Kerryn’s ISIS workshop to train new practitioners in the use of narrative evaluation action research (101kb, MS PowerPoint)

Kerryn’s ISIS workshop notes (38kb, MS Word)

Jaime and Melanie’s WRHC in N.E.A.R. Narrative Evaluation Action Research workshop (275kb, MS PowerPoint)

Jaime and Melanie’s WRHC workshop example (Safe and Strong) (47kb, MS Word)

top of page

Dissemination

Dissemination is the active, purposeful process of knowledge transfer. Like evaluation processes, dissemination requires resources, infrastructure and planning and is essential in the feedback link to informing future planning. Reviews of the dissemination processes for findings from health promotion practice indicate that these processes are complex, easily underestimated and often devoid of deliberate and systematic approaches. It is also noted that many health promotion programs in the past were not disseminated widely or findings were disseminated prematurely, limiting the full evidence of effectiveness being recognised or shared.⁴

Key findings and learnings can be disseminated via a range of strategies, such as training through workshops, train-the-trainer and continuing professional education; communication through print; communication through video and computer technologies such as databases of good practice stories, library search systems and websites; personal face-to-face contacts; consultancies; policies, administrative arrangements and funding incentives; committees and other decision-making structures; and collaborative applied research programs.

The stages of dissemination can be summarised as:

1. Providing and seeking information.
2. Persuasion about the relevance and applicability of the innovation or findings.
3. Making a decision to adopt the findings or try the innovation.
4. Changing practices and using the innovation.
5. Sustaining the changed practice.⁵

The Loddon Mallee Regional DHS office recently held a Regional Dissemination Workshop. Below are the presentations from the day.

Dissemination of Research Findings in Health Promotion - Dr Erica James (130kb, MS Powerpoint)

Health Promotion Dissemination Planning - Sue Heward, Public Health Group, Rural and Regional Health and Aged Care Services Division (83kb, MS Powerpoint)

Developing effective PowerPoint presentations - Shane Taylor, Publishing Consultant (682kb, MS Powerpoint)

Factors supporting successful dissemination and uptake (44kb, MS Word)

For more information download Section 6 of the IHP Resource Kit: A practice guide for service providers.

top of page

Footnotes:

1. South Australian Community Health Research Unit (2000), Evaluation workshop session 1, Mid North Regional Health Service, Adelaide.
2. International Union for Health Promotion and Education (2000), The evidence of health promotion effectiveness: shaping public health in a new Europe, Report for the European Commission, Brussels.
3. Adapted from Hawe, P., Degeling, D. and Hall, J. (1990), Evaluating health promotion: a health workers guide, Maclennan and Petty, Sydney.
4. King, Hawe and Wise (1996) From research into practice in health promotion: a review of the literature on dissemination, Sydney. ISBN:1 86451 228 8.
5. NSW Health and Australian Health Promotion Australia (2003), Workshop material used for the Effective Information Dissemination Strategies Workshop, Department of Human Services, Melbourne January 2003.

   top of page

Evaluation skill development forums May, June 2005

In May and June 2005, the Primary and Community Health Branch hosted the Statewide Integrated Health Promotion Evaluation Skill Development Forums. This involved working in partnership with DHS regional offices, Deakin University and other providers. The forums were part of a broader program to support quality health promotion evaluation.

Janet Laverick, Director, Primary & Community Health, Department of Human Services launched the new resource " Planning for effective health promotion evaluation".

Planning for effective health promotion evaluation, May 2005 (445kb, pdf)

A Keynote presentation was given by Professor Elizabeth Waters, Chair of Public Health, Deakin University on "What evidence can help make practice decisions about what works and what doesn't".

What evidence can help make practice decisions about what works and what doesn't (86kb, MS Powerpoint)

Sessions included:
Evaluating physical activity promotion
This session shared the learnings and practical skills for evaluating physical activity promotion across a range of interventions focused on individual, community and population level change. Two organizations discussed their approach to measuring the impact of strength training, community walking programs and modification to the physical environment in terms of participation in and benefits of physical activity.
Chair: Kellie Horton, VICFIT
Panelists: Kirstan Corben, Caulfield Community Health Service, John Sawtell, Castlemaine District Community Health Service.

Strength training program at Caulfield Community Health Serivce, 30 May 2005 (440kb, MS Powerpoint)

Stregnth training assessment form with instructions (75kb, MS Word)

Strength training evaluation form, August 2004 (97kb, MS Word)

Evaluating physical activity workshop, June 2005 (184kb, MS Powerpoint)

top of page

Evaluating mental wellbeing and social connectedness
Many agencies and PCPs have identified goals and objectives related to improved mental wellbeing and social connectedness in their integrated health promotion plans. While knowing how to work towards these outcomes has been a challenge in some cases, in nearly all cases, agencies and PCPs have struggled to identify ways of evaluating these programs. This interactive workshop looked at some of the key questions that agencies would want answered about these programs, in relation to achieving the desired goals and objectives, and at some of the measures available to track changes in social connectedness and mental health. Participants will use the materials supplied to plan how they could collect and analyse relevant qualitative and quantitative data about these outcomes.
Presenter: Bernie Marshall, Deakin University

Evaluating mental wellbeing and social connectedness (2,998kb, MS Powerpoint)

Planning for Effective Health Promotion Evaluation
One of the most visible outcomes of the DHS 'Evaluation Skills Development Project' has been the development of an evaluation resource designed specifically for the Victorian context: Planning for Effective Health Promotion Evaluation (Round, Marshall and Horton, 2005). This workshop will introduce participants to the resource, and provided an opportunity for them to use some of the tools in it to plan how to evaluate one of their current programs.
Presenters: Jen Missing and Sue Rosenhain, Primary Health Care in Action (May 2005), Bernie Marshal, Deakin University (June 2005).

Evaluation planning framework, 30 May 2005 (30kb, MS Powerpoint)

Planning for effective health promotion evaluation (4,371kb, MS Powerpoint)

Strengthening Catchment Planning & Evaluation
This session shared the work that three PCPs have been involved with to strengthen catchment level planning for evaluation and how evaluation is undertaken at a catchment level.
Chair: Melissa Yong, Regional Health Promotion Officer Dept of Human Services SMR
Panelists: Wendy Mason, South East PCP & Valerie Kay, Inner South East Partnership in Community Health (ISEPICH) – joint presentation, Emma Brentnall - Campaspe PCP, Fiona Heenan - Southern Grampians & Glenelg PCP

Strenghtening catchment planning and evaluation (125kb, MS Powerpoint)

Southern Grampians and Glenelg PCP - Strengthening catchment planning and evaluation (128kb, MS Powerpoint)

South East PCP - Strenghtening catchment planning and evaluation (177kb, MS Powerpoint)

Catchment planning and evaluation in Campaspe PCP (123kb, MS Powerpoint)

top of page

Narrative Evaluation Action Research in Health Promotion (NEAR)
This session discussed the DHS N&WMR Narrative Evaluation Action Research Capacity Building Project from the multiple perspectives’ of the projects collaborators: Community and Women’s Health agency practitioners, university consultants and regional health promotion staff. The practitioners highlighted their action learning experiences in developing and applying processes and skills for writing health promotion narrative evaluation reports and mentoring other agency staff. The consultants covered the piloting and trialing of the manual developed in collaboration with the project partners over the course of the project: Writing narrative action evaluation reports in health promotion-guidelines, resource kit and case-studies. The regional Health Promotion staff provided an overview of the evolution of the NEAR project and possible future directions.
Presenters: Karen Goltz, Regional Health Promotion Officer, DHS N&WMR Office
Yoland Wadsworth, Adj Prof Action Research Program, Institute of Social Research,
Swinburne University of Technology. Kerryn McNaught Acting, Health Promotion Co-ordinator, ISIS Primary Care. Jaime Timmerman, Dietitian/Health Promotion Co-ordinator, Western Region Health Service

top of page