NSW SAFE COMMUNITIES PILOT PROJECTS—EVALUATION METHODOLOGY
Cath Sefton
Department of Public Health and Community Medicine
University of Sydney
Citation: N S W Public Health Bull 2002; 13(4): 76–77
This article describes the methodology currently being used to evaluate the NSW Safe Communities Pilot Projects (SafeComm). The evaluation, by the University of Sydney, has been funded for three years as part of the pilot program by the NSW Department of Health and the NSW Roads and Traffic Authority to investigate the Safe Communities model.
OBJECTIVES
The evaluation project has both local and statewide objectives.
Local level
Objectives at the local level are to:
- develop and monitor measures of attitude and operational change, including: key informant–agency participation and participation change over the period of the project; changes in the incidence and form of local media knowledge and reporting; incorporation of the SafeComm project and/or its components into the business of local government; and level of integration of cross agency collaborations into the local planning processes;
- identify and benchmark indicators of hazard reduction;
- identify and report on suitable measures of injury outcome.
State level
Objectives at the state level are to:
- develop and implement a biannual SWOT analysis (of strengths, weakness, opportunities, and threats), which will be used to collate data from across all field methods, and to identify critical findings and trends over time.
METHODS OF DATA COLLECTION AND ANALYSIS
The evaluation employs six main methods of data collection and analysis:
- inter-organisational network analysis;
- in-depth interviews with key informants;
- impact logs;
- capacity-building indicator checklists;
- media content analysis;
- injury data reporting.
Each methods is briefly described below.
Inter-organisational network analysis
Network analysis is a quantitative mapping technique. A survey is conducted among different organisations to examine how, and to what extent, different agencies collaborate. The survey also measures the strength of those connections. It is these connections that are mapped in the network analysis.
The survey is conducted by telephone with a representative of each of the participating organisations. An initial survey was conducted during the first year of the projects (2000) and this will be repeated in the third year (2002). The data are being analysed using the UCINET data software package,[1] and graphs constructed using KrackPlot.[2]
In-depth interviews of key informants
A series of in-depth face-to-face interviews is being used to collect detailed qualitative information on the projects from the perspectives of the participants. Interviews are being conducted with participants in each project at regular intervals covering the following core issues:
- expectations of the project and the reasons behind these;
- understanding of the aims and objectives of the project;
- coalition characteristics;
- conflict resolution;
- roles of members and project officers;
- the costs and benefits of participation;
- project achievements and external influences that might affect these;
- problems encountered in the project.
In addition to these core topics, interviews are tailored to the specific interests and objectives of each project. The interviews are a significant source of self-reported behaviour change among target groups.
The interviews are semi-structured, consisting of a series of standard questions covering the range of topics listed above, followed by an opportunity for more free ranging questions and discussion. The content of interviews at different stages of the projects varies depending on a number of factors, including the results of capacity-building indicators. Issues of interest that arise through other methods are also canvassed with project participants through the interview process; and, likewise, issues raised in interviews guide the timing and use of other aspects of the evaluation, in particular the use of the indicator checklists.
Impact logs
An impact log is a written account of the major activities that occur in a project. The log can be used to record any changes in the community that are brought about by the project, such as new road safety measures or other hazard reducing actions or structures. Each impact in the log is coded according to the type of activity and when it occurred. Using this information the evaluators produce regular reports on the frequency and nature of project activities.
Capacity-building indicator checklists
As specified in the tender document, the indicators of health promotion capacity, developed by Hawe et al.,[3] will be used at various stages of the evaluation. The exact timing and use of these checklists will vary between communities and will be determined on the basis of activities and project development in each community. The checklists assist in assessing a range of conditions, including the strength of a coalition, if a program is likely to be sustained, and capacity for organisational learning. Some checklists may be used several times throughout the life of the projects while others will be used only once or twice
Media content analysis
A content analysis will be conducted on local newspaper items related to both the project and to safety issues in each community. The content analysis will involve coding each media article according to the following factors:
- size and type of article (for example, news story, letter to the editor, feature article);
- type of safety message reported;
- safety issue;
- organisations–groups involved;
- how the safety issue is ‘framed’ (for example, person-blaming rather than situation-blaming).
Data will be collected over four two-month periods and analysed to identify any changes in the quantity or type of safety related articles that have appeared in the local media.
Injury data reporting
Data collected by each project will be analysed over the life of the project in order to identify any changes in injury rates. It is important that each project establishes its own data collection methods and forges alliances with organisations that collect relevant information, in order for those data collections to be sustained into the future. A variety of injury data are already available to individual SafeComm projects, and the evaluation proposes to build on this existing capacity.
CONCLUSION
Three progress reports from the evaluation project have been provided to the funding bodies and to each of the three pilot projects. A final report will be completed at the conclusion of the evaluation at the end of 2002.
REFERENCES
- Borgatti S, Everett , Freeman L. UCINET IV 1.0 Columbia: Analytic Technologies, 1992.
- Krackhardt D, Blythe J, McGrath C. KrackPlot 3.0: an improved network drawing program. Connections 1994: 17(2):53–55.
- Hawe P, King L, Noort M, Jordens C, Lloyd B. Indicators to help with capacity building in health promotion. Sydney: NSW Department of Health and the Australian Centre for Health Promotion, University of Sydney, 2000.
|