|
Building Bridges - Volume III, Number 1, 2003
Counting What Counts
By M. Elaine Nugent
Director, Office of Research and Evaluation
American Prosecutors Research Institute
Careful evaluation of a community prosecution program is important for monitoring progress, identifying program strengths and weaknesses, and documenting success.
An evaluator should answer these important questions:
- Is the program being properly implemented?
- Do the tasks accomplish the goals and objectives;
- Are the results as envisioned?
- Are we making a difference?
- Are new problems arising?
- Should we expand to new geographic areas or different types of crime?
Evaluation is a tool for making sure that your efforts are on track and to determine if the community prosecution effort is making a difference. The key to effective evaluation is to begin thinking about it as part of the planning process.
Planning for Evaluation
Typically, two types of evaluations may be conducted:
- The process evaluation, which examines whether community prosecution is being implemented as planned; and
- The outcome or impact evaluation, which answers the question of whether community prosecution has made an impact.
Together, a process evaluation and an outcome evaluation may document a cause and effect relationship: the outcomes were caused by the community prosecution effort.
Planning for evaluation includes the following three steps:
Step 1: Establish Measurable Goals and Objectives
To establish measurable goals and objectives, it is important to understand what each represents. Goals and objectives are not the same!
Goals describe the desired end result -- not the means to the end. What is the long-term expected impact of community prosecution?
Objectives are the shorter-term "benchmarks" that indicate progress is being made toward the goals. They describe in measurable terms who or what will change, by how much, and over what period of time.
As part of the planning process, problems are identified that need to be addressed. Desired solutions to these problems will form the substance of your goals and objectives. Questions to be answered include:
1. Why do you want to implement community prosecution, and what do you hope to achieve?
The answer to this question represents the goal of your community prosecution effort. Is it to reduce serious crime? Perhaps it's to improve quality of life or to enhance your community's sense of safety.
2. What type of information will convince you that you've obtained your goal?
If the goal is to reduce serious crime, then appropriate measures would include arrests, calls for service, and reported victimizations. If the goal is improving quality of life, measurements may include resident occupancy rates, property values, municipal code violations, and the community's perception of quality of life. This information may be tracked over time to determine if long-term change is occurring.
3. What must happen in the community in order to realize the long-term change? What events/issues are contributing to or causing the problem?
The answers to these questions are linked to the objectives of your community prosecution effort. For example, information gathered from the community indicates that residents are afraid to go to the neighborhood convenience store because large groups of teenagers loiter in and around the store, often fighting. Thus, changes must occur in and around the convenience store to prevent such fears.
4. Now that you know what is needed, what is a reasonable amount of change to expect and over what period of time?
The answers to this question will represent the information to be collected to demonstrate progress toward goal attainment. For example, it may be reasonable to reduce the number of calls about fights around the convenience store by 50 percent in one year.
* Important: Be realistic in establishing your goals and objectives; otherwise you may be setting yourself up for failure.
Step 2: Determine Which Activities Will Produce Change
There must be a realistic relationship between the activities and the objectives. For example, if one objective is to reduce the number of complaints about public drunkenness by 25 percent during summer months, would it be achieved by implementing an education program in elementary schools about alcohol use? Probably not ... a more realistic activity might be stricter enforcement of open container laws.
In deciding which activities to pursue, consider the following:
- Specifically, how will this activity achieve the objectives?
- What is the likelihood that this activity can produce the change desired?
- Are there other activities that can be conducted to produce the change?
Step 3: Identify Implementation Milestones
The final step in planning for an evaluation focuses on the "Process" side of evaluation. If one activity is to work toward enactment of an ordinance to prohibit public alcohol consumption, an implementation milestone would be the passage and enforcement of such an ordinance. If another activity is to organize neighborhood watch groups, the milestone would be the organization of a specific number of groups.
Collecting Information for the Evaluation
Tracking Implementation Milestones
Tracking implementation milestones shows whether or not the community prosecution effort is on course and provides immediate information that will allow you to make adjustments if needed. The implementation milestones tell the story, "This is what we're doing, how we're doing it, and how well we're doing it."
You will be gathering information on your implementation milestones as part of your routine day-to-day activities. The following list provides a few examples:
- Progress reports that detail contacts with community members and the specifics of planned activities;
- Calendars that show the timeline of planned activities and meeting schedules;
- Activity logs completed by community prosecutors detailing number, nature, and substance of contacts with the community; number of problems identified and addressed; and types of activities performed by the community prosecutor, such as participating in community-based crime prevention events.
To track implementation milestones using this type of information, you will need to: 1) ensure that staff who prepare the materials report the relevant milestone; and 2) conduct ongoing reviews to maintain current information on the milestones. Regardless of how implementation milestones are tracked, it is important that you summarize and review them on a regular basis -- daily, weekly, monthly, quarterly, or semi-annually.
Tracking Outcomes and Impacts
Information on attainment of goals and objectives can be gathered through a variety of formal and less formal mechanisms. Virtually all impact evaluations will include quantitative data, but this is not the only method for measuring outcomes and impact.
One easy mechanism for tracking changes, particularly environmental changes, is through observation. Observations can be tabulated, thereby creating quantitative/numeric information, e.g., the number of drug-related complaints received about a particular location over time.
Another mechanism is to take before and after pictures. (You may want to take "after" pictures periodically to demonstrate that the outcome has been sustained over time.)
Consider conducting an annual needs assessment. If a problem is identified as a priority in one year and subsequently diminishes in priority, it is possible that the problem has been appreciably resolved. This approach can help quantify problems that have been prevented or eliminated.
Informal surveys of community residents can be done at community meetings or by "man-on-the-street" interviews to determine whether they perceive a change in their neighborhood. More formal surveys might include an annual quality of life questionnaire distributed throughout the neighborhood or placed in the local newspaper for residents to complete and drop off at central locations (e.g., grocery or convenience stores, schools, or libraries). Interns from local colleges are a good resource for assistance in designing a survey, conducting in-person or phone interviews, and analyzing the results.
Statistical information does not have to be complicated. Many of the agencies that are partners in the community prosecution effort produce annual reports that track information on the things you're working to change. Local law enforcement agencies maintain numerous relevant data, such as criminal complaints, calls for service, arrest rates, and crime rates. The courts, or even your own office, will have information on charges filed, dispositions, and convictions.
Finally, never underestimate the power of the press. Over time, you can review media reports and determine whether coverage changes from negative to positive, if the reports drop off because the problem has diminished, or if the problem has changed for other reasons.
Interpreting the Evaluation Information
This task includes six guiding questions:
- What difference did your efforts make?
- What changes have occurred?
- To what can you attribute these changes?
- What other factors could have caused these changes?
- What other changes occurred that you didn't anticipate?
- What are the implications of these changes?
To answer the first two questions, review the information you've collected and compare it to the "baseline" information collected as part of your planning process to determine if there has been a change. Then, examine your implementation milestones to determine if these interventions caused the changes. Milestones that were missed along the way may help to explain why certain intended changes were not achieved.
If there is an observable change, either positive or negative, examine other events/activities that were happening over the time period. Did any other significant socio-economic changes occur? For example, was there a surge in unemployment or a horrific event that impacted the community? Did another agency implement activities or programs that targeted the same issues as your community prosecution effort? These other events may bolster, diminish, or supercede the impact of your efforts.
Community prosecution initiatives may have unanticipated positive or negative impacts on related problems. For example, an effort to make playgrounds safer for parents and their children may drive drug dealers into abandoned houses or to another neighborhood. Unanticipated outcomes might include decreased vacancy rates, increased home ownership, increased school attendance, etc.
Don't be discouraged if you find no change; it's quite common. Stop and consider whether the problem would have been worse without community prosecution. Look at other parts of the jurisdiction where community prosecution is not operational to see if they experienced the same problems and if the degree of negative change is greater than in your community prosecution area. If it is, you may conclude that without community prosecution, the problem would have gotten significantly worse.
Also, quite often it appears initially that a problem has increased in severity. Is this a result of increased awareness and increased reporting of the problem? Early efforts to address domestic violence and child abuse created a tremendous amount of public knowledge, which resulted in more reports being made and subsequently more arrests. By tracking the same indicators over time, the evaluation will allow you to answer this question -- first by seeing an increase and then a slow but consistent decrease.
- Important: Don't keep your evaluations under lock and key; share information with the community, the media, other justice agencies, local government officials, and potential funding sources. By communicating evaluation results along the way, you can build support for your efforts.
|
