Evaluation Clock


The Evaluation Clock (adapted from Pietro 1983) unpacks the "evaluation or systematic study" component of the Action Research cycle. It indirectly addresses the “planning component” by making you look ahead to consider which people might be influenced by the results and what they could do based on the possible outcomes.

The ultimate goal of using the Clock framework is that you can use it to design your own evaluation or systematic study mindfully, working both:
In particular, evaluation and planning or design should be inextricably linked. For example, when you think about what could be done differently (step 11) on the basis of the specific measurements or observations you include in the evaluation (step 3), you may refine your measurements or observations. You may even decide to separate out two or more different sub-issues within the overall issue (steps 0-2), each requiring a different evaluation. As Pietro (1983, 23) says: “The clock marks time in an unusual fashion, since it does not necessarily move in a clockwise direction, but rather jumps from one number to another until all the questions have been struck.” It has been suggested that the Clock is more like a combination lock on a safe. Working sequentially and recursively is characteristic of Action Research as a whole, except that with the Evaluation Clock each step might require a tight, self-conscious method (e.g., statistical analysis).

Comparisons

When the evaluation is a systematic study of effects of some intervention or engagement, there is always a comparison involved. The comparison might be before versus after some intervention is made, or it might be a comparison of one situation (with a particular curriculum, treatment, etc.) versus another situation (without that curriculum, etc.) (steps 2 and 3 of the Clock). Did it have the intended effects? Was it better than other approaches? The idea of comparison can also be applied to continuous data, e.g., on the incidence of violent crimes in relation to unemployment rate. This is, more or less, like asking is there more (or less) violent crime in times of high unemployment than in times of low unemployment?
In valid comparisons all other factors are supposed to be equal or unchanged. If they are not, then the comparison is suspect. Perhaps it needs to be broken into a number of comparisons, e.g., before versus after for privileged schools, and before versus after for poor schools.
When the evaluation is a systematic study of what has already been happening, it may only involve collecting information about one situation, e.g., finding what % of adults are able to read competently. The formulation of the evaluation criteria and interpretation of the results depends, however, on an implicit comparison with a desired situation, e.g, one in which there is full adult literacy.

In order to get acquainted with the Clock, the comparison at its heart, and the sequential and recursive aspects of using the Clock, it is helpful to reconstruct an evaluation that has been conducted. When you do this you have to put yourself in the shoes of the group or person(s) who conducted the evaluation and fill in the steps they appear to have taken.
In order to get the hang of comparisons, focus on steps 2 and 3 for a simple case (e.g., Goode on the effects of a smoking in bars). Steps 0, 4 and 5 may help you as well. (See “stripped down clock” appended after the full Clock.)

When you have the hang of the comparison idea, pay attention to the sequential and recursive aspects of the Clock:
The sequential part of this reconstruction means that the answers at each step are logically related to the previous ones, especially the immediately preceding one. For example, the lessons in step 10 are lessons learned from the reasons (step 9) for what is happening (step 8a). Similarly, the outlets (step 8b) should take into account the sponsors' goals and audience (step 1). Sequentiality also means that the key issues of the evaluation (step 2) are not the issues that emerge after the results (steps 8-12). The key issues are the ones that the evaluator saw needed studying before they knew the actual results.

The recursive part of this reconstruction means that when you think about what the evaluator or their sponsors did with the results (steps 10 and 11)—or what they could conceivably do with the results—you might go back and revise your interpretation of what decisions or policies or actions were at stake (steps 0 and 1). For example, an evaluation that points out that a low % of NY City high school students are passing the Regents exam says little about causes of the low % or about ways to improve education in the school system. We might even suspect that what concerns the sponsors of the evaluation (step 0) was to discredit public education. This conjecture would have to be validated, of course. in the meantime, however, we can note that someone wanting to learn how to improve public education would want to design a quite different evaluation.

When you try to make sense of evaluations that others have done or are proposing, you may see that parents, teachers, administrators, and policy makers want different things evaluated, even if the different wishes have been lumped together. For example, regarding high-stakes standardized tests, evaluations of the following different things are mixed together:
• students' knowledge • new curricular frameworks as a means to improve students' knowledge
• performance of teachers • performance of schools and • performance of school districts.
You have to separate the different kinds of evaluation for any issue you are interested in, and address each evaluation appropriately.
More generally, you should add notes from your own critical thinking about what others have done: Why evaluate in this situation? Why this evaluation and not another? What theories are hidden behind the intervention that was implemented? What supports are given to people to make the intervention?

A note on working from newspaper articles: In using the Clock to reconstruct an evaluation that has already been conducted, you have to put yourself in the shoes of the group or person(s) who conducted the evaluation in order to fill in the steps they appear to have taken. You should not answer the earlier steps with information that the people did not have until after they had conducted the evaluation. Often a newspaper article will not give you information for every step in the clock. For the missing steps, fill in what you would do as someone in the corresponding position, i.e., designing an evaluation (for the early steps), interpreting it (for the middle steps), or deciding on proposals to make (for the later steps). Deciding what you would do is a matter, as is the case in Action Research, of making proposals that follow from research results and presenting the proposals to potential constituencies who might take them up if the research supports them.

FULL CLOCK

[Copy and paste the clock into a wordprocessing file to allow room for your responses.]

0a. The intervention whose effect or effectiveness needs to be evaluated is...
0b. Interest or concern in the effect/iveness of the intervention arises because...


_
1a. The group or person(s) that sponsors the evaluation of the intervention are...
1b. The people they seek to influence with the results are...
1c. The actions or decisions or policies those people might improve or affirm concern...



_
2. General Question: The comparison needed to evaluate the effectiveness of the intervention is between two (or more) situations, namely a. a comparison of...

b. with respect to differences in the general area of…..
_
3. Specific observables: To undertake that comparison, the effects of the intervention will be assessed by looking at the following specific variable(s) in the two (or more) situations...


_
4. The methods to be used to produce observations or measurements of those variables are...(survey, questionnaire, etc.)


_
5a. The people who will be observed or measured are...
5b. The observing or measuring is done in the following places or situations... or derived indirectly from the following sources...

_
6. The observations or measurements will be analyzed to determine whether the two situations are significantly different in the following manner...


_
7a. Given that people who will interpret (give meaning to) the analysis are...
7b. the analysis will be summarized/conveyed in the following form...

_
When the results are available, the following steps can be pinned down. In the design stage, you should lay out different possiblities.
8a. The results show that what has been happening is...
8b. This will be reported through the following outlets...

_
9. What has been happening is happening because...

_
10. The lessons learned by sponsors of evaluation are that...

_
11. What the sponsors should now do differently is...


_

STRIPPING DOWN THE "CLOCK" TO FOCUS ON THE COMPARISON INVOLVED IN EVALUATING THE EFFECTS OF ANY EDUCATIONAL INTERVENTION

[Copy and paste the clock into a word file to allow room for your responses.]

0. The intervention whose effect or effectiveness needs to be evaluated is...



_
2. The comparison needed to evaluate the effect/iveness of the intervention is between two (or more) situations, namely comparing...



_
3. To undertake that comparison, the effects of the intervention will be assessed by looking at the following specific variable(s) in the two situations...



_
4. The methods to be used to produce observations or measurements of those variables are...(survey, questionnaire, etc.)



_
5. The people who will be observed or measured are...
This is done in the following places or situations... or derived indirectly from the following sources...


References

Goode, E. (1998) "When bars say no the smoking," New York Times, 15 Dec.
Pietro, D. S., Ed. (1983). Evaluation Sourcebook. New York: American Council of Voluntary Agencies for Foreign Service.