Theory Building and Theory-Based Evaluation

When we are convinced of something, we believe it. But when we believe something, we may not have been convinced. That is, we do not come by all our beliefs through conscious acts of deliberation. It’s a good thing, too, for if we examined the beliefs underlying our every action we wouldn’t get anything done.

When we design or evaluate programs, however, the beliefs underlying these actions do merit close examination. They are our rationale, our foothold in the invisible; they are what endow our struggle to change the world with possibility.

The beliefs underlying programs are a specific sort. They are theories that describe what is, what can be and how we can transform one to the other. The word theory can be off putting because it is associated with big ideas, like the theory of relativity, which only experts are allowed to have. But we all have theories about how the social world works; we simply don’t credit them as such.

To make matters worse, theory is often used in a pejorative sense as in, “Maybe that works in theory, but don’t try it in practice.” But theory and practice are not at odds because the theories that are the most useful to us are the ones we use to explain our own practice.

Theories take center stage in what is called theory-based evaluation. As the name implies, this is an approach to evaluation in which theories about programs, specifically those held by program designers and managers, guide the work of the evaluator.

If these theories are left implicit, it can diminish the utility of an evaluation. For example, it is generally considered a good idea to align outcome measures with the program being evaluated. On a superficial level, this can mean ensuring that the content of an educational test is consistent with what is taught in the classroom.

On a more fundamental level, it can also mean incorporating into the test the ways in which learning is expected to take place. For example, if the theory that guides instruction affects the nature of questioning in the classroom, as is the case with inquiry-based science, it also affects how test questions should be asked. If certain misconceptions about a subject are predicted by the program theory and then addressed through instruction, this probably merits developing test questions that probe these misconceptions.

Getting at the program by getting at its underlying theory makes an evaluation more likely to detect the impact of the program (because you are looking in the right places) and provides a clear understanding of how to improve the program (because you can compare your actions and results to your theoretical expectations).

One of the most useful things about theories in particular, and beliefs in general, is that they are contingent — if better ones come along we can adopt the new ones in place of the old ones. But some beliefs are easier to give up than others because our beliefs are connected in a complicated way that is somewhat akin to a pile of pick-up-sticks. Removing or disturbing those nearest the surface has little effect on the others. Disturbing just one at the core can upset all the others.

Evaluation is one way to productively explore and rearrange our tangled beliefs into more orderly and robust theories that help us accomplish our larger purposes. And in doing so we can be a little bit like Einstein who, by examining his early beliefs about relativity, realized that they were seriously flawed, refined them and along the way changed how we think and act.