Category Archives: Program Evaluation

Should We Fear Subjectivity?

ringsmedium1

Like many this summer, I found myself a bit perplexed by the way Olympic athletes in many sports received scores. It was not so much the scoring systems per se that had me flummoxed, although they were far from simple. Rather it was realizing that, while the systems for scoring gymnastics, ice skating, boxing, and sailing had been overhauled over the past few years in an effort to remedy troubling flaws, the complaint that these scores are subjective — and by extension unfair — lingered.

This dissatisfaction reflects an unwritten rule that applies to our efforts to evaluate the quality or merit of any human endeavor: if the evaluation is to be perceived as fair, it must demonstrate that it is not subjective. But is this a useful rule? Before we can wrestle with that question, we need to consider what we mean by subjective and why we feel compelled to avoid it. Continue reading

Leave a comment

Filed under Commentary, Evaluation, Program Evaluation

Randomized Trials: Old School, New Trend

surfers

To my mind, surfing hit its peak in the 1950s when relatively light longboards first became available.

Enthusiastic longboarders still ride the waves, of course, but their numbers have dwindled as shorter more maneuverable boards became more fashionable. Happily, longboards are now making a comeback, mostly because they possess a property that shortboards do not: stability. With a stable board novices can quickly experience the thrill of the sport and experts can show off skills like nose walks, drag turns, and tandem riding that are unthinkable using today’s light-as-air shortboards.

The new longboards are different — and, I think, better — because their designs take advantage of modern materials and are more affordable and easier to handle than their predecessors. It just goes to show that everything old becomes new again, and with renewed interest comes the opportunity for improvement.

The same can be said for randomized trials (RTs). They were introduced to the wider field of social sciences in the 1930s, about the time that surfing was being introduced outside of Hawaii. RTs became popular through the 1950s, at least in concept because they can be challenging and expensive to implement. During the 60s, 70s and 80s, RTs were supplanted by simpler and cheaper types of evaluation. But a small and dedicated cadre of evaluators stuck with RTs because of a property that no other form of evaluation has: strong internal validity. RTs make it possible to ascertain with a high degree of certainty — higher than any other type of evaluation — whether a program made a difference. Continue reading

2 Comments

Filed under Commentary, Evaluation, Program Evaluation, Research