One of my first evaluation projects started with a phone call (e-mail was rare in those long ago days). The conversion went something like this …
“John, we have this grant that requires that we do an evaluation. Sounds great. Love it. Can’t wait to get started. Just one question—What’s an evaluation?”
While a great deal has changed in the years since I fielded that call, I still like to joke that evaluation is the largest profession that no one has heard of. The American Evaluation Association has over 6,000 members, and during the past five years their ranks swelled by 40 percent. Virtually every grant awarded today by a government funding agency, philanthropic foundation or corporation requires an evaluation. Yet evaluators and their work are unknown to most Americans, so much so that at dinner parties I find myself feeling uncomfortable for the poor soul seated next to me who innocently asks, “What do you do?” How can I possibly explain before the table is cleared?
My standard answer is that I help professionals who manage educational and social programs figure out how effective their programs are and find ways to make their programs more effective. I go on to explain that I run a firm in which I and my colleagues specialize in something rather particular and technical called randomized trials. Essentially, these are experiments similar to the ones that doctors and drug companies conduct to ensure that medical treatments are effective. Continue reading
EvalBlog 
Asking Questions, Getting Answers
Douglas Adams
The late satirical author Douglas Adams spun a yarn about a society determined to discover the meaning of life. After millennia, they had developed a computer so powerful it could provide the answer. Gathering around on that long anticipated day, the people waited for the computer to reveal the answer. It was 42. Puzzled and more than a little angry, the people wanted to know how this could be. The computer responded that the answer to the big question of the meaning of life, the universe and everything was most definitely 42, but as it unfortunately turned out neither the computer nor the people knew exactly what the question was.
A similar fate can befall evaluations, more than one of which has produced a precise answer to a question never framed or a question framed so vaguely as to be useless. It is easy enough to avoid this fate when you realize that, at its most basic level, evaluations address only three big questions: Can it work? Did it work? Will it work again? We call them “Can,” “Do,” and “Will” for short. Of course, we can ask other questions, but they tend to be in support of or in response to the big three. What good is asking, for example, “How does it work?” before you believe that it can, did or will? Continue reading →
Leave a comment
Filed under Commentary, Elements of Evaluation, Research
Tagged as douglas adams, questions for evaluators