One of my first evaluation projects started with a phone call (e-mail was rare in those long ago days). The conversion went something like this …
“John, we have this grant that requires that we do an evaluation. Sounds great. Love it. Can’t wait to get started. Just one question—What’s an evaluation?”
While a great deal has changed in the years since I fielded that call, I still like to joke that evaluation is the largest profession that no one has heard of. The American Evaluation Association has over 6,000 members, and during the past five years their ranks swelled by 40 percent. Virtually every grant awarded today by a government funding agency, philanthropic foundation or corporation requires an evaluation. Yet evaluators and their work are unknown to most Americans, so much so that at dinner parties I find myself feeling uncomfortable for the poor soul seated next to me who innocently asks, “What do you do?” How can I possibly explain before the table is cleared?
My standard answer is that I help professionals who manage educational and social programs figure out how effective their programs are and find ways to make their programs more effective. I go on to explain that I run a firm in which I and my colleagues specialize in something rather particular and technical called randomized trials. Essentially, these are experiments similar to the ones that doctors and drug companies conduct to ensure that medical treatments are effective.
“So it’s like social science research?” Not exactly. Academic researchers set out to answer questions that they believe are important, and while their research may push the boundaries of knowledge it often has little direct impact on policy or programs. We do use social science research methods, but we focus on finding the best ways to answer the questions that our clients believe are important in order to guide policy decisions and real-world actions. We typically develop specialized surveys, tests, and other measures to quantify and describe what our clients hope to achieve.
“So it’s like management consulting for nonprofit organizations?” Sort of. Like management consultants, we work with clients to build the capabilities of their organizations. However, the outcomes that our clients work to bring about are rarely about their organizations or denominated in dollars. Our clients strive to make a social impact on a wide variety of levels. Measuring the social impact using tests, surveys, observations and other means is a substantial part of our work.
“So it’s like standardized testing?” A little. Standardized tests are intended to help states and school districts manage large school systems. While we may use a standardized test as one measure of a program’s effectiveness, they are rarely sufficient to serve our purposes.
“So exactly what do you do?” Well, everyone wants to make a difference in the world. We help our clients to see the difference that they made.
“Hmmm…that sounds useful.” I think so. And I think that sums up what evaluation really is—useful.