Should We Fear Subjectivity?

Like many this summer, I found myself a bit perplexed by the way Olympic athletes in many sports received scores. It was not so much the scoring systems per se that had me flummoxed, although they were far from simple. Rather it was realizing that, while the systems for scoring gymnastics, ice skating, boxing, and sailing had been overhauled over the past few years in an effort to remedy troubling flaws, the complaint that these scores are subjective — and by extension unfair — lingered.

This dissatisfaction reflects an unwritten rule that applies to our efforts to evaluate the quality or merit of any human endeavor: if the evaluation is to be perceived as fair, it must demonstrate that it is not subjective. But is this a useful rule? Before we can wrestle with that question, we need to consider what we mean by subjective and why we feel compelled to avoid it.

Subjectivity is a problematic concept. Sometimes we use the term as a synonym for biased. In the technical sense, bias occurs when average measures (based on quantitative data) or aggregated observations (based on qualitative data) systematically yield incorrect results. And it is especially troubling when those incorrect results favor one group over another.

At other times we use subjective as a synonym for inconsistent. Again in the technical sense, inconsistency occurs when the same method for collecting quantitative or qualitative data is applied to the same object of study under the same circumstances, yet it yields data of vastly disparate magnitudes or meanings.

What is clear from these definitions is that bias and inconsistency are matters of degree. However, it turns out that we can usually determine the degree of bias and inconsistency quite well, ascertain the impact these shortcomings might have on our conclusions, and most importantly we can use well established techniques to avoid, reduce or compensate for both.
But subjectivity is more than bias and inconsistency. In particular, it imposes an entirely different set of challenges when we use the term as a synonym for arbitrary. In this sense, subjectivity is no longer a technical issue, nor is it merely a random process like flipping a coin to determine who wins. Rather it is process based on caprice, sometimes trustworthy and sometimes not, which allows personal agendas, conscious or subconscious, to be projected onto the results that an evaluation produces.

In the Olympics, athletes have scoring systems imposed on them. Fearing that these systems are arbitrary, athletes use their achievements to judge the quality of the scoring rather than using the scoring to judge the quality of their achievements. There are stakes associated with Olympic competition, so results are rarely satisfactory for every participant, which in turn leads to further examination of the scoring systems. Sadly, in the end, the achievements of the athletes are all but forgotten.

Just as with the Olympics, program evaluations can take a similar turn unless the issue is addressed at the start. Unlike Olympic athletes, program stakeholders have much greater latitude to choose their own way of keeping score. By working together, stakeholders and evaluators can build consensus about what constitutes a fair approach to measuring and assessing the effectiveness of a program.

One part of this fair approach should be what I call practical objectivity. That is, the evaluation should be undertaken in such a way that the results of the evaluation do not depend in any material way on the results that are desired by stakeholders, evaluators, funders or anyone else.
There are a number of ways to go about this, depending on the program and the exact circumstances under which it is being implemented. Evaluators should understand how to achieve practical objectivity, and clients should be keen to ask about it. This sort of an evaluation takes time — and trust — but it yields results program staff can use and knowledgeable, skeptical audiences will value.

So should we be afraid of subjectivity? No. Certainly we should do our best to minimize the impact that bias, inconsistency and arbitrariness may have on our results, but subjectivity is not something to be feared.
We all bring with us our own perspectives and values that can (and should) be integrated into evaluations in constructive ways. Our most basic — and subjective — beliefs of what constitutes a problem, why the problem deserves a solution and how the solution should be structured lie at the the very foundation of programs and evaluations. In this sense, everything we do is subjective, and it is that sort of subjectivity that makes what we do meaningful.