Fruitility (or Why Evaluations Showing “No Effects” Are a Good Thing)

The mythical character Sisyphus was punished by the gods for his cleverness.   As mythological crimes go, cleverness hardly rates and his punishment was lenient — all he had to do was place a large boulder on top of a hill and then he could be on his way.

The first time Sisyphus rolled the boulder to the hilltop I imagine he was intrigued as he watched it roll back down on its own.  Clever Sisyphus confidently tried again, but the gods, intent on condemning him to an eternity of mindless labor, had used their magic to ensure that the rock always rolled back down.

Could there be a better way to punish the clever?

Perhaps not. Nonetheless, my money is on Sisyphus because sometimes the only way to get it right is to get it wrong. A lot.

This is the principle of fruitful futility, or as I call it fruitility.

How many times did Sisyphus roll the boulder up the hill only to watch it roll back down?  Someone as clever as he was surely would have tried something new, something imaginative.

Perhaps he buried the boulder on the hilltop.  If that failed, he might have built up the earth around the hill to such a degree that the hilltop became the bottom of a deep well.  Maybe he beat the boulder to dust, rendering it “unrollable”.  Time was on his side and I feel certain that he eventually found a solution.

Whatever that solution turned out to be, it must have looked very different from the one he stared with.  Intractable problems demand innovative solutions.  If they didn’t, they wouldn’t be intractable problems.

The problems that social and educational programs set out to solve are some of the most intractable that society faces.  So why is it a surprise that recent evaluations of educational programs sponsored by the federal Institute of Education Sciences—all of them randomized trials—show no effects?

The question is not whether the small number of solutions that were tested worked, but if, upon discovering that most did not work as well as intended, that information sparked innovation.  That remains to be seen.

The history of educational research reaches back to antiquity. We have learned a great deal, but in many ways we are no better off than Sisyphus rolling the boulder to the top of the hill for first time.  As a society, the forces pushing us forward — imagination, motivation, feedback, memory, or opportunity — seem to lack the power of those pushing Sisyphus to try new things

Eternal punishment has a way of focusing your attention.

Evaluators are rarely affected by the failure of the programs they evaluate. This has advantages (it promotes objectivity, for instance). But it slows innovation because old solutions, even poor ones, demand less effort and feel less risky than inventing new ones. Why champion something new? It can get to the point where evaluators may find themselves being rewarded for promoting the status quo rather than seeking to improve it.

By providing less ambiguous information about the impact of programs, randomized trials help us all from simply rolling the rock the same way over and over.  But a technical approach to gathering better information — which includes clearly identifying failure —  is only half of the equation that leads to new ideas.  The other half is more elusive. It is a commitment to finding solutions.

So, while innovation is the fruit of futility, it arises from more than failure and our ability to recognize it.  It depends on our will.