On the Recursive Nature of Recursion: Reflections on the AEA 2013 Conference

John Gargani Not Blogging

Recursion is when your local bookstore opens a café inside the store in order to attract more readers, and then the café opens a bookstore inside itself to attract more coffee drinkers.

Chris Lysy at Freshspectrum.com noticed, laughed at, and illustrated (above) the same phenomenon as it relates to my blogging (or rather lack of it) during the American Evaluation Association Conference last week.

I intended to harness the power of recursion by blogging about blogging at the conference. I reckoned that would nudge a few others to blog at the conference, which in turn would nudge me to do the same.

I ended up blogging very little during those hectic days, and none of it was about blogging at the conference. Giving up on that idea, I landed on blogging about not blogging, then not blogging about not blogging, then blogging about not blogging about not blogging, and so on.

Once Chris opened my eyes to the recursive nature of recursion, I noticed it all around me at the conference.

roe_aea_2013_evalblogFor example, the Research on Evaluation TIG (Topical Interest Group) discussed using evaluation methods to evaluate how we evaluate. Is that merely academic navel gazing? It isn’t. I would argue that it may be the most important area of evaluation today.

As practitioners, we conduct evaluations because we believe they can make a positive impact in the world, and we choose how to evaluate in ways we believe produce the greatest impact. Ironically, we have little evidence upon which to base our choices. We rarely measure our own impact or study how we can best achieve it.

ROE (research on evaluation, for those in the know) is setting that right. And the growing community of ROE researchers and practitioners is attempting to do so in an organized fashion. I find it quite inspiring.

A great example of ROE and the power of recursion is the work of Tom Cook and his colleagues (chief among them Will Shadish).tom_cook_aea_2103_evalblogI must confess that Tom is a hero of mine. A wonderful person who finds tremendous joy in his work and shares that joy with others. So I can’t help but smile every time I think of him using experimental and quasi-experimental methods to evaluate experimental and quasi-experimental methods.

Experiments and quasi-experiments follow the same general logic. Create two (or more) comparable groups of people (or whatever may be of interest). Provide one experience to one group and a different experience to the other. Measure outcomes of interest for the two groups at the end of their experiences. Given that, differences in outcomes between the groups are attributable to differences in the experiences of the groups.

If on group received a program and the other did not, you have a very strong method for estimating program impacts. If on group received a program designed one way and the other a program designed another way, you have a strong basis for choosing between program designs.

Experiments and quasi-experiments differ principally in how they create comparable groups. Experiments assign people to groups at random. In essence, names are pulled from a hat (in reality, computers select names at random from a list). This yields two highly comparable but artificially constructed groups.

Quasi-experiments typically operate by allowing people to choose experiences as they do in everyday life. This yields naturally constructed groups that are less comparable. Why are they less comparable? The groups are comprised of people who made difference choices, and these choice may be associated with other factors that affect outcomes. The good news is that the groups can be made more comparable–to some degree–by using a variety of statistical methods.

four_arm_study_aea_2013_evalblogIs one approach better than another? At the AEA Conference, Tom described his involvement with efforts to answer that question. One way that is done is by randomly assigning people to two groups–one group that will be part of an experiment or another group that will be part of a quasi-experiment (referred to as an observational study in the picture above). Within the experimental group, participants are randomly assigned to either a treatment group (e.g., math training) or control group (vocabulary training). Within the quasi-experimental group, participants choose between the same two experiences, forming treatment and comparison groups according to their preference.

Program impact estimates are compared for the experimental and quasi-experimental groups. Differences at this level are attributable to the evaluation method and can indicate whether one method is biased with respect to the other. So far, there seems to be pretty good agreement between the methods (when implemented well–no small achievement), but much work remains to be done.

coming_together_aea_2013_evalblog

Perhaps the most important form of recursion at the AEA Conference is membership. AEA is comprised of members who manage themselves by forming groups of members who manage themselves by forming groups of members who manage themselves. The board of AEA, TIGs, local affiliates, task forces, working groups, volunteer committees, and conference sessions are all organized by and comprised of groups of members who manage themselves. That is power of recursion–3,500 strangers coming together to create a community dedicated to making the world a better place. And what a joy to watch them pull it off.

About these ads

8 Comments

Filed under AEA Conference, Conference Blog, Evaluation, Program Evaluation, Research

8 responses to “On the Recursive Nature of Recursion: Reflections on the AEA 2013 Conference

  1. This adds new light to our conversation. I thought you ended up with that concept randomly based on our seemingly directionless conversation (I drew it but that was really John’s cartoon). I had no idea you were caught up in some type of in your own head feedback loop.

    Glad I could help nudge you out of it.

    Also, who is going to evaluate the evaluators evaluating evaluation? I’m not serious, just really wanted to say that.

  2. And what a joy to read this blog! I find your ideas creative and interesting, and your writing superb. :-)

    • Thanks. I appreciate the kind words. If you keep reading, I’ll keep writing.

      • You got it. If you keep writing, I’ll keep reading. And if I keep reading, you’ll keep writing. But how will you know I’ll be reading if you’re writing? I’ll have to write to let you know I’ve been reading, and then you’ll have to write to let me know you’ve been reading that I’ve been reading, so that you’ll do more writing, so what you’re really saying is that you’ll be reading if I’m writing, right?

      • Ah….the student has become the master, and the master the student. But realizing that, the student again becomes the master, and…

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s