Draft 1

At the meeting: “Which metrics would we include? Maybe stats about the number of members, number of events held, brown bag topics….?”

Look closely, can you see the small multiples blog post peeking through on the other side of the page? Start with sketching and your life will be 1,000,000 times easier.

Draft 2

Revisiting the initial brainstorming, about a week later: “I’ll need a header section, and a 2-sentence description of the organization, and maybe a list of Board members, and some demographic charts on members, and something about our brown bags and happy hours. And of course we’ll need contact information at the bottom… Okay this seems easy… Is this too easy…? Sketching is always the easiest step, right…?”

Draft 3

“The ‘Who We Are’ section is looking huge. Which topics do I even have data on? And how many charts can I realistically fit in that tiny space? I’d really love to keep this summary to just one page… And how would I actually visualize that stuff under ‘What We Do’ …  hmm … maybe a graphic timeline? … okay I’ll go with that for now…”

Draft 4

“This will be so easy, I’ll just stick these charts and text boxes into a PowerPoint slide so that I have plenty of control over moving the various objects around on the screen… I’ll just set my slide dimensions to a portrait layout… This’ll be printed in black and white and shared online in color PDFs, so I can use some color, but I better stick with a pretty simple color scheme – probably the burgundy from our logo and that’s it. There’s some yellow too, but there’s no way that yellow will show up good when this is photocopied in black and white… Darn copying and pasting charts from Excel into PowerPoint is such a drag, I wish there was some way around it…”

Draft 5

“Oh geez, oh course there’s a shortcut that helps me avoid copying and pasting charts back and forth from Excel into PowerPoint – I’ll just create the entire report inside Excel. Just like I often do… Why didn’t I think of that 20 minutes ago?! … C’mon Ann, maybe you shouldn’t be working on this ‘just for fun’ project at 2am…”

Final Report

Three weeks later, revisiting the Excel-based summary: “Vertical text boxes? What was I thinking?! … And those burgundy boxes around each section… They seemed fine earlier but now they seem way too dark… I want the headers to stick out, so let me increase the font a little and add light gray lines instead of burgundy boxes.”

Want to view the full report? Download the PDF.

Evaluator in the DC area? Join the Washington Evaluators and I for brown bags and socializing in the new year. Thanks for a great 2014!

I’m presenting a brown bag for the Washington Evaluators on Monday, September 15, 2014 from 12 – 1:30 pm at George Washington University in Washington, DC.

What: Visualizing Evaluation Results: Trends, Best Practices, and Software Strategies
Who: Ann K. Emery for the Washington Evaluators
When: Monday, September 15, from 12 – 1 pm
Where: George Washington University Marvin Center, Room 403 (800 21st St NW, Washington, DC 20052)
Nearest Metro station: Foggy Bottom (blue/orange lines)
Registration: Please RSVP in advance at http://washingtonevaluators.org/events

Have you ever emailed an evaluation report to a client and wondered whether they even read about the results? Have you tried sharing results during in-person meetings only to watch their eyes glaze over? If so, this brown bag is for you. As data becomes easier and cheaper to collect, it’s easy to get overwhelmed by the sheer volume of information available to us, to our coworkers, and to our clients. Visualizing data through charts, graphs, and diagrams is one strategy to deliver bite-sized pieces of information that stakeholders can understand at a glance and retain for the long run.

We’ll discuss how to select appropriate chart types; emphasize key findings with color; and take the guesswork out of your graphs through titles, subtitles, and annotations. We’ll focus on evaluation-specific visualization considerations: designing with stakeholders’ information needs front and center, using readily available software like Microsoft Excel, and thinking through a dozen chart types–dot plots, small multiples, heat maps, and more–that can be applied to nearly any evaluation project. We’ll conclude with strategies for sharing visualizations through videos, handouts, slidedocs, social media, and dashboards. You’ll leave with the critical thinking skills and technical prowess needed to create visualizations faster and easier than you ever thought was possible.

The dusty shelf report: The best way to keep evaluation results far, far, away from decision-making

After spending the past couple years as an internal evaluator, I decided to start addressing other internal evaluators’ questions, comments, and concerns. I’ll be sharing their questions on my blog, connecting them to other evaluators, and offering advice from my own experiences with internal evaluation.

Here’s a question from Jonathan O’Reilly, my friend from the Washington Evaluators. Jonathan recently accepted an internal evaluation position with the Circuit Court for Prince George’s County in Maryland. He writes:

“I’d like to know more about internal evaluators’ experience with translating research to practice. My experience as an external evaluator witnessed the final report being the absolute end product – whether or not the client used the recommendations or had working groups around the evaluation report were beyond our involvement. In my new position, I am more optimistic about my evaluation findings being used to effect change. As an internal evaluator my vision is to call together a working group to present evaluation findings and start the conversation about modifying procedures where necessary. What has your experience as an internal evaluator been with getting research findings to be a key part of administrative decision-making?”

Do you have any advice for this internal evaluator? Please share the good karma below.

Thanks, Ann Emery

————————————————-

Eric Abdullateef

Hi I’m Eric Abdullateef, and I recently worked on a study of the Horn of Africa Relief public engagement campaign in the context of United States Agency for International Development (USAID) FWD, which stands for Famine-War-Drought Relief. I started asking myself, how do social media messages during the heightened political discourse of a presidential campaign affect what the American public knows and feels about USAID and American’s role in the world?

Evaluation offers several ways to assess media messages and political discourse, and the evaluation process is useful for a variety of situations. Like:

• For academics: Who want to identify dif?culties in the audience research process and ways to improve the collection, analysis, and interpretation of data.
• For evaluation practitioners: Trying to surmount practical barriers to conducting and evaluating engagement processes.
• For public engagement campaigners: These types of campaigns promote reflection and debate within the donor community around issues related to the ethics and effectiveness of their domestic communications. Donors need to basically sort out which of the follow two main lines of effort, education or advocacy, will yield the best results.
• When there are limited resources: Assessment takes on increased significance in times of limited resources. In an era of budget tightening, the battle over resources may come down to accountability for how effective past allocations of resources have been spent. It is critical that agencies comprehensively measure the performance of their strategic communication efforts to understand which efforts are most effective and, in turn, determine how to make most efficient use of limited resources.
• When our understanding of the public’s sentiment is cloudy: Finally, assessment is important given the nebulous nature of public knowledge and sentiment, especially on international issues. Studies on U.S. public opinion on international development assistance have shown that while Americans generally favor humanitarian aid, they have very little idea how much money is allocated to aid programs (Council on Foreign Relations, 2009).

My project involved rapid assessment procedures, which helped me tease out the complexity of public attitudes towards foreign aid and USAID efforts to reshape them against the backdrop of best practice, political economy and a rich body of literature on American foreign affairs attitudes.

Have you ever worked on a knowledge, attitude and behavior change interventions like this one? How was the evaluation useful and valuable during that project?

— Eric Abdullateef

I hope you enjoy reading Mika Yoder Yamashita’s explanation about the best presentation she’s seen. Mika’s my evaluation friend from the Washington Evaluators, and the best presentation she’s seen was a systems dynamics presentation at the Stevens Institute.

— Ann Emery

————————————–

“The best presentation I saw was a presentation by a professor at Stevens Institute, and it was a quick walk through of systems dynamics (a part of systems thinking).  It was 20 minutes (and the presenter mentioned at the beginning that he would be talking for 20 minutes).   There are a couple of factors that I think made me to think this presentation was great.

First, there was a kind of book reading assignment.  The presentation happened as a part of a book club meeting.

Second, the book was difficult to me.  I did not read deep into systems dynamics before beyond Sociology 101 class, so it took a lot of my brain to go through the book.

When the presenter made the overall argument in the theory into a  concise diagram, I thought “we are seeing the same thing.” Yes, the presentation lost some details, and because it was simplified, I am sure the message was open to interpretation.

The purpose of the presentation was not to convey the same message. It was a warm up or quick introduction to a discussion.

Of course,  we could see the diagrams projected on the wall, and the presenter used probably 10 slides or less.   I do not remember if there was a handout, but since the presenter walked us through the diagrams, no need to read a handout.”

— Mika Yoder Yamashita

I recently read about an evaluator’s experiences at the U.S. Government Accountability Office (GAO) and I was intrigued. How is it possible that evaluators all seem to face the same challenges, no matter what the setting? My own internal evaluation work at a non-profit youth center is truly a microcosm of the larger evaluation world.

I’m a firm believer that the best way to get better at program evaluation is to talk to other evaluators and I thought, hey, I wonder whether I can have coffee with a GAO evaluator sometime and learn more??

Well, I emailed someone who I’d met through the Washington Evaluators, and one thing led to another…  and I have the lovely opportunity to interview not one but two evaluators from the GAO tomorrow over lunch! My Washington Evaluators friend and the article’s author! One of my favorite things about being an evaluator in DC is being close to so many skilled evaluators.

Stay tuned to hear some highlights from tomorrow’s conversation!

Ann Emery