Tuesday, October 8, 2013

What to Measure, Why and When

Research, Assessment, and Evaluation
by Ben Delaney, © 2013

How many meetings have you attended in which the term “data-driven” was tossed about, its utter essentiality stressed?

And how many times did you have the feeling that no one had a clue what “data-driven” really means?

As one who has made a living doing market research, I learned to like data. Now I just love data. One of my greatest thrills at work in the last year was finally discovering a seasonal cycle in the sales of our social enterprise store. This information helped us plan a sale at the right time that doubled the store receipts that month. That's an example of data-driven decision making.

I can;t provide you with a course on statistics. Even if I were qualified to do so, we don't have the space for that level of detail. What I do hope to provide is a framework for thinking about data and evaluation that will make your work a bit easier.

Data doesn't just measure results

I think it is very important to use data to shape programs, both in initial planning and through a reiterative, ongoing analysis. Changes are driven by the findings, and often, the answer to one question raises other questions. 

Data-driven programs work this way:
  • Program planning is based on research, with measurement points built in, plus
  • Ongoing, reiterative analysis of the collected data used to refine the program and deepen understanding, then
  • Programs are changed as new knowledge emerges from the data.
That is what a true data-driven organization does.

It takes rigor and discipline to work this way, but the resulting improvements in programs designed like this are worth the effort. That's why all major retailers use a similar model, as pioneered by Wal-Mart.

Planning for data

The very first thing to consider when planning assessment is what you want to know, and why. Having a clear picture of how the information you collect will positively impact your organization makes the process easier and enables good decisions as you design your research or evaluation protocol.

Key in determining what you want to know is evaluating your questions in regard to their impact on your program and the ability to collect meaningful data.

For example, a hypothetical child nutrition program, which we'll call Kids.org, is planning a new child nutrition program. Their questions include: What is the dietary value of the average child’s meals? Does smoking in the home affect a child's appetite? What foods are both nutritious and appealing to kids? If these are significant issues, what Kids.org programs will impact them positively?

There are five concerns that must be addressed when creating the Kids.org assessment plan. Let's address each of the five key aspects of their, or your, assessment plan.

1:   What do we want to know, and why?
Assuming that good nutrition promotes good health and better learning, Kids.org wants to know the following about the kids it serves:
  • What is the dietary value of the average child's meals? Are they getting enough of what they need? Are any key nutrients missing from their diet?
  • Does smoking in the home affect a child's appetite? If so, is there a correlation with illness or learning/behavioral issues?
  • What foods are both nutritious, inexpensive, and appealing to kids? What can we afford to provide that the kids will like and is good for them?
2:   What information will tell us what we need to know?
  • What do the kids eat for some period of time. A detailed diary may be required.
  • A census of smokers in the children's homes.
  • A list of affordable, nutritious foods, taste tested with the kids.
3:   Has anyone already answered this question?
  • There are probably studies available to provide dietary information that is good enough. It will be hard to have enough diaries completed to gather significant data.
  • Kids.org probably will have to find the smokers' houses, though a questionnaire or personal contact with the parents.
  • Nutritional information for the foods Kids.org can afford can probably be easily obtained. Taste testing can take place by evaluating orders for food, or servings eaten, and by asking questions.
4:   How do we collect the data we need?
  • Research in online sources, including government, universities, journals, and general web searches.
  • Online, written, phone, or personal surveys.
  • Measuring and tracking food ordered over time. Frequently interacting with clients to ask what they think of the food, your organization, how you do business, and more.
5:   How will we analyze the data to inform our future actions?
  • Someone on staff knows enough to collect and analyze the data. Offices that use programs like SalesForce and QuickBooks can output reports into Excel for analysis. Many CRM/accounting systems offer advanced and customizable reporting to provide much of the data you need.
  • A local college or business school can provide interns who understand how to manipulate data to find the information you need. It's important to have these interns carefully document their methods and cross train staff to take over when the intern leaves.
  • Reporting experts can be hired on contract to periodically provide the information you need from your data.
Once you have the information you sought, you can modify your programs to be even more effective. Kids.org finds that there are tons of reports on average child diets in various locations, including a city near them that has very similar demographics. A quick check with a few of their clients indicated that their clients were eating pretty much what the study reports. The results of the study showed that kids ate too much sugar and salt, and not enough fruits and veggies. Kids.org starts an education campaign while also finding particular foods that provide needed nutrients.

Finding that smoking in the house caused bad effects on the kids, Kids.org started providing information for parents to explain the importance of a smoke-free environment for their kids. They continue to measure smoking vs. achievement to determine the impact of the smoke-free program, and modify it until it has the desired impact.

Kids.org also changed their food offerings and started requesting different foods from their donors. They discover that small variations in sourcing can make significant improvements in child nutrition.

Added benefits

Not only does Kids.org have a better understanding of its clients, it also has better impact data, and is able to make some changes based on what was found in the data. They can collect data continuously, and evaluate it at any time to assess their work. They can also provide greater insights and impact, which will please their funders. 

All of this applies equally well to marketing. You can, and should, design all of your marketing campaigns with measuring points built in. You can count clicks, calls, and customers. With opportunities like Google's AdWords, Twitter hashtags, specialized landing pages and other tools, you can evaluate the success of online campaigns. Online marketing can change by the minute as new data arrive. Print ads, direct response, press releases, even the Yellow Pages (yes, still good for some businesses), can be measured and adjusted. Obviously, donations provide their own inherent measurement systems, but even in fund raising you can measure other variables that enable you to better craft your message and delivery to improve giving.

The bottom line is the bottom line. No matter how you measure your success, be it families helped, revenue from a social venture, kilowatts saved, jobs created, or new money raised, you can determine significant measuring points. By taking frequent readings, and acting on the data you collect, you can make any organization work better and have greater impact.


No comments: