What to Measure, Why and When

Research, Assessment, and Evaluation

by Ben Delaney, © 2013

How many meetings have you attended in which the term “data-driven” was tossed about, its utter essentiality stressed?

And how many times did you have the feeling that no one had a clue what “data-driven” really means?

As one who has made a living doing market research, I learned to like data. Now I just love data. One of my greatest thrills at work in the last year was finally discovering a seasonal cycle in the sales of our social enterprise store. This information helped us plan a sale at the right time that doubled the store receipts that month. That's an example of data-driven decision making.

I can;t provide you with a course on statistics. Even if I were qualified to do so, we don't have the space for that level of detail. What I do hope to provide is a framework for thinking about data and evaluation that will make your work a bit easier.

Data doesn't just measure results

I think it is very important to use data to shape programs, both in initial planning and through a reiterative, ongoing analysis. Changes are driven by the findings, and often, the answer to one question raises other questions. 

Data-driven programs work this way:

• Program planning is based on research, with measurement points built in, plus
• Ongoing, reiterative analysis of the collected data used to refine the program and deepen understanding, then
• Programs are changed as new knowledge emerges from the data.

That is what a true data-driven organization does.

It takes rigor and discipline to work this way, but the resulting improvements in programs designed like this are worth the effort. That's why all major retailers use a similar model, as pioneered by Wal-Mart.

Planning for data

The very first thing to consider when planning assessment is what you want to know, and why. Having a clear picture of how the information you collect will positively impact your organization makes the process easier and enables good decisions as you design your research or evaluation protocol.

Key in determining what you want to know is evaluating your questions in regard to their impact on your program and the ability to collect meaningful data.

For example, a hypothetical child nutrition program, which we'll call Kids.org, is planning a new child nutrition program. Their questions include: What is the dietary value of the average child’s meals? Does smoking in the home affect a child's appetite? What foods are both nutritious and appealing to kids? If these are significant issues, what Kids.org programs will impact them positively?

There are five concerns that must be addressed when creating the Kids.org assessment plan. Let's address each of the five key aspects of their, or your, assessment plan.

1:   What do we want to know, and why?

Assuming that good nutrition promotes good health and better learning, Kids.org wants to know the following about the kids it serves:

• What is the dietary value of the average child's meals? Are they getting enough of what they need? Are any key nutrients missing from their diet?
• Does smoking in the home affect a child's appetite? If so, is there a correlation with illness or learning/behavioral issues?
• What foods are both nutritious, inexpensive, and appealing to kids? What can we afford to provide that the kids will like and is good for them?

2:   What information will tell us what we need to know?

• What do the kids eat for some period of time. A detailed diary may be required.
• A census of smokers in the children's homes.
• A list of affordable, nutritious foods, taste tested with the kids.

3:   Has anyone already answered this question?

• There are probably studies available to provide dietary information that is good enough. It will be hard to have enough diaries completed to gather significant data.
• Kids.org probably will have to find the smokers' houses, though a questionnaire or personal contact with the parents.
• Nutritional information for the foods Kids.org can afford can probably be easily obtained. Taste testing can take place by evaluating orders for food, or servings eaten, and by asking questions.

4:   How do we collect the data we need?

• Research in online sources, including government, universities, journals, and general web searches.
• Online, written, phone, or personal surveys.
• Measuring and tracking food ordered over time. Frequently interacting with clients to ask what they think of the food, your organization, how you do business, and more.

5:   How will we analyze the data to inform our future actions?

• Someone on staff knows enough to collect and analyze the data. Offices that use programs like SalesForce and QuickBooks can output reports into Excel for analysis. Many CRM/accounting systems offer advanced and customizable reporting to provide much of the data you need.
• A local college or business school can provide interns who understand how to manipulate data to find the information you need. It's important to have these interns carefully document their methods and cross train staff to take over when the intern leaves.
• Reporting experts can be hired on contract to periodically provide the information you need from your data.

Once you have the information you sought, you can modify your programs to be even more effective. Kids.org finds that there are tons of reports on average child diets in various locations, including a city near them that has very similar demographics. A quick check with a few of their clients indicated that their clients were eating pretty much what the study reports. The results of the study showed that kids ate too much sugar and salt, and not enough fruits and veggies. Kids.org starts an education campaign while also finding particular foods that provide needed nutrients.

Finding that smoking in the house caused bad effects on the kids, Kids.org started providing information for parents to explain the importance of a smoke-free environment for their kids. They continue to measure smoking vs. achievement to determine the impact of the smoke-free program, and modify it until it has the desired impact.

Kids.org also changed their food offerings and started requesting different foods from their donors. They discover that small variations in sourcing can make significant improvements in child nutrition.

Added benefits

Not only does Kids.org have a better understanding of its clients, it also has better impact data, and is able to make some changes based on what was found in the data. They can collect data continuously, and evaluate it at any time to assess their work. They can also provide greater insights and impact, which will please their funders. 

All of this applies equally well to marketing. You can, and should, design all of your marketing campaigns with measuring points built in. You can count clicks, calls, and customers. With opportunities like Google's AdWords, Twitter hashtags, specialized landing pages and other tools, you can evaluate the success of online campaigns. Online marketing can change by the minute as new data arrive. Print ads, direct response, press releases, even the Yellow Pages (yes, still good for some businesses), can be measured and adjusted. Obviously, donations provide their own inherent measurement systems, but even in fund raising you can measure other variables that enable you to better craft your message and delivery to improve giving.

The bottom line is the bottom line. No matter how you measure your success, be it families helped, revenue from a social venture, kilowatts saved, jobs created, or new money raised, you can determine significant measuring points. By taking frequent readings, and acting on the data you collect, you can make any organization work better and have greater impact.

Systems Marketing™ Ties it All Together

Why integrating consistent communications into every activity gives you way more bang for the buck.

System Marketing™ means that your marketing is a system, in the same way that your financial procedures form a system. In either case, the specific task informs and is informed by the total organization.

System Marketing directs that you align your goals, procedures, and communications to all pull in the same directions, with verbal, nonverbal, electronic, print, and attitudes all reinforcing the same message.

Most importantly, System marketing requires a deep understanding of the marketplace and the customer, and the ability to address the customer's expressed, implied and inferred needs and desires. This requires research. That research may be a simple as a comment sheet on your front counters, or as complex as a multivariate, blind, controlled test. The cost varies with the number of words used to describe the research. For example, putting a comment sheet on your front counter requires nothing more than a piece of paper, a pen, and some scotch tape. It will result in some of your customers providing valuable insights into your operation, with minimal expense. The multivariate, blind, controlled test will probably take several people several months, will require a series of letters after the author's names, will result in a colorful bound report with footnotes, and will cost appropriately. In either case, when research is done thoughtfully and with well-defined goals, it is almost always worth the money.

Let me  give you an example of how research helps. A while back, I was asked to provide a campaign to increase interest and visits to a nice retirement home in Mill Valley, California. As we talked, I realized that the staff had only the vaguest of ideas about why people chose to come, or not come, to live there. So we started some research.

First we conducted a written survey of the residents at the Redwoods, asking them what they liked about living there. Along with a few other questions, we also asked where they had lived before. From this we gained a lot of insight. People, as expected, liked the beautiful grounds and that it was easy to get into Mill Valley for shopping. The food was good, as were the maintenance and staff interactions. What surprised us was the most important factor in the move-in decision: Folks at the Redwoods really liked that they could bring their own furniture!

We then sent out a mailing, by postal mail, to a large population. I don't remember the exact numbers, but we mailed to over 10,000 homes in the Bay Area, to a radius of 40 miles from Mill Valley. Why 40 miles? Because that's maximum distance from which Redwoods residents had come. (Some came from further, but more than 80% had previously lived within 40 miles.) 

The mailing included a brochure illustrated with professional photos, taken on the grounds,  of people who actually lived at the Redwoods, sporting the headline, “Come Home to the Redwoods”. We emphasized the hot-button items we knew from the survey: just like home, extra secure and safe, bring your favorite furniture, and close to your friends, family, and familiar landscape.

This was the most successful direct mail effort I've ever done. We received a 24% response rate, and a 10% conversion rate, thereby beating expectations by a mile, and filling the waiting list. I am convinced that the research set the tone that enabled this successful effort. But equally important was the participation and buy-in of the staff, the truthfulness of the messaging, and the ability of the intake staff to model exactly what people expected. That's System Marketing!

Your organization can establish System Marketing as SOP (Standard Operating Procedure). Do the research needed to truly know your customers, the marketplace, and the outside factors that impact that marketplace. Share staff knowledge about current customer-facing processes and communications. Listen to complaints, and don't dismiss them as trivial. Be sure everybody is involved and heard. Review your mission, vision, objectives and impacts to be sure they are current and actually reflect what you do and want to do, and how it happens. 

Then you can unify communications and attitudes. Why attitudes? Because a large part of your customer and prospect communications is old fashioned conversation, as well as emails and other official, personal interaction. It is essential that everyone understand and buy in to the official message, and be able to reflect it in every action and utterance. Answering the phone, responding to an email, completing a proposal, talking at the bar at a conference: the language, style, conditions and outcomes should all reinforce your messaging. When you add that to consistent public messaging, including your website, logos, business cards, brochures, and advertising, then, my friend, you have a marketing system and System Marketing.

Frankly, I don't consider this rocket science. I have been thinking about it for a long time, and have seen the theory proven. Much of this is common sense, and just plain good business, be it for- or nonprofit. This book breaks down various marketing tasks and offers suggestions on how and when to use them. Regardless of the marketing mix you chose, when you keep system Marketing in mind, all your marketing will be more effective.