What is necessary in a case study? A few basic elements are necessary for a case study to have objective validity:
- State the campaign costs. “The test campaign spent $32,000 over 45 days.” (Why state the cost and time of the campaign? So other companies can know if the case study is relevant to their situation. You can’t compare campaigns where one spent $1,000 over six months and the other spent $2 million in 60 days.)
- Describe the tests “We used A/B split tests as follows. We used multivariate testing as follows.” (Why describe the testing? The case study should use a good set of data. Otherwise, skeptics will say the low results were due to poor ads. For example: “Before we started the campaign, we tested twelve ads and selected the best performer for the case study.”)
- Describe the tracking. “The documents were tagged to show traffic, clicks, leads, and conversions in Omniture web analytics.” (Why describe the tracking process? So we can see if the authors know what they’re doing. If they aren’t tracking, then the data is probably flawed.)
- State the full results. “30,000 leads converted into 10,000 sales, i.e., 33% conversion rate.” (Why state the number of leads, conversions, and the conversion rate? It’s useless to say that a campaign got 100,000 leads if it doesn’t include the conversions. Anyone can get 100,000 junk leads by just putting a contest on Facebook. What counts are the conversions (registrations, sales, etc.)
- Use statistically reliable data sets. If the data set is too low, the results are statistically unreliable. For example, “We polled 30 shoppers and 25 liked the product, so we predict a 83% sales rate!” (What’s wrong with a low sample? With only 30 samples, the margin of error is so large that results are random. When they go to market, that 83% could turn into 2%. If you rely on their poor data, you’ll get bad results. To get a result with a 3% plus/minus range, the case study should test at least 1,067 people. If you increase the survey to 2,401 people, the margin of error reduces to 2% plus/minus. For more, see the Wikipedia article on the margin of error.)
- Describe the control group. “We selected ten US cities with similar demographics. We ran the campaign in the following seven cities. There was no campaign in the following three cities. This chart shows the difference in response. In seven cities, results changed by X. In the three cities, results changed by Y.” (Why use a control group? Because without a control, there is no proof that the campaign had any effect, neither positive nor negative. The study can conclude “The campaign produced a 12% increase in sales!” Did another division release a new product that got attention? Did three major competitors not release new products? Worse yet: did the market and competitors grow 21% last year, so the campaign’s impressive 12% success actually a serious failure? See a list of control cities.)
Any case study without these points is just an anecdote. An anecdote can mislead you either to do something (“hey, it worked for them, so we should do it!”) or not do something (“well, it didn’t work for them, so we shouldn’t do it.”) An anecdote is useless as evidence because it is not objective, so it can’t be used to make decisions.
A case study with these points is useful to others. They can use the results as a guide for their own campaigns.
How to Use this List
When you look at a case study, see if it includes these items. It’ll help you in reading the case study critically.
What about Case Studies for Content Marketing?
In nearly all of the content marketing case studies that I’ve seen so far, most of these elements were missing. Does content marketing actually work? I’ve talked with a number of agency directors and published authors. None of them have a valid case study. It’s plausible that content marketing works better than PPC. The Kapost Content Marketing ROI document makes sense. But without a valid case study, there is no proof that content marketing works.
The critical item is the control group. If that is missing, you can’t tell if the campaign had any effect. Control groups are standard in experiments in physics, biology, and medicine. Any test of new medicines in pharmacology must include control groups. If a test lacks a control group, it has no validity. However, very few, if any, marketing case studies use a control group. See Wikipedia, Control Groups
Mark Spanner posted a case study in the Content Marketing Group forum at LinkedIn. He concludes: “Results: 3.2 times more leads rated as “hot” provided to salesforce. Franchise sales increased by a factor of almost 11 (3 to 38)” (i.e., an 11X sales increase.) He includes screen shots and a description of how they did it. See his case study at Spaner.com (at bottom left of his site, click “Conversion Marketing“) His case study isn’t complete, but it’s the best so far. I’m working on writing a proper case study and will release it when it’s ready.