Lean Analytics – Qualitative and Quantitative metrics
I have been doing a lot of customer research recently and also have had my head deep in lean analytics. There are a few things that you need to keep in mind about metrics that is really important:
- Have a clear idea of the questions that you need answers to before performing research
- Understand in which scenarios are qualitative vs quantitative metrics relevant
- Know how data will change behaviour before performing research
If you are not sure of above, it is likely you will get lost in the metrics, interpret the data incorrectly, and it will most likely result in inaction.
So let us take startups, typical questions that need answers are:
- Is the problem worth solving? Do people feel that the problem is urgent enough that they are willing to invest time/money on it.
- Who is your target customer?
- What features should you build for the MVP?
In order to answer the question if the problem is worth solving, you typically start of with qualitative research. The primary form of research is problem interviews with prospective customers to understand pain points. The questions you ask are open ended and there is going to be nothing formulaic about the answers you get.
An example scenario is if you decide that you want to open a restaurant. You first conduct interviews to learn about people in the area, what their likes/dislikes are, what options are already available and the general trends.
It is guaranteed that the data you get is going to be hard to measure, and what you are looking for is patterns and insights. The danger in qualitative metrics is it is subjective and can easily be faked. It is easy to convince yourself that your assumptions about the problem are correct.
“Qualitative data is messy, subjective, and imprecise. It’s the stuff of interviews and debates. It’s hard to quantify. You can’t measure qualitative data easily.” Lean Analytics Book
Once you have done a good number interviews, and validated that the problem is indeed worth solving, you can move to quantitative analysis. Again, you have to be very clear what type of uncertainty you are trying to quantify.
In the case of the restaurant example, if you ask people how they find out about various restaurants, what will you do this with this information? Will this influcence the channels you choose? If you ask people how much they spend on food weekly, will this inform your pricing strategy?
Asking questions is the easy part, but what behaviour will change when you look at the answers is difficult.
There are several tactics for quantitative research, common examples are surveys and landing pages.
“These give you the opportunity to reach a wider audience and build a stronger, data-driven case for the qualitative feedback you received during interviews.” Lean Analytics Book
Surveys can have quantifiable questions that can be analyzed statistically, such as ratings, true/false statements, and choosing from a list of options. The resulting quantitative data is easy to understand. It can be ranked, and it is scientific.
If quantitative data answers “what” and “how much,” qualitative data answers “why.” Quantitative data abhors emotion; qualitative data marinates in it. Lean Analytics Book