Lean Analytics – What makes a good metric?
The Build->Measure->Learn loop forms the heart of Lean Startup, everyone gets and focuses on the Build component, the Measure component is the weakest link. As David Bland explains in this article, not a lot of people are doing it correctly for various reasons.
Ben Yoskovitz and Alistair Croll wrote the Lean Analytics book and it is a deep dive into the Measure component.
Analytics is about tracking metrics that are critical to your business and this is directly related to your business model e.g. how you make money, how many customers you have, how you acquire customers etc.
They have defined four characteristics that make up a good metric:
It should be able easy to take a metric and compare it, compare it against a group of users, time period or competitors e.g.
“We had a 20% conversion rate of customers opting for a digital receipt.”
In the above case, it is not easy compare the 20%, was the 20% applicable to a specific time period, or group of users e.g. users who opted in October. A more useful metric would be:
“We had a 20% conversion rate of customers opting for a digital receipt in October”
This helps to understand how you are moving towards your goals and gives a lot more context.
Easy to understand
If you are sitting in weekly meetings and debating what a metric you selected means, it is a clear indication that the team does not understand the metric and it needs to be simplified. It is critical to choose metrics that are easy to understand and this is an important step towards building a data driven culture.
“If we want data to be used effectively as an input and filter into the product management/development process, then it’s on us to make the data simple for people to understand.” Ben Yoskovitz
A good metric is a ratio or a rate as it is easier to act on and ratios are inherently comparative.
A good example is distance travelled, this is just information, whereas distance travelled per hour is a lot more useful, it can be used to calculate how long it will take to reach the destination.
This is the most important characteristic of a metric, it should trigger a change in behaviour.
Experimental metrics helps to optimise the product, pricing or market. It is important to agree on what behaviour will change before conducting the experiment e.g. If conversion rate for blue button is twice that of a gray button, then the button will be blue. If more than half the respondents indicate that they won’t pay for the feature, don’t build it.
It is important to draw a line in the sand and follow a disciplined approach, and this is the most difficult part!