“To know how to measure the success, is to know how to achieve it”
A few days ago while discussing complexities of managing a product portfolio, Uday asked me a simple question: what is the most important metric one should use to measure product performance? I suddenly felt dumbfounded because I didn’t have a clear answer. Though I could have used my heuristics to throw suggestions such as ARPU (average revenue per user), engagement, and churn rate, I knew that the thought process was more important than the metric itself. A good metric allows taking action, a bad one only provides some data. So Uday and I decided to jointly answer this question by creating a framework, which Uday will illustrate with examples from his experience. So here are the four steps:
1) Define what the big success looks like
This is the most difficult step and perhaps half the work. This requires you to see the big picture and yet relate to all the parts. But you can perhaps do it much easily, if you start thinking of success in pure binary terms. So everything succeeds, if this one or two things happen right, else everything fails.
Uday: At my previous startup, I was looking at the dashboard which tracked conversion funnel. I knew that just looking at growth in number of visits, free users, and paid users would only paint a rosy picture, and not necessarily the most accurate one. Clearly, the single most important goal for my company to succeed at this stage was ‘growth in paid user’ and ‘longevity of paid users’.
2) Break down and prioritize on the basis of need and controllability
Now that you have one or two big metric from the first step, you need to start breaking them down. Once you break it down in different parts, start prioritizing on the basis of what answers your most pressing needs and what you can control. It’s okay to combine multiple variables to create a super metric.
Uday: I decided to steer away from vanity metrics like number of page views, which were simply non-actionable for me. I focused on 1) (CLV), which captured engagement (longevity) and reflected whether we were on track to become a sustainable business (revenue/growth); I used cohort analysis for this. 2) Net subscribers (new paid subscribers – cancellations on a given day), which helped us capture change in two important things in one attempt – number of free users converting to paid users and cancellation (super metric).
3) Locate the right data and evaluate cost
Now you know what you want to track, you need to find information to do that. Remember, available information can be cheap, but retrieving it in desired format, not always. If some information is easily available and helps you answer ~85% of what you want to track, make adjustments in your metrics and go ahead.
Uday: Although all data was readily available, I had to make additional efforts for conducting cohort analysis with CLV (the cost!). Later on I got a dashboard developed to do the same, without having to resort to excel every time.
4) Develop, present, and revise
This is more like a hypothetical step, before you actually present the final metrics. Imagine that you’re presenting your metrics to someone important. Think the questions they would typically ask. If you are able to justify your choice of metrics and explain why you didn’t include a lot of other metrics, you’ve done a good job. Else, revise.
Uday: I made all the reports / dashboards simple, comprehensible, and differentiable. I was ready to answer questions like how I was already baking in conversion rate, active users, and paid subscribers in my choice of metrics. Sometimes I would separately include information for my marketing team which would typically ask for these baked-in metrics. Lastly, since cohort analysis was the basis of my analysis, I’d put a foot-note for those who didn’t understand it well.
Remember
- Understand the difference between absolute and relative terms. Often, we need a combination of both to create an accurate picture.
- See the forest for the trees. Your selection of information or metric should try to capture major trends, opportunities, or challenges.
- Don’t go online right away as you start brainstorming. You have to understand what answers your business questions, not someone else’s.
- Go online once you know what questions you’re trying to answer. Taking ideas from how others did it efficiently is good once you understand your problems better.
- Keep it simple. Does it require any explanation? 🙂
A well-thought metric allows us to deal better with uncertainty of decision making, with a precision of action. And the next time someone asks me to include another metric in my analysis, I don’t want to respond by saying ‘sure, I will include that next time’. Because everything which is necessary, would already be there!
(Co-author: Abhyuday “Uday” Chakravarthi, image credit: Dennis van Zuijlekom)
The test, or the challenge, as we call it, gives us a good understanding of a candidate’s programming proficiency. If they have performed well in the challenge, we know the candidate is good. We then aggregate their coding activities from online sources like 