What does it mean to be a data-driven Product Manager?
SQL queries? VLOOKUPs? Definitely. Add to that: metrics, data, KPIs. These terms have become commonplace at technology companies. If you're interviewing for a Product Manager role in 2019, I guarantee you'll be asked some of these questions about your past experience:
- What were your KPIs? Why did you pick those?
- Talk about a time when you used data to make a decision.
- What metrics do you use to illustrate if a feature is successful or not?
- When is it not appropriate to use data to make a decision?
- Sketch out your data model.
- Talk about a time when the data suggested you should go in a different direction from your strategy.
You'll need to succinctly explain what you measured, how you measured it, and most importantly why you measured it. You should demonstrate the ability to hypothesize and associate metric(s) to evaluate.
Demonstrate an ability to focus. "Out of the 10 things I could have measured these three were most important". Focus may be the biggest value a Product Manager can offer. The ability to say these are the few things we should measure and why. And then have the awareness to know when those metrics have served their purpose and it's time to measure something else.
Demonstrate the ability to question. Was there a time when the data misled you? How did you adapt? What was your goal and why was monitoring metrics part of the solution? A concerning answer for why you measured a certain metric/KPI is "we've always done it this way". Even if you do resort to status-quo industry standard measurements, explain the reason for that. It will demonstrate that you at some point questioned the status-quo, and received a sufficient answer that resulted in you maintaining it.
Some practical examples.
Today, tech companies are vying for your attention. YouTube prefers you watch their videos instead of Netflix's, or going to the movies, or reading a book. They want your time allocated to YouTube. This is why when you finish a video the next one is already queued up and a long list of tantalizing recommended videos is in clear view.
The way to measure attention is through Retention & Engagement.
Retention, getting you to come back (e.g. open YouTube X times per month). Engagement, getting you to use the product (e.g. watch 10 videos per day on YouTube).
The gold standard Retention measurement is "N-Day Retention". The goal with measuring Retention is to understand who and how often is coming back to your product. Amplitude, a tool I currently use has a great overview of measuring N-Day Retention.
Engagement is about measuring who and how often is performing the "key action" in your app. In YouTube's case one of those actions may be "watch video". The gold standard Engagement measures are: DAU ("dow"), WAU ("wow"), MAU ("mm-ow") DAU/MAU ("dow-mm-ow"). These metrics measure: Daily Active Users, Weekly Active Users, Monthly Active Users. These are the unique number of people that perform the key action (such as "watch video") on a daily, weekly and monthly basis.
DAU/MAU will demonstrate how engaged your user base is by reflecting the % of monthly active users that come back everyday. This can also be a measure of your apps "stickiness". Again Amplitude has great overview of this concept. It's also worth noting that although DAU/MAU is an industry standard metric for Engagement, it has shortcomings.
If your focus is Engagement & Retention, DAU, WAU, MAU, and DAU/MAU are great pulse metrics. Define a company-wide standard to an active user. Be very specific. For example an active user is an account holder that watches at least 10 seconds of video in 24 hour period. Then measure them consistently. They will help track if you're product is improving over time, and signal if things are getting better or worse.
You will also need to come up with metrics that are more specialized to your product and goals. What metrics are you going to try to improve that will directly impact DAU, WAU, MAU?
Here is a famous example from the early days of Facebook. When Facebook opened up beyond colleges, they entered hyper user acquisition and retention mode. Facebook's growth team united around the following insight: 7 friends in 10 days. The team discovered that users that added 7 friends within 10 days of creating a Facebook account were likely to remain an active Facebook user. Therefore their focus (features, experiments, design decisions) was channeled to getting as many users as they could into the "7 friends in 10 days" cohort. This "north star" behavior metric became one of their primary metrics for growing their engagement KPIs.