Prevent Good Metrics From Going Bad

Guest post by Chris Cummings of Product Management Meets Pop Culture.

"It’s not my fault!" Those four words underscore so many of the discussions (or, as I like to call them, blamestorms) that accompany product management.

A reasonable person might expect that to be less of an issue when we’re talking about metrics. After all, good metrics are:

• Aligned with the business
• Honest
• Actionable
• Measurable
• Objective

Who could argue with a number that’s honestly calculated, from credible sources, without bias or subjectivity?

Turns out, many people. Especially if they’re on the wrong end of a good metric.

Metrics Affect People

Many discussions about metrics in product management do a great job of looking at things logically but remain silent on the consumption of those metrics or the behavioral reaction to those metrics.

We measure what we figure is important. And if your job is connected to a metric, and you want to look good at your job, then you want that important metric to look good, too. It’s human nature.

Avoiding The Misuse Of Metrics

As a Product Manager, we play a key role in how metrics are collected and conveyed. Here are three tips on how to avoid the misuse of metrics.

Use the right mix of metrics

Too many metrics can lead to analysis paralysis. Too few metrics and you can end up really fine-tuning one particular aspect of your product (the part being measured) at the expense of something critical that’s not being measured.

For example, in online games, there are many things that could be measured (visitors, time spent, etc.) but the most critical metrics generally relate to viral growth (retention and referral) and total sales (revenue per user per day).

The goal is to determine the right mix of metrics to provide an insightful, actionable, balanced view of your product. Measuring in this way amounts to leadership, and encourages others to think about the product holistically as well.

Use metrics to make decisions

Want to a) kill a metrics program, b) get a lot of people mad at you, and c) fail at an importance aspect of your job? Then pay no heed to applicable metrics when making a decision where the collected data could prove informative and valuable.

A reasonable person might say, "Well, duh." But I’ve seen this happen multiples times within multiple organizations, and the end-result is never good–particularly in terms of morale.

No one should be a slave to the data, but if you can’t legitimately explain why you’re opting for what’s in the Box when all the applicable data says to take what’s behind Door #2, you’re demonstrating a flagrant disregard for the agreed-upon decision-making and encouraging others to play the same way.

Remember: Metrics are not maces

It can be really, really tempting at times to wield metrics like a mace and figuratively bash in the heads of those who seem happily misinformed and proud of their ignorance.

Don’t do it.

By all means, convey the information in a way that it can be received and processed. But if you set an example of using data as a weapon, it suggests that others do the same — which inevitably contaminates the data as it gets twisted, sometimes right at the source, in a nuclear arms race where nobody wins and the organization suffers.

"It’s Not My Fault!"

People are going to feel how they’re going to feel and react how they’re going to react. But if you use the right mix of metrics to get a full view of the product; use metrics in your decision-making; and create a context where data is use constructively — to solve problems rather than assign blame — then you’re on your way to diffusing the danger of collecting metrics and using them to propel your product forward.

Chris leads product management for the Lycos Network’s games division, including Gamesville.com and white label game solutions. Prior to Lycos, Chris served as product manager for GameLogic, Inc., where he drove alignment among business, technical, and creative resources to create casino-style online games for land-based casinos in the United States and the United Kingdom. These games are available at Foxwoods Resort & Casino, Dover Downs, and Trump Taj Mahal Casino Resort, among others. Chris graduated with distinction from Merrimack College with a degree in English and a minor in Religious Studies. He received a Master of Business degree from University of Phoenix.

Interested in being a Guest Blogger on The Product Guy? Contact me.