How to identify meaningful product metrics
If you check out my ../Product Kick Off Template, you'll see I have a short section on the top Product Metrics.
To illustrate how I would get to them, I'm going to do a use-case on Kleros.io
You'll see that much of the upfront leg-work is the same as what is in the ../Product Kick Off Template. This illustrates why that template is important as a foundation building block to downstream work and re-usable summary throughout the life of the product.
While this can be used in "normie" products, Web3 creates a new level of urgency: What is a Web3 PM For?
So let's begin!
What is Kleros' mission?
"Enabling access to justice and individual freedom" according to their about page.
What is their core product?
Kleros provides decentralized arbitration for disputes and provides transparency on the blockchain.
Who are their core personas?
According to their websites, the types of organizations are in "e-commerce" and "finance." Here are some possible personas to consider:
"Wronged buyer" in a transaction
In a given transaction, someone often feels "cheated" or "wronged."
As the "wronged party," this persona feels something unfair happened and wants resolution and justice. This is likely in the case of a being sold something by another counterparty, and they are dissatisfied with the outcome (rightly or wrongly).
The "buyer" has likely provided money and does not feel got something in return as expected.
Their pain is difficulty in recovering their amount if the seller doesn't have a legitimate warranty or refund service.
"Wronged seller" in a transaction
The seller also feels that they were wrongly accused of not acting in good faith.
They have the money but feel that they are being asked to return the money or fix the item or service after it has been delivered under their terms of service.
This persona really cares about their reputation, and then strongly consider the costs associated with remediation and refunds.
Marketplace platforms trust and safety leader
Although marketplaces explicitly try to stay out of any remediation because it hurts their business model (they want to be "hands-off", they often need to do so to satisfy both "wronger buyer" and "wrong seller." Ultimately this gets folded into reputation management and user experience.
They rarely provide a remeditation process and usually address it by offering required or optional insurance products to cover faulty products.
Doing so off-loads the risks of a fraudulent transaction to an insurance product and gives a more predictable unit cost. If they care about the UX and can manage the cost, they will include it within the total user experience as part of the platform fees rather than adding additional costs to the users.
Payment systems fraud team
Credit cards, banks, and perhaps the payment rails providers themselves (e.g. Stripe) may want to reduce those disputes.
Credit card companies often have a dispute resolution and, in the United States at least, because credit cards are so profitable, often eat the cost of a dispute on behalf of the "wronged buyer."
For them, the pain is to reduce fraud and reduce fraud payouts.
They already have sophisticated fraud detection services, but the dispute resolution is primarily manual and tends to favor the "wronger buyer" persona.
Which persona drives the business?
In this section, it's important to think across all personas but still focus on a primary persona as part of the kick-off.
This is where "product management" kicks in to make a more detailed case for the trade-offs and the ecosystem dynamics. I'll do a preliminary one, but this should really involve much more research and metrics gathering to propose this.
What's key is that the Product Leader picks a primary persona but builds within the reasoning or the roadmap a way to address the entire ecosystem despite focusing on a single persona.
This is a key skill for the Composable Product Manager Project -- because without that ability to manage trade-offs and risks, you are just a project manager or feature herder.
| Persona | Pain | Impact on Klerios | Rank (10-1 10 highest) |
|---|---|---|---|
| Trust and Safety | Time and Brand | Slow but rapid scale | 8 |
| Payment Platforms | Cost and customer retention | Very, very slow but broad scale | 6 |
| "Wronged Seller" | Hassle factor, brand risk, and write-offs | Slower but direct outreach possible | 4 |
| "Wronged Buyer" | Highest individual incentive | Difficult to reach other than through consumer advocacy organization, burden still on Buyer | 2 |
Who is the Primary Persona and why?
After reviewing this at a high level balancing time to market, impact on the Kleros.io business and the impact on the buyer, Marketplace Trust and Safety teams seem to be the highest value.
Because the actual end-users of "Wronged Seller" and "Wronger Buyer" will experirence their pain and seek resolution within a marketplace context, it seems like this is a good place to focus on to get both.
Pros
- Marketplaces, particularly the smaller or emerging ones, will be more easily contactable directly.
- If we are successful, we will be able to address both sides of the market in transactions at the point of pain, scaling on both sides of the market
Cons
- The marketplace or platform ultimately becomes the Customer. Does this fit the ethos or the business?
- This will take time to get initial engagement
- This could give signal on market demand that is initially discouraging
What are the metrics?
Let's consider a general TOFU approach and then some product life-cycle considerations (if needed).
I think the product most needs outbound acquisition before evaluating product life-cycle. These product growth metrics should target the persona at 100+ marketplaces.
These are a few that seem to make the most sense and then scored by Rank (it happens to be in order.)
| Stage | Metric | Notes | Rank |
|---|---|---|---|
| Awareness | Click on message link | This could also be just an email open, but I think email links are better although a much lower number | 1 |
| Engagement | Content engagement | This could either be download or attend a webinar, this seems to be important to measure | 2 |
| Engagement | Onboarding engagement | Putting a link on their page to an onboarding experience | 3 |
| Engagement | Platform users onboard | The link should onboard the user with a success metric | 4 |
As you can see, most of these are focused very much on the awareness and engagement on the Marketplace Trust and Safety persona.
This looks different from what seems to be the primary users, the "Wronged Buyer" and "Wronged Seller." Their UX will be very important, but for the core value of the product, the adoption for smaller Marketplaces is critical to:
- Give a repeatable baseline for customer acquisition
- To gain local network effects within a given platform (better to have 90% of a small platform than 3% across multiple platforms)
- Controllable without major code or product updates (important consideration since we work on the blockchain)
Conclusion
So the top three metrics for Klerios are the top three above:
- Awareness (% of people willing to actually engage with message - slide deck, landing page, email)
- Engagement (% of people will to engage, we need to evaluate what that means in terms of content -- I prefer things with deeper engagement that is async)
- Product Engagement (% of people who put the link and embed it in the product)
Of course, the better the blockchain design is the more interesting we can get on the actual flywheel metrics. Check out Why Helium is a great project to learn the power of blockchain.
Content Status
- Instantiate 2023-02-03
- Publish 2023-02-03