How would a Product Manager evaluate developer tool products?
If a company has scaled to where it needs for a Product Manager, one of the things that will help them and the product is an initial framework for what makes a good developer product.
I've found that most PM's for developer products, in an interview, don't have some starting point of looking at a developer facing product and being able to reason through what the considerations would be to tell what adoption is.
Instead, the approach (which isn't prima facie wrong) is process -- talk to dev rel, talk to developers who are customers, and the "look at the data."
The second comment which is also not wrong but to me needs better clarity and rigor, has been "I'll look at the data."
While it is true data can be helpful (more data is always better than zero data), the question really is getting at how would a product manager approach understanding a given tool and it's likelihood of succeed in the market and building a business.
While the generic product management flow can work, there's enough commonality that a developer-specific framework as a starting point is more appropriate and shows a level of familiarity.
So, to me, the critical answer should begin with who is the persona?
Just as there are personas for consumer facing and b2b products, there are also personas that needs details for a given market.
If a candidate is talking about their product, they still need to start with the persona or personas they identified and targeted.
What are possible developer personas?
Think about products that have been successful and reverse engineer the personas.
Replit - the beginner or student programmer
Postman - the API builder or integrator
Warp - the all-command line power user
Sourcegraph - large legacy code base developer
ngrok - edge deployment developer
Each of these can then identify unique problems and goals and journeys.
But without anchoring on who is the customer, it becomes very hard to be credible in defining a product.
For me, this is an important signal, and it actually is one of the areas where a PM can help differentiate from an engineer building an engineering product "for themselves" as sufficient.
This could be true, in which case, the need for a PM isn't critical.
But if the engineer is building a product that they themselves don't use while working on building the product, a PM needs to help "get out of the building" and help define who that really is (because the engineers can't self-reference).
If used in the context of a specific tool (or even a tool they have worked on), being able to structure multiple personas and then drill into one shows breadth and depth.
The second area of focus would be to talk about what is the Persona's Job to be Done (or what are their goals and overarching story and journey).
For example, in the case of Replit, it might be "to learn how to get to live, working code with as few distractions as possible such as installation, environment, and sample code."
In the case of ngrok, it might be, "to deploy code built on localhost without waiting for devops or needing to become a networking expert on different cloud environments."
In the case of blockchain tooling, it could be, "to let a DeFi dapp developer add dashboards and metrics quickly and easily."
The next area would be to talk about what the goals would be, and what features you like (if you're talking about an existing tool) and what areas you would want to improve or address.
To me, breaking down features that you would propose to build or that you particularly like for the tool (and it could just be one feature) is useful.
I think also, because much of the workflow is similar, regardless of the persona (with some minor adjustments), I do think it's fair to ask about the life-cycle or workflow from the start -- and while documentation is a start, there's much more:
- Getting Start documentation (but is that always necessary -- some tools don't require this)
- Packages / SDK
- Environment
- Templates, scaffolding
- Command line or CLI interfaces
- API keys
- Debugging / Telemetry
- CI/CD / Testing
- Deployment
- Product Metrics
This is just one, there are ways to condense it or focus on it or perhaps put a different framing or focus.
For example, a developer of firmware for medical devices may likely have some similarity, but also unique and critical attributes compared to say a mobile developer.
The last consideration for evaluating a product is its ability to gain "traction" in the market.
Typical marketing doesn't really work, so having a sense for what really helps a developer product take off matters.
The most important is being able to have a feel for "developer experience" -- what are things, perhaps elements along the workflow or lifecycle -- that really make a give product stand-out.
What is the product element that you appreciate for a product's "aha" moments?
The reason why this is important is that often those little subtleties is a good ground truth for initial organic word-of-mouth growth and adoption.
Engineers don't always build products that other engineers want to use.
Sometimes engineers have a high threshold for pain that when they build products, they bake in all the pain because they are numb to it already; or the tool is better than nothing (which isn't really enough).
What are elements that make things really "just work"?
Some for me that I have observed:
- Syntactic "sugar" -- the Ruby language or Ruby on Rails ability to chain and add methods that made sense and more human readable; ngrok's
.chaining of ingress features like rate limiting or security. - Scaffolding - this is becoming very common, but still talking about experience, such as using Firebase Functions CLI tool; YAML in Kubernetes (hey some people like it); css frameworks or "compilers" like sass;
- Reusability or "done for you" - Postman's experience of working with APIs reuses authentication, for example, across grouped API calls saves time and works.
A good answer ties the feature to an important attribute or product principle which feeds the design or shapes your hypothesis.
As with these types of interviews, the correctness of a given answer is less important than both structure -- did the person think as a PM and try to frame things -- and context -- this should show some domain awareness or even better, demonstrates real-world on the job experience for a product being built.
E.g. if you like your own product and pick that, then you should be able to go pretty deep into the details, into the trade-offs, what made you proud of the feature?
The last part that I consider a real value of a product manager is to identify the trend, the shift, the theme, the edge -- some larger hint at the strategy for the tool in terms of the product and its market.
To me, the ability to reason through the market dynamics of adoption does matter.
The "just doing" of product management may have been sufficient when I got started -- talk to users, write up requirements, review the data -- but markets move fast, competitors and adjacent solutions are plenty, and developers are also subject to the shifting tides of digital trends.
To me, not being able to contextualize the broader market -- and demonstrate this both with your own market and by just thinking about the domain space, if it's entirely new -- is mid.
At times, candidates who failed to do this I would still pass along on the interview, but as I write this article, I ask myself, "What do we need a PM for when it comes to developer products?"
I don't like one answer that says "what to make the product and the team successful."
What makes a product successful is too vague....what if cold-calling prospects would make it successful? What if filling in for PMM because they can't understand the product well enough to market would make the product successful?
What if being the scheduler, the project manager, the herder of sheep is considered making the team successful?
I think the underlying skill that puts many of these things together is the ability to deliver business result by attracting and delighting paying customers with the higher return on resources.
An engineer may be able to describe why a set of features would please him or her.
But they may not know whether that is a market of one or a market of one million: but the product manager should be able to do a Fermi estimate on this.
An engineer may say the opposite, and say that the market is too small targeting a specific use case; but the PM should say whether that is actually a wedge into a larger market or not.
An engineer may love the complex, generalizable architecture that can do "all the things"; but the PM should be able to look at what aspects a given audience will really want, and what will differentiate this tool for the other options on the table.
To me, this level of judgement requires an ability to categorize markets, user behavior, competitive positioning, while also going deep into the tactile, almost craftsmanship, experience of the tool itself.