Measuring the Success of Your Online Developer Community

What to measure at different stages of your community's development.

Here’s a question that nags at every community manager. Is my online developer community truly delivering value? Communities need to show value for the participants as well as the sponsors.

There are plenty of metrics to help you make that judgment, and it’s critical to find the right one. There may be no single metric that proves success, but likely an evolving set of metrics.  Initially, communities should focus on the growth of membership, content, and interactivity.  As the community grows and matures, the emphasis shifts to metrics tied to outcomes such as the percentage of questions answered.  The metrics should represent benefits for both the users and the community stakeholder.

Why focus on user benefits?  Because the reality is that close to a third of communities fail and only 13% of community managers feel confident that they even know what metrics to use to measure success. Both statistics come from CMX and Leader Networks.

We took a deep dive into measuring success in a recent webinar. Let’s take a little time to share the highlights.

Different Metrics for Different Stages

Community failure is often attributed to lack of adoption by users or lack of support from the sponsor. There are common challenges faced by both internal and external communities – both need to provide value to users and benefit that the sponsoring (and funding) management can measure and value.  Both also face competition from other tools.

For instance, an internal community often struggles because developers are reaching out to each other through channels and messaging platforms that predate the community. Old habits are hard to break even if the new processes are better. An external community may flop because non-employee devs have found other forums to communicate through, especially ones that already have well-established content and a critical mass of active users.

And communities can fail if they lack support from one of the constituent groups: A community that gets a lot of support from management but has no champions in the developer community will never blossom. Conversely, if the developers embrace it but management doesn’t see or understand the ROI, the community may not get the funding it needs.

To help your community succeed, take its temperature in different ways at different stages. This will help you better understand the health of the community as you launch it and build to maturity or take on an existing community, diagnose it, and guide it back to a positive growth trajectory.

 The Metrics That Help Sell And Keep an Online Developer Community Vital

The cost of a community is often justified by a return on investment in the following areas:

  • Case deflection. A mature community can answer a lot of questions that participants would otherwise open a ticket for.
  • Internal efficiency gains. Particularly in large, spread out organizations, a community helps increases the chance devs can learn from each other more quickly.
  • Company loyalty. Whether a strong community encourages devs who build on the platform to stay with you or an internal community gives your devs a sense of belonging that reduces the chance they’ll leave, both can lead to cost-savings.
  • Product innovation. By encouraging idea exchanges, comments, and suggestions your product team receives a wealth of ideas.
  • Developer sentiment meter. Too many bugs? Too few resources? A new product launch that hits the mark? You’ll hear about it in the community. Communities also provide a quick way to survey participants.

Build Success by Understanding a Community Lifecycle

Metrics are critical to measuring success, but not every metric makes sense to use at every stage.

When the community is first launched, you should measure how many people are signing up and track what they’re doing after signup. Are they engaging through posts and comments? Are they using all the features you’ve rolled out?

It’s OK if there are more readers than commenters and posters initially, but you’ll want to see the scales tip a bit each month. If it is an external community, it’s OK to have internal staff posting initially, but you’ll also want to see that evolve to include more users offering answers and advice.  If engagement metrics are sluggish, you’ll need to step up your marketing efforts.

In the growth phase, carefully measure the number of contributions and comments. Your goal is to have a broad and deep cross-section of customers, employees, and developers participating in the community. And, of course, at this stage, customer service tickets should begin to trend downward as more people find solid answers to their issues through the community.

Once your internal community reaches maturity, start measuring how much you’ve saved in support costs and how much higher productivity is (for internal communities). For external communities, the community should begin to play a role in bringing in new customers and generating product innovation and ideas.

The Nitty-Gritty of Measuring Cost Savings

OK, you’re thinking, “Showing that my community has grown is nice, but to get long-term buy-in from executives, I’ll need to show how the community is saving money.”

If one of your community goals is to reduce the number of calls to your service center, you can calculate the return on investment by multiplying the cost of a service call, email, or chat by the percent decrease in tickets opened. Let’s say that a service or support ticket costs $6 dollars, and since the community launch, the number of tickets was reduced by 1,000 – that’s $6,000 in savings that can be attributed to the community.

Over time, you need to factor out company growth. So, look at tickets vs. customers or users each year to judge how much growth affected customer issues. If you had 1,000 users and 100 tickets in Year A, and 2,000 users and just 150 tickets in Year B, you still have cost savings attributable to the community.

Another way to show the value of the community is to measure sentiment. An explosion of negative comments around a patch that doesn’t work, a security issue that isn’t being addressed, or some other glitch is like your canary in the coal mine. It can help you understand an emerging issue before it drives developers, or customers, away. Can you measure the time it took to identify a similar problem before you launched a community with the time it took after the community launched? Can you indirectly tie this to renewals, platform growth, or customer satisfaction scores?

Communities also help you meet your SLAs (service level agreements) for answering questions and addressing issues.

While support is typically the first reason to adopt a community, other benefits often emerge over time. One advantage for mature communities is the ability to detect issues early by isolating variables. This aids quality initiatives and beta programs. The community can also become a great source of innovation that helps drive product direction. That can happen through careful analysis of the discussion content or by using features to capture new ideas and gauge applicability through commenting and voting trends. Each of these benefits will have its own set of metrics and success goals.

Metrics From Successful Online Developer Communities

Hortonworks measured the success of their community by looking at unique visitors by department, overall users, and the ratio of community users to Hortonworks employees. They also looked at the number of content pieces. The community has succeeded with double-digit weekly growth and strong external growth. You can read more here.

SAP Hybris measured their community’s success by looking at growth of content, new users by group, response time, new Q&As, up and down votes and sentiment, and questions with best answers. The result has been a reduction in the time it takes to provide customer support. You can read more here.

Want to Learn More About Measuring Success?

Our Measuring the Success of Your Online Community webinar addresses the topic in detail with additional material on marketing a community, how to develop positive energy around your community, and what to measure for advocacy.

RELATED ARTICLES