Metrics

It could be very difficult to acess the contributions of the community members by metrics alone. However, metrics when combined with pragmatic analysis techniques can offer insights that can be used to give us a better understanding of the dynamics of a community. The purpose of this section is to help you make an informed choice on the metrics to be measured, that will help your community.

Overview

“Not everything that can be counted, counts and not everything that counts can be counted”. Albert Einstein


Metrics have always played a key role in businesses and processes. But, in communities like software development, it is hard to quantify important aspects that contribute immensely. For ex.,

  1. Long technical discussions on IRC/Email
  2. Code review comments
  3. Wiki and Documentation
  4. Bug reports and stack trace attachments

The goal of metrics such communities is to provide useful pointers about

  • where the community is headed
  • help identify potential bottlenecks

In this post we will identify the challenges, pitfalls to avoid and guidelines to arrive at the metrics for a community.

Metrics & Community: Recommended Approaches

Preliminary Steps
  1. Even before beginning to collect metrics one should consider the target audience for the metrics.
    • Reports targeted at the engineering department will not fit the needs of marketing department.
  2. Metrics are numbers that can be manipulated by gaming the system.
  3. The rule of participation: 90% of people are lurkers, 9% contributors, 1% creators.
Sources of Metrics
  • Tracking web page views.
  • Forum events.
    • Events provides clarity of purpose and sets the context for the communinty.
  • Community users.
    • Identifying members of the community with unique ID helps to trace actions.
  • User feedback
    • Detecting lack of features, by tracking user actions and receiving user feedback.
  • Deploying programs on the community server.
    • Send the user feedback to customer support portal.
Condensed list of Metrics
  • Q & A from forum or mailing list.
  • Community Portal - How many returns versus new logins.
  • Average time between when a bug is submitted and it is fixed.
  • Average time for a contribution to be entered into the code base.
  • Why is code review taking so long?
  • People who left (for example, to make exit interview).
  • How long it takes to set up the development environment, to start contributing.
  • Number of commits
  • Number of projects each person contributes to
  • Bugs and Tickets, with categorization
    • Response time for resolution
    • Bug count
    • Regressions
    • Other code quality issues
  • How often code extracts are making it into other projects (Example : GitHub)
  • Core project versus side project.
    • Who is helping with the existing code?
    • Who is contributing a new feature?
    • Why weight the different groups?
    • How to assign weights for contributions?
  • Consider two very different cases of contributions
    • individual, independent volunteer contributions
    • developers hired by a company

What actions should the community manager take based on the above? By-laws of the project should be careful about cases where companies are trying to influence the project versus cases where companies adopt a passive stance.

Pitfalls To Avoid

Why Collect Metrics
  • Protect the project from management.
  • Know what is going on helps them make smarter decisions.
  • Management or PR department need numbers to prove estimated outcomes.
    • The business perspective of community.
  • Focusing on which companies are contributing the most, that can cause land-grabbing

For open source, numbers (and transparency in numbers) should be important. Metrics can inadvertently polarize communities with questions like,

  • Developers are important?
  • Communities are important?

So, use them judiciously and transparently.

Points of Interest

  • Tracking the actions online, and offline, and matching both.
    • How is the community's online visibility and presence influencing its accessibility.
    • How are Meetup groups helping your community.
    • How many people joined the community online because of an in-person event.
    • How many people got to know a contact point from the community via online activity (StackOverflow, reddit etc.).
  • What causes spikes/dips in contributions?
    • why does it take long time for people to contribute?
    • Is it ramp up time?
    • Lack of Training?

This gives the community managers a sensing report. It also helps understand which online channels are benefitting the community and what can be improved.

Tools

Further Reading