This is an old revision of the document!


Metrics

It could be very difficult to acess the contributions of the community members by metrics alone. However, metrics when combined with pragmatic analysis techniques can offer insights that can be used to give us a better understanding of the dynamics of a community. The purpose of this section is to help you make an informed choice on the metrics to be measured, that will help your community.

Overview

"Not everything that can be counted, counts and not everything that counts can be counted".

Albert Einstein

Metrics have always played a key role in businesses and processes. But, in communities like software development, it is hard to quantify important aspects that contribute immensely. For ex.,

  1. Long technical discussions on IRC/Email
  2. Code review comments
  3. Wiki and Documentation
  4. Bug reports and stack trace attachments

The goal of metrics such communities is to provide useful pointers about

  • where the community is headed
  • help identify potential bottlenecks

@jgbarah has a tremendous post on how to derive the most meaningful metrics. Please refer to it below. http://www.communityleadershipforum.com/t/7b-analytics-metrics-for-community-based-software-development/268

In this post we will identify the challenges, pitfalls to avoid and guidelines to arrive at the metrics for a community.

Preliminary Steps
  1. Even before beginning to collect metrics one should consider the target audience for the metrics.
    • Reports targeted at the engineering department will not fit the needs of marketing department.
  2. Metrics are numbers that can be manipulated by gaming the system.
  3. The rule of participation: 90% of people are lurkers, 9% contributors, 1% creators.
Sources of Metrics
  • Tracking web page views.
  • Forum events.
    • Events provides clarity of purpose and sets the context for the communinty.
  • Community users.
    • Identifying members of the community with unique ID helps to trace actions.
  • User feedback
    • Detecting lack of features, by tracking user actions and receiving user feedback.
  • Deplying programs on the community server.
    • Feedback to customer support portal.
Condensed list of Metrics
  • Q & A from forum or mailing list.
  • How many returns versus new logins.
  • Average time between when a bug is submitted and it is fixed.
  • Average time for a contribution to be entered into the code base.
  • Why is code review taking so long?
  • People who left (for example, to make exit interview).
  • How long it takes to set up the development environment, to start contributing.
  • Number of commits
  • Number of projects each person contributes to
  • Bugs and Tickets, with categorization
    • Response time for resolution
    • Bug count
    • Regressions
    • Other code quality issues
  • How often code extracts are making it into other projects (Example : GitHub)
  • Core project versus side project.
    • Who is helping with the existing code?
    • Who is contributing a new feature?
    • Why weight the different groups?
    • How to assign weights for contributions?
  • Consider two very different cases of contributions
    • individual, independent volunteer contributions
    • developers hired by a company
    • This can be a large difference with respect to how the project is performing.

What actions should the community manager take based on the above?

Points of Interest to Community Managers

* Tracking the actions online, and offline, and matching both.

  • How is the community's online visibility and presence influencing its accessibility.
  • How are Meetup groups helping your community.
  • How many people joined the community online because of an in-person event.
  • How many people got to know a contact point from the community via online activity (StackOverflow, reddit etc.).

This gives the community managers a sense of the diversity and says a lot about the appeal of the community. It also helps understand which online channels are working in your favour and where you can improve.

Metrics and Community Managers

Community Managers protect the project from management. Their asking for data is usually just to prove their stance. “why collect information of this kind?” Know what is going on helps them make smarter decisions. I have an outcome in mind, I can prove it using metrics. Usually management or the PR department want this. For some open source projects (ie Apache), in some cases they try not to consider too much the company hiring each developer, consider they speak on their own behalf. There is a balance between companies riding the project and companies not being important for it. By-laws of the project should be careful here.

Focusing on which companies are contributing the most, that can cause land-grabbing

Case found in the Drupal Community. What was causing some spikes in contributions? The increase in participation happens about three months after their community event. Now that you know that, why does it take so long? Ramp up time? Training? The metric had to be looked at a few months after the sprint, then you need continuous tracking.

Final thoughts

For open source, numbers (and transparency in numbers) should be important. Metrics can inadvertently polarize communities with questions like,

  • Developers are important?
  • Communities are important?

So, use them judiciously and transparently.

Pitfalls To Avoid

Provide some guidance of common pitfalls and problems to avoid.

Tips and Tricks

Provide some guidance of tips and tricks to bear in mind when approaching this topic.

Further Reading

Provide a bullet list of addition pages to look at our additional resources.

* Something to look at. * Something to look at.