This is an old revision of the document!


Metrics

It could be very difficult to acess the contributions of the community members by metrics alone. However, metrics when combined with pragmatic analysis techniques can offer insights that can be used to give us a better understanding of the dynamics of a community. The purpose of this section is to help you make an informed choice on the metrics to be measured, that will help your community.

Overview

"Not everything that can be counted, counts and not everything that counts can be counted".

Albert Einstein

Metrics have always played a key role in businesses and processes. But, in communities like software development, it is hard to quantify important aspects that contribute immensely. For ex.,

  1. Long technical discussions on IRC/Email
  2. Code review comments
  3. Wiki and Documentation
  4. Bug reports and stack trace attachments

The goal of metrics such communities is to provide useful pointers about

  • where the community is headed
  • help identify potential bottlenecks

@jgbarah has a tremendous post on how to derive the most meaningful metrics. Please refer to it below. http://www.communityleadershipforum.com/t/7b-analytics-metrics-for-community-based-software-development/268

In this post we will identify the challenges, pitfalls to avoid and guidelines to arrive at the metrics for a community.

Preliminary Steps
  1. Even before we start collecting metrics we should consider the target audience for the metrics. Reports targeted at the engineering department will not fit the needs of marketing department.
  2. We need to bear in mind that metrics are numbers that can be manipulated by gaming the system.
  3. Remember the rule of participation: 90% of people are lurkers, 9% contributors, 1% creators.
Sources of Metrics
  • Tracking web page views.
  • Forum events.
    • Events provides clarity of purpose and sets the context for the communinty.
  • Community users.
    • Identifying members of the community with unique ID helps to trace actions.
  • User feedback
    • Detecting lack of features, by tracking user actions and receiving user feedback.
  • Deplying programs on the community server.
    • Feedback to customer support portal.
Condensed list of Metrics
  • Q & A from forum or mailing list.
  • How many returns versus new logins.
  • Average time between when a bug is submitted and it is fixed.
  • Average time for a contribution to be entered into the code base.
  • Why is code review taking so long?
  • People who left (for example, to make exit interview).
  • How long it takes to set up the development environment, to start contributing.
  • Number of commits
  • Number of projects each person contributes to
  • Bugs and Tickets, with categorization
    • Response time for resolution
    • Bug count
    • Regressions
    • Other code quality issues

Points of Interest to Community Managers

* Tracking the actions online, and offline, and matching both.

  • How is the community's online visibility and presence influencing its accessibility.
  • How are Meetup groups helping your community.
  • How many people joined the community online because of an in-person event.
  • How many people got to know a contact point from the community via online activity (StackOverflow, reddit etc.).

This gives the community managers a sense of the diversity of their community and says a lot about the appeal of the community.

Meetup groups: for how many people the first point of contact was an in-person event, and now they are joining the online group? Or just knowing the first point of contact, if it was asking and maybe answering questions on StackOverflow

Bugs? Regressions? Other code quality issues?

How often code extracts are making it into other projects (would love to get this). GitHub can do this

Core project versus side project. Split again - who is helping with the existing code, or who is contributing a new feature. Why weight the different groups?

What actions to take based on the above.

  Community Managers protect the project from management. Their asking for data is usually just to prove their stance
  There are two very different cases: individual, independent volunteer developers commit the code, or developers hired by a company are the ones making contributions, This can be a large difference with respect to how the project is performing.

“why collect information of this kind?”

  Know what is going on to make smarter decisions. Usually community managers are interested in this
  I have an outcome in mind, I can prove it using metrics. Usually management or the PR department want this

For some open source projects (ie Apache), in some cases they try not to consider too much the company hiring each developer, consider they speak on their own behalf. There is a balance between companies riding the project and companies not being important for it. By-laws of the project should be careful here.

Focusing on which companies are contributing the most, that can cause land-grabbing

Case found in the Drupal Community. What was causing some spikes in contributions?

  The increase in participation happens about three months after their community event. Now that you know that, why does it take so long? Ramp up time?  Training?
  The metric had to be looked at a few months after the sprint, then you need continuous tracking.

Some final thoughts:

  For open source, numbers (and transparency in numbers) should be important.
  Developers are important; or communities are important?

Approach 2

One approach. Be detailed in how to apply this method of approaching the topic - where possible, provide practical steps.

Pitfalls To Avoid

Provide some guidance of common pitfalls and problems to avoid.

Tips and Tricks

Provide some guidance of tips and tricks to bear in mind when approaching this topic.

Further Reading

Provide a bullet list of addition pages to look at our additional resources.

* Something to look at. * Something to look at.