Making Sense of the World of Metrics

Measurement and metrics can be deceptively simple. We pick an aspect of our business and ask some basic questions about it, for example:

  • How many tables did each of the servers take care of each day?
  • How many sales calls did each of the sales staff make each day?
  • What is the company’s net profit each month?
  • How many people viewed our latest Google ad last week?
  • Which of our products gives the highest profit and which gives the lowest?

Answering such questions can help a manager understand a bit more about the business, however, there is a lot more to establishing metrics than simply asking and answering a few questions. It matters a great deal that we ask the right questions, that we get correct answers in a timely fashion and that we analyze the answers carefully then apply what we have learned.

Why are you measuring?

Metrics can be used for a variety of reasons. Purpose drives design, that is the design of the measure changes depending on how it will be used. Sometimes the desire will be to assess the state of the business for a one time decision that needs to be made. Other times the goal will be to establish a base and ongoing input for process improvement and management.  For example, a loan company will likely make an assessment of the business for a simple yes or no answer to the question “Is this company capable of repaying the loan?” Or an outside owner may want an answer to the question “Is management accomplishing its objectives?” However, an internal manager is likely to ask questions such as “Are we on track with our sales plan and if not, how can we get back on track?” The manager’s question is likely to be a process question, looking for diagnostics. The investor’s question is to make a one-time decision. The outside owner’s question falls somewhere in between, sometimes it would be a matter of replacing the manager and other times—one hopes—it would be aimed at helping the manager improve performance.

Ways to collect metrics

We find that the way metrics are collected can generally be categorized along two separate dimensions:  1) Who is doing the collecting; and 2) How does the collecting process relate to the normal business processes. This can be represented in a two-by-two grid, resulting in four classifications:

metrics table

Types of metrics

There are many different things that can be measured and many ways to measure them. The choice of what is measured, when it is measured and how it is measured will impact the way measures are actually used. Appendix A includes details of a variety of different types and distinctions in metrics used in business, including:

Lead and Lag Indicators

Most traditional measures are Lag Indicators; telling us what has happened in the past. For example, Profit and Loss reports, Production Efficiency, Sales and Customer Satisfaction all tell the user what happened in the period reported. In contrast, a Lead Indicator gives an indication of what is likely to happen in the future. This is done by first developing a theory of the process and then measuring an “upstream” activity to indicate what is likely to happen “downstream”.

For Lead Indicator Measures to be effective the following should be true:

  • You should have a reasonable belief that the process works—that the Lead Indicator actually has a causal relationship with the final desired outcome.
  • The activity being measure should be controllable—an action the company takes rather than a result to be recorded.

When Lead Indicators have been put in place it is then possible to set targets for the activity, monitor their performance and then track the target outcome using a Lag Indicator to confirm the relationship.

Quantities and Qualities of Process and Impacts

In the general mission community there is often discussion about whether we should assess the results of a ministry team or merely the activities or faithfulness of the workers.  This debate is reflected in the business world, where some metrics measure the process (e.g. amount of raw material purchased, number of hours worked) and some measure the results (number of cars sold). More sophisticated businesses—such as those implementing lean manufacturing—will refine those measures to assess both the quantities and quality of the processes and the quantities and qualities of the results or impacts.

Thinking about those in the context of measuring discipling relationships, for instance, we could measure the number of hours of face-to-face contact and the number of times Christ is directly mentioned (both process quantities). We could assess the amount of noise and distraction that was going on in those meetings because discussion in a quiet room has different intensity from discussion in a crowded street (a process quality measure). We could measure the number people asking to be baptized (an impact quantity measure) and we could assess the maturity of the believers in community (an impact quality measure).

Character and Behavior

The fact that it is generally easier to measure quantities rather than qualities is particularly important when it comes to ethical issues and social or spiritual transformation. When it comes to individuals, it is relatively easy to measure behaviors but it is harder to measure character. It is relatively easy to see their gifts being exercised but it is more difficult to weigh the fruit of the Spirit in their lives. Nevertheless if we are to seek to be people of integrity then we must be at least as concerned about the heart of the matter as the surface behavior. As we disciple individuals, we will look for ways to gently help them reflect on their character issues as well as their behaviors. In the same way it is relatively easy to measure the behavior and activity of a business, but it is more difficult to assess the health and depth of its culture and corporate character.  

If individuals are to help in Kingdom transformation, then they must not only do and say the right things, they must have the right character too. We need to find ways to help them address both their behavior and their character. Similarly, businesses that are concerned about being transformational must address their behavior and culture.

There is difficulty in measuring character because it is often not very tangible or well defined. This means that direct measurement is difficult, but we can look for signs of maturity and character. Those signs are only surrogate measures (see below) but they are often helpful in facilitating reflection and growth.

Discrete and Continuous Quantities

Another fundamental distinction in metrics is the difference between measuring things that can be counted (“discrete quantities”, like books) and measuring things that only have quantity (“contiguous quantities”, like water). It is relatively easy to count pennies and books but to measure water (let alone water flow) you need specialist tools. The fact that measuring discrete quantities is easier means that many businesses and missionaries concentrate on measuring things that can be counted. We can convince ourselves that it is easy to count “converts” or “churches planted” because we can count heads and meetings. It is easier to ask “How many churches are there in the country?” than “How much church is there in the country?”

Generally we do not like ambiguity “Is that one church or two?” “Is that person a believer yet or not?”. However, our reaction to ambiguity can lead us to value counting and definitions over valuing what is really at the heart of the matter.  As with all metrics, but especially when it comes to counting, we must tread with holy reverence, listening to God’s insight and interpretation as we go along.  

Direct and Surrogate Measures

Not everything lends itself to a direct measure. It is easy to measure how often people show up at a church meeting, but it is impossible for any of us to know the real condition of someone’s heart. It is easy to measure sales, harder to know how happy customers are with the product or service. Often what we want to know is impossible to measure directly and we are faced with the choice of giving up on measurement or measuring something we think might correlate with the factor we are really concerned about. When we choose the latter and measure a correlate, we are using a Surrogate Measure.

Surrogate Measures are an essential bridge for us in measuring and evaluating some of our more important goals. We want to see lives transformed in the image of Jesus. We want to provide products and services that address real needs for people. We want to do it in a way that brings glory to God. But how do we know when we have successfully done it? This is where Surrogate Measures can help.

A surrogate is a substitute, something that takes the place of the original. When choosing a Surrogate Measure we need to find something that can be measured which we believe will correlate with the real goal of our measure. Particular care must be taken with surrogate measures when working cross culturally.  Choosing to use a surrogate measure means that you think you can see a connection between the thing being measured and the thing you actually want to measure. Such assumed connections can be based on cross-cultural misunderstandings.

Excerpt from the BAM Think Tank Measuring BAM Impact report. We recommend reading this report in full and we post some additional extracts from it in the coming weeks.