Comment: Think before you measure!

This blog originally appeared on the LSE's Measuring Business & Human Rights site as part of a series.

I will use this blog to reflect on some of our experience, rather than theory, of working with large companies to find ways of measuring corporate practice on human rights. I don’t seek to claim that any of these approaches as role models but they may stimulate some ideas, and I reflect on my learning at the end.

The first example is a mining company. Over several years we helped them to set up their approach to responsible sourcing. Rather than outsource its responsibilities to a third party, the company sought to build the capacity of its procurement teams – organized by category – to assess their supply chains and build an appropriate response tailored to the level of risk. We developed measures for the category teams designed to encourage behaviours to support program implementation and these evolved as the program progressed over 3-4 years:

  • In year 1, the category teams were measured on the percentage of suppliers to which they had communicated the Company’s Supplier Code;
  • In year 2, the category teams were measured on the number of high-risk suppliers where an audit had taken place, and the percentage of those where an improvement plan had been agreed;
  • In year 3, the completion of improvements plans was assessed and we moved towards beginning to collect supplier data to assess impact, e.g. in the area of health and safety such as lost time injury rates.

The measures worked well in encouraging the category team to progress their practice, year by year – with a little bit of competition appearing between them.

The second and third examples relate to work within the agricultural sector.

A recent agricultural project uses an audit-derived approach, featuring observations across multiple standards-based indicators. It is generating masses of data across multiple dimensions, but our review is asking some fundamental questions:

  • Process versus impact indicators: are we really interested in how many farmers are trained, or are we more interested in the impact generated?
  • Breadth versus quality: when resources are constrained, is it better to try to collect data across many dimensions, or to collect fewer indicators well?
  • Meaningfulness of data: is the approach to collecting the data reasonable and likely to give reliable data, e.g. can you assess whether workers are free of harassment without off-site checks?

For the time-being, what we have done is to focus on those indicators of impact which the current approach can collect meaningfully and, with some investment in training, consistently. Reliable data will then gives us comparability and encourage continuous improvement. In time, we will introduce other approaches to assess whether changes in practices have also also brought about changes in harder to measure areas, such as worker participation.

And finally, one of the most interesting assignments in recent years was to consider how you might measure a coffee sustainability program called ‘Coffee Made Happy’. This led us into an extensive review of the recent literature on Happiness indicators, including the widely publicised example of Bhutan’s Gross National Happiness Index. The basic building blocks of the approach to assessing the happiness of coffee farmers included human rights such as health, education and economic sufficiency.

We supplemented these quantitative indicators with qualitative indicators such as a farmer’s confidence in his/her future and that of their children. Such measures present a particular challenge in data collection, but they do testify to core goals of human rights such as freedom from fear, as well as freedom from want and talk to the lived experience of the farmer.

Reflecting on this experience leads me to draw out these thoughts which really speak to the importance of putting some careful thought and effort into the design of your measures:

  • We’ve taken considerable care to think about what behaviours we want to see, and to design measures which encourage these whether they relate to program implementation or impact;
  • We are getting accustomed to dealing with both quantitative and qualitative measures, despite the challenges of measuring the latter;
  • In practice, we are finding it helpful to begin modestly and relatively simply and develop the complexity over time;
  • We are being careful to challenge whether the data being collected is meaningful, and can be reliably used in decision making;
  • Particularly at the bottom of supply chains, there is a challenge around quality and consistency of data collection, so we are encouraging collecting less with greater accuracy;
  • Even with the availability of high-tech solutions and data processing capabilities, if the data is to be inputted by people there is a significant investment required to ensure it is being collected consistently.