The thorny problem of metrics in long-term, holistic family support work

“Not everything that counts can be counted, and not everything that can be counted counts”.

This quote is often attributed to Albert Einstein, but apparently, this is likely to be incorrect (if you’re interested in who really said it, it’s complicated). But the fact that this misattribution is so widely memed is an indication of the quote’s resonance, and the sense that it must be a wisdom from someone really smart who really knows about such things.

One of our most enduring conversations at the Birmingham Pathfinder is about metrics, and the tension between the undeniable importance of measurement – to track progress, evidence efficacy, and provide accountability – and the fact that some of the most important outcomes are also the hardest to measure. This is particularly true for holistic and long-term outcomes.

Our experience of establishing an alternative model of family support has taught us much about how the existing systems of metrics – such as KPIs – can often have unintended consequences. They can become drivers of practice, instead of measures of impact, for example. They can skew practice towards actions that generate those things that are easiest to measure, incentivising these things even at the expense of actions that would ultimately lead to better overall results for families. The ultimate example of this is where case closures are used as a measure of success of a programme, generating perverse incentives for the person or agency providing support to close cases too soon.

As a project which offers support to our families ‘on tap’, and doesn’t close a case until a family is ready to move on, we face several issues with metrics. One central one is that over the course of long-term support, things can sometimes move backwards: life happens to our families and we remain alongside them for the ups as well as the downs. This means that our families trajectories are rarely linear, even with us walking alongside them, and this can be taken as an indicator that our support isn’t ‘working’, especially when the metrics are taking a ‘snapshot’ that focusses on too short a time period. Even when things are actually progressing well, people tend to disclose deeper and more difficult issues as more trust is built – this means that the family’s situation can appear to be getting worse as a result of our support, when in fact we are just learning more about what’s really going on under the surface.

Another is that in the reality of a family’s life, overall wellbeing may require that one ‘good’ is traded off against another. An interesting example is school attendance. When  a child’s mental health is being adversely affected by being at school, school attendance may not necessarily be a good thing for that child at that time. In this circumstance, continued or even increased attendance could indicate that there simply isn’t the capacity to support that child’s mental health at home. Sometimes school is a safer, or less neglectful place than home, and in this case a child attending school even when they are sick may not be an indicator that all is well. Again, it may actually indicate a problem or lack of capacity  to care for them at home. And yet increased school attendance is often used uncritically as a proxy for family wellbeing. Metrics like school attendance that may be useful to indicate trends on aggregate, may be unhelpful when applied to individual families.

Our concerns also go beyond these relatively practical concerns. We have also found ourselves questioning the very assumptions embedded in the pro-metrics side of the tension that I mentioned in the first paragraph. Yes, a family support approach needs to be accountable, for example, but to whom? At the moment ‘accountability’ is generally taken to mean to the funder/commissioner, not to families or to the long term change we seek. Efficiency and efficacy are other examples: on whose terms? It’s an issue fraught with power dynamics.

When you are trying to prove the worth of an alternative approach which doesn’t fit well into the existing system of measurement, it can feel like fighting a battle on two fronts – you must argue not only for your unconventional practices (such as not closing cases), but also argue against metrics which are often uncritically conflated with the thing they are intended to measure.

This is why it was amazing that in 2016 we were fortunate enough meet the Lankelly Chase Foundation, whose approach to funding was guided by their understanding of change in complex systems. They not only funded us and secured our survival at a really crucial time, but they were also able to engage with in thinking about complex and intractable problems like the problem with metrics. More recently, in 2023, Lankelly Chase announced their decision to redistribute their assets and close as a grant-making body. As a part of their subsequent transition pathway, they have produced a legacy website which documents their journey and the work of the many different organisations that they helped support over the years. We are really proud to be included in this number, with this interview that the legacy team undertook with our founder, Rob McCabe, where he discusses some of these issues with Tchiyiwe Chihana.

Read the article here.