The challenge of evaluating system change

Recently I found myself in a lot of conversations about system change which got me thinking. How do we know we are making progress when working on something this complex?

Locally I was chairing a meeting about the new Northumberland Food Strategy. One of the attendees quizzed the presenter on what metrics the council would use to know if the strategy was working. What would be the KPIs? What metrics?

If this was a project these would be the right questions. But how do you measure change in a complex system? The way our local food system works is a complex mixture of national and local economic, environmental and business policy, of geography, impacted by the behaviour of the residents, tourists, farmers, supermarkets, public sector institutions, big companies and small local businesses.

Earlier the same day I heard about a system change in health that at the end of 3 years had benefited just 6 patients. Should it have been abandoned as not working? Yet by the end of 8 years, it had reached 50,000 patients. We’re trying to measure system change with tools designed for simple projects and it’s distorting what success looks like.

Where does this leave charities who know the issues are complex but have to seek funding in 3 year cycles? More and more funders require a theory of change as part of the bid, setting out the assumptions ‘if we do x then y will happen’. Is a linear theory of change useful in the context of system change where cause and effect are not linear? How can it be adapted to make it more relevant to the complex world we work in?

I’m going to be doing some thinking about better ways to measure progress at a system level. Look out for more from me on this and get in touch if you are interested in a conversation about this.