Jeremy Spafford, director of Arts at the Old Fire Station, tells us about a new report, and a different way of measuring impact.

Evaluation can be such a pain.

Like many organisations, we shout a lot about how important our work is and we persuade people to give us money to support this work. We do want to learn about what works well and why and we do understand that we have to be accountable for what we do.

BUT asking people to fill in forms about how confident they feel and trying to count how many differently labelled people have done different things with us is a miserable task for everyone involved and, to be honest, I never really believe the numbers.  It’s not that we make them up, it’s just that there are so many variables and inconsistencies in how we collect the data that I just don’t know if it means anything.

And when you set yourself up as an organisation that enables people to re-write their labels and move away from being a set of problems to being a co-creating member of the team, you undermine the whole process as soon as you produce the form because suddenly I’m the professional and you’re the beneficiary again.

When we write funding applications, we describe the outcomes we’re aiming for. Fair enough – why should anyone give us money if we can’t describe what impact we want it to have? Our evaluation process is then understandably focussed on measuring how well we’ve achieved those outcomes. But this means we might miss what actually happened because we’re only looking for what we wanted to happen and desperately trying to prove that it did to justify the funding.

We spent the best part of two years trying to develop a database that would count and track engagement and before/after surveys that would measure impact based on predetermined outcomes. It was a miserable process involving complex negotiations with the database designers and endless conversations with people trying to get them to go through a process they didn’t enjoy and didn’t believe in.

So we stopped and pressed re-boot.

Through storytelling (based on Most Significant Change) we think we’ve found a methodology which

• Enables people to set their own outcomes (describing what change happened and how)
• Avoids counting things that are not directly relevant
• Is fully owned by those involved at all stages (including analysis of findings)
• Can be done by and with anyone – regardless of their relationships to us
• Provides scope for creative ways of presenting findings
• Is an engaging project in itself and, crucially, is enjoyable and fun to do.

And what did we learn? For me, there were two big take-aways that I wasn’t expecting.

Firstly, regardless of the reason story tellers engage with us (artists, people who are homeless, volunteers, staff), relationships seem to be central. Feeling part of something and being amongst people who are friendly and interested counts for a lot.

And secondly, making a contribution at the Old Fire Station changes how people feel about Oxford and their ability to contribute elsewhere. Is this place-making?

Do read the report by Anne Pirie and Liz Firth – it’s not long and it’s well written. And do let us know how you evaluate. It would be good to know what you do.