Navigating Complexity: Lessons From a Portfolio Evaluation of Impaired Driving Rehabilitation Programmes
A portfolio evaluation approach
Impaired driving – driving a vehicle while under the influence of alcohol or other drugs – is a major issue in Aotearoa, implicated in nearly half of the nation's fatal motor vehicle accidents each year. Strategies like the Road to Zero and investment in road policing aim to deter the harmful behaviour through greater roadside testing, speed management and stringent sentencing. But for recidivists – those who repeatedly offend – rehabilitation programmes have been identified as a key avenue for long term change.
In 2023/24, Dovetail was commissioned by Health NZ | Te Whatu Ora to conduct a portfolio evaluation of nine government funded Impaired Driving Rehabilitation (IDR) programmes across the country. They aim to reduce reoffending, harmful substance use, and shift the attitudes and behaviours that lead people to drive while impaired. The evaluation aimed to:
Understand how each programme is designed and delivered
Explore the evolution of the service models over time
Assess the impact of both mainstream and kaupapa Māori approaches
Identify strengths, challenges, and promising best practice for future implementation across all approaches.
But undertaking a national-scale evaluation across nine diverse, locally grounded services brings a lot of complexity. In this post we reflect on some of the challenges we faced, many of which are in portfolio evaluations, and we outline how we addressed them.
Capturing the diversity of programmes
Each IDR programme varied in format and length drawing on different delivery models and practices from CBT-based interventions to group based wānanga. To speak to the uniqueness of each programme and its delivery model, we designed flexible evaluation criteria that were sensitive to different local contexts and avoided oversimplifying or misrepresenting the intent and impact of each unique programme.
Embedding cultural responsiveness
Many programmes were grounded in kaupapa Māori and mātauranga Māori frameworks where rehabilitation is not just about behaviour change and a reduction in recidivism but about restoring mana, reconnecting with whakapapa, and strengthening cultural identity. Our evaluation methods reflected this, recognising culturally relevant outcomes such as whānau healing, community belonging, and reconnection with wairua; outcomes that sit outside traditional Western measures but are no less indicative of success.
Navigating data inconsistencies
A recurring challenge in service-level evaluations is the inconsistent availability and format of data across providers. IDR programmes often used different reporting tools, outcome measures, or data collection systems to varying degrees of completion. We addressed this by providing qualitative insights drawing on interviews, hui, and participant stories, and developing survey tools that could be adopted across all settings. We triangulated across multiple sources and were transparent about data limitations throughout our reporting.
Frame evaluation as learning, not judgment
Portfolio evaluations risk being seen as accountability exercises. From the outset we worked to build trust with providers meeting kanohi ki te kanohi and were transparent about our process. We clearly communicated that we were not comparing programmes against each other but rather surfacing service level insights. This created space for honest reflection and allowed providers to share both successes and challenges openly.
Highlighting value amidst data diversity
Even with the data limitations, we were able to show that many of the interventions were consistent with a chain of value creation that was consistent with the aims of the programme overall. Although there wasn’t enough data to undertake a reliable return-on-investment analysis, we were able to demonstrate that, against pre-agreed criteria, the programmes as a whole delivered equitable and efficient resource management, effective programme operations, and good social value for families, communities, and taxpayers. We also identified the data requirements that future monitoring would need to assess these areas more consistently and comprehensively.
A challenge worth embracing
Portfolio evaluations like the IDR can be a powerful tool for learning and developing responsive solutions. They offer the chance to step back and ask not just “Is this working?” but “What does working really look like in different places, for different people, in different ways?”. They must be done with care and integrity to ensure local and culturally grounded approaches are honoured and explored for the value they offer, allowing us to surface insights and good practice for wider implementation.
To read the full IDR Programme Evaluation Report here.
Our thanks to all who participated in this evaluation. Dovetail was commissioned by Health NZ | Te Whatu Ora.