The Diana Award
Project summary
The Diana Award (TDA) were awarded a grant from Youth Futures Foundation to deliver a new employability and mentoring programme, the TDA Mentoring Programme. Alongside this delivery funding, Youth Futures commissioned independent evaluators at IFF Research to conduct a process study of the programme. The study aimed to engage a target of 160 young people (out of a service delivery target of 240), aged 15-18, who were attending schools in Leeds and Birmingham and were at-risk of leaving education and not being in education, employment, or training (NEET).
The programme
The TDA Mentoring Programme aimed to improve employment prospects for young people by providing a range of activities relating to career skills, career insight and work experience, social action, and physical and mental wellbeing. Activities were to be delivered over a 12-week ‘core’ programme, and young people could continue onto a further nine-month programme called Future Focussed.
Evaluation aims
Through this project, Youth Futures wanted to understand more about how the programme was delivered, the outcomes that participants achieved, and the mechanisms through which these outcomes were achieved. In addition, Youth Futures wanted to understand whether the Diana Award mentoring programme would be suitable for an impact evaluation (i.e. a randomised controlled trial or quasi-experimental design).
Evaluation approach
- A mobilisation stage which aimed to understand the understand the overall purpose of the programme and how its impact could be measured. The mobilisation phase included: stakeholder interviews, the development of the programme’s Theory of Change (ToC), the development of an evaluation framework, and a scoping report.
- A data collection and analysis stage which involved a combination of primary research and secondary data analysis. This stage included: surveys of young people and mentors, analysis of performance and management information, observation of programme sessions, and qualitative interviews with TDA staff, mentors, and teachers.
Evaluation conclusions
Programme design and delivery were challenged by contextual and programme factors, including the COVID-19 pandemic, school strikes, Diana Award staff turnover, and elements of the programme’s design, such as delivering the programme to students during exam years, which limited participation because school staff did not want young people to miss class or exam preparation.
The programme was significantly redesigned in the Summer and Autumn of 2022, with further refinements made to adapt to the needs of schools and young people. Ultimately, the 12-week core programme was delivered to fewer eligible young people than intended and the nine-month Future Focussed programme was cancelled after the one participating school withdrew participation.
The programme did not operate to the scale or scope it intended, and therefore, early indication of medium-term outcomes for young people was not measured. However, the limited available qualitative evidence provides indicative evidence for the short-term outcomes of developing confidence, communication skills, and increasing awareness and passion around social responsibility.
Based on the findings from the evaluation, Youth Futures decided that the programme was not appropriate for the next phase of investment and evaluation. The challenges to delivering an RCT or QED included: defining and operationalising the participants’ eligibility criteria, clarifying the scope and scale of the programme, maintaining programme fidelity, and clarifying the routes to outcomes.