UpRising
Project summary
Why did Youth Futures Foundation fund this project
Youth Futures Foundation wanted to understand how the programme was delivered, the outcomes that participants achieved, and the mechanisms through which these outcomes were achieved. Alongside this, Youth Futures also wanted to understand whether Stand Out would be suitable for an impact evaluation (i.e. randomised controlled trial or quasi-experimental design). IFF Research were awarded an evaluation grant to answer these questions.
Summary of evaluation approach / programme activities
The evaluation was comprised of three strands:
- A scoping stage aimed to develop a programme Theory of Change.
- A programme evaluation aimed to test the Theory of Change, understand the outcomes the programme achieves and to identify any refinements which could be made to improve delivery.
- A feasibility study aimed to understand whether the programme was suitable for testing through an impact evaluation.
The evaluation drew on evidence from:
- Performance and management information.
- Online surveys administered with young people before and after starting the core programme; and at three- and six-month follow-up points.
- 30 interviews with people involved in programme recruitment and delivery.
- 60 interviews with young people who participated in the programme.
- Peer-to-peer research.
Evaluation conclusions
Stand Out was delivered to 444 young people during the funding period. The programme was largely delivered as intended, with some key exceptions – the most of notable of which was the move to a fully online model due to the COVID-19 pandemic.
It had been anticipated that participants would engage with most programme activities. However, young people reported ‘dipping in and out’ of activities, and attendance at some live activities was low due to some young people engaging via video catch-up.
Survey attrition limited the analysis that was possible in relation to programme outcomes. However, there was evidence to suggest significant, and sustained, improvement in tacit skills, wellbeing, professional networks, and employability skills.
Based on the findings from the feasibility study, Youth Futures Foundation decided that the Stand Out programme was not yet ready to proceed to an impact evaluation. This was because:
- The online delivery model and flexible nature of young people’s engagement made it challenging to identify the ‘core’ elements of the programme that would be useful to test through an impact evaluation (i.e. the elements that most young people engaged with, and were leading to positive outcomes).
- The evaluation concluded that experimental approaches would be challenging to conduct within Youth Futures’ budgetary restrictions. The sample sizes that would be needed for a randomised controlled trial or quasi-experimental design would be very large, and identifying a suitable counterfactual for a quasi-experimental design would also be a challenge.
Recommendations
The evaluation provided several recommendations for programme refinement which may also be applicable for practitioners delivering programmes like Stand Out. It is worth noting that these recommendations should be treated with caution given the methodological limitations of this study.
Delivering online
- For online programmes, consolidating activities onto fewer platforms, and ideally platforms that most participants are already familiar with, may help to reduce confusion, and help young people to feel less overwhelmed.
- When using multiple communication channels, ensure that important information, such as timetabling, is repeated across all platforms.
Delivering appealing content and support
- Tangible skills-based activities (such as getting personalised CV advice) may be more popular than more general types of support (such as daily check-ins).
- Mentoring arrangements that facilitate providing ad-hoc support may better reflect young people’s needs during their job search.
Providing flexible, tailored support
- Flexibility in support delivery is important for engaging diverse young people, and requiring young people to attend all sessions may not be effective.
- Consider how programme content can be tailored to suit young people on different career paths and at different stages of their career journey.
- To support attendance and relationship building, consider running programmes outside term time, keeping sessions to a maximum of an hour, and providing more than a one-time slot for each session.
Building relationships
- To support peer networking, group young people with other participants who are close in age/life stage or have similar sector interests.
- Mentor matching is important to get right. Both young people and mentors would benefit from knowing why they were matched together and being provided with the opportunity to change matches after an initial meet.