Pilot Evaluation Report: Evaluation of the Liverpool Talent Match

Summary 

Merseyside Youth Association (MYA) entered our ‘Development’ stream as a project supporting young people who are further from the labour market into positive EET outcomes.​ 

Liverpool Talent Match, delivered by MYA, works with highly marginalised young people experiencing complex needs such as poor mental health, lack of self-confidence, disabilities, food or housing insecurity, substance misuse, or experience of the criminal justice system. This group of young people experience significant labour market disadvantage and low (EET) outcomes. 

Study aims: 

• Identify how the support model operates and the programme’s Theory of Change 

• Support MYA to understand which elements of delivery work most effectively and why​ 

• Support improvements to the programme and data collection processes, to build organisational capacity and assess the feasibility of further evaluation.​ 

• Assess the costs and benefits of the programme 

Evaluation approach 

The evaluation included a process study to develop and test the Theory of Change, understand participant outcomes and identify any refinements which could be made to improve delivery.  

The mixed method evaluation drew on evidence from: 

• monitoring and outcomes data 

• a repeated survey for participants 

• a cost analysis survey for programme mentors  

• qualitative interviews with 16 young people (5 at multiple timepoints), 8 frontline staff, 3 strategic staff, 2 employers 

• multi-media digital diaries from 8 participants 

• observations and staff workshops 

The study also aimed to understand the feasibility of conducting an impact evaluation of the Liverpool Talent Match programme. 

Key insights 

The study found that there were associations between higher levels of engagement with the programme and a higher likelihood of transition into EET outcomes, compared with a comparator group drawn from Understanding Society. The qualitative work suggest that Liverpool Talent Match helps participants to achieve a range of positive outcomes such as improved confidence and self-esteem, social and interpersonal skills, and increased knowledge of the job market and job searching skills.  

The study highlights that a main driver of these outcomes is the mentor-mentee relationship, which is a key element of the intervention.  

Aside from these outcome findings, the study found that the programme is delivered as intended and as indicated by the Theory of Change. Although the participant journeys could vary considerably from person to person, the phases of the intervention align with the Participant Journey map.  

The Cost-Benefit Analysis study found that there are positive net benefits association with Liverpool Talent Match. The cost-benefit ratio was 2.2, meaning that the total benefit associated with the programme was 2.2 times greater than the total cost.  

Next steps

We commissioned a feasibility study alongside this work to investigate the potential for an experimental or quasi-experimental design. This study found that there could be further evaluative work around Liverpool Talent Match programme; the intervention is well-defined, and MYA recruits ample numbers of participants for further evaluation to be possible.  

There were some barriers to carrying out such a study in this case, however, and after significant exploration and discussion, we were not able to overcome the challenges. An experimental design (such as an RCT) was found to be unsuitable for the delivery partner. Quasi-experimental designs were explored at length but it was ultimately found that the administrative data set to facilitate this would not be accessible within a feasible timescale.