About the Youth Employment Toolkit

The evidence reviews

The Youth Futures Foundation commissioned a team of researchers from the Centre for Evidence and Implementation, the Institute for Employment Studies, and Monash University to conduct Rapid Evidence Assessments (REAS) on seven key interventions. We identified these interventions through consultation within Youth Futures and our stakeholder networks.  

The researchers considered over 700 articles and reports—starting with those in the Youth Employment Evidence and Gap Map (EGM)—to see if they were suitable for inclusion in the REAs. In the end, the REAs drew on material from around 70 different evaluations that met the criteria for quality and relevance. The REAs are published alongside the Toolkit.    

Most youth employment programmes consist of more than one kind of intervention. This means that young people can potentially benefit from different types of support. However, it raises challenges for evaluation because it can be difficult to work out which components within a multi-component programme have contributed to observed impacts. Previous meta-analyses of data from evaluations of youth employment interventions have often categorised these interventions based on the components that are most prominent rather than seeking to separate out individual components.  

The research that underpins the Youth Employment Toolkit includes a ‘Network Meta-Analysis’ (NMA) of evaluation findings. NMA is a method that allows researchers to synthesise different kinds of quantitative data from evaluations in order to compare multiple different kinds of intervention, and tease out the impacts of specific components. A blog written by the team who conducted the NMA explains the method in more detail.  

Researchers at Youth Futures conducted additional analyses of information on implementation and processes from the studies in the NMA, to identify how to implement the intervention well. This was supplemented with some additional literature from the EGM and other high-quality sources.  

The information in the Youth Employment Toolkit

The evidence presented in the Toolkit is organised into sections that correspond to different types of intervention that can be used to support youth employment. This first edition of the Toolkit presents evidence on: 

• Wage subsidy programmes
• Apprenticeships 
• ‘On the job’ training (programmes that last between six weeks and one year) 
• ‘Off the job’ training (where offered as a targeted youth employment intervention) 
• Basic skills development (numeracy, literacy and/or digital skills) 
• ‘Life skills’ development (primarily psychosocial skills) 
• Mentoring or coaching 

Each ‘intervention’ page provides information about: 

• How the intervention is defined in the Toolkit, and some examples of its use. 
• The likely impact of the intervention on youth employment (whether young people who take part are more likely to get a job afterwards), along with some context for this finding.  
• The strength of the evidence on which this estimate of impact is based.  
• How the intervention is likely to operate to improve youth employment.  
• How to implement the intervention well.  
• The number and types of input that are typically needed to implement the intervention.  
• Additional links and resources with further information about the intervention.  

You can find detailed information about how to use the Youth Employment Toolkit here. 

For each intervention, the Youth Employment Toolkit provides three ratings: 

1. A rating of the likely impact of the intervention on youth employment  
2. A rating of the strength of the evidence for this impact 
3. An indication of the number, type and likely costs of the inputs needed to implement the intervention.

Below is some additional information about these ratings. You can find a full description of how they are generated in the technical guide to the Toolkit.  

The impact rating

The impact rating refers to the average  impact of each intervention where it is implemented within a youth employment programme. This ‘programme’ could include multiple different kinds of intervention, or it could include just one component, the intervention itself. The impact of each rating shows the difference that the intervention is likely to make to young people’s chances of getting a job. 

Many of the interventions included in the Youth Employment Toolkit have an impact on other aspects of young people’s lives, such as their participation in education, their wellbeing, and their psychosocial development. These are important to many stakeholders who want to improve youth employment outcomes. We have not included these impacts in the main impact rating for the Toolkit but they are discussed in the summaries for each intervention.

The impact ratings are based on meta-analysis of data from multiple studies that examine the impact of youth employment interventions on employment outcomes. All the studies in the research for the Toolkit work out this impact by comparing employment rates for young people who take part in the intervention, or who are offered the intervention, with employment rates for a similar group of young people who do not (a ‘comparison group’ design). The young people who do not take part in the intervention are said to receive ‘services as usual’, or the services that are available to young people where no targeted intervention is offered.

The studies in the research for the Toolkit either use randomisation to compare young people who do or do not get offered the intervention, or use ‘quasi experimental designs’ to compare participants with non-participants. This approach makes it possible to see how many young people got a job because of the intervention, and how many would probably have got a job anyway. 

The ratings (‘high’, ‘moderate’ and ‘low’) reflect judgements about what level of impact is likely for an intervention that aims to improve youth employment outcomes. We have based these judgements on the levels of impact identified in previous studies that use a meta-analysis to estimate the reported impact of youth employment interventions.   The technical guide to the Youth Employment Toolkit contains more information about how the impact rating was calculated.  

All the studies in the Toolkit include a measurement of whether young people who take part in the evaluated programme or who are in the comparison group get a job. However, the studies included in the meta-analysis use a lot of different kinds of measurement. For example, the point in time  at which a young person’s employment is recorded varies between studies. The meta-analysis combines these different kinds of measurement.   

Like all averages, these ratings cover variation between the use of the intervention in different contexts, at different times, and when it is implemented in different ways. The summary for each kind of intervention includes information about the extent to which impacts could vary; this is based on the confidence intervals identified in the research. 

The research summarised in the Toolkit was conducted in many different countries and different economic conditions. These factors are likely to affect the impact that each intervention has on youth employment. The way in which an intervention is implemented, as well as the context and place where it is implemented, will also affect its impact. You can find more information about this in the guide to using the Youth Employment Toolkit. 

The evidence strength rating

The evidence strength for each finding is rated as ‘high’, ‘moderate’ or ‘low’.  The ratings must be interpreted in the context of the Youth Employment Toolkit, which is based on a robust rapid evidence assessment and meta-analysis of data. 

The evidence strength rating reflects the number of studies that were included in the meta-analysis for interventions of each type, and on a critical appraisal of the degree of confidence that researchers can place in these studies. It may also reflect some aspects of the statistical analysis involved, such as the extent to which researchers found variations between the findings of studies examining the same intervention.  More detail about the evidence strength rating can be found in the technical guide for the Youth Employment Toolkit.

Researchers used a well-established critical appraisal tool for systematic reviews to judge the confidence that could be placed in the evaluations included in the meta-analysis (following the approach used in the Youth Employment Evidence and Gap Map). Studies are scored depending on how they report their design, sample size, approach to attrition within the sample, and definitions of interventions and outcome measures.  Because it is only possible to score a study on the information that it includes about these factors, a lack of information can lead to a lower score than might have been assigned if the same evaluation had been described in detail. 

The cost rating

The cost rating reflects the range, nature, and likely costs of the various inputs needed in order to implement each intervention effectively.

Many youth employment interventions are delivered in partnership, and need diverse inputs from multiple stakeholders with different roles in implementation. These inputs include contributions to the financial costs of the intervention, expertise, and staff time.

The programmatic nature of many youth employment interventions means that it is often difficult to identify the financial costs of individual interventions that are part of a whole programme. Few evaluations report the cost of different elements of a multi-component programme. From the information in published evaluations, it would be very difficult to extract this information in a reliable way.  Costs also vary considerably depending on duration, setting, and intensity. 

Where detailed information about costs is available, we have included this in the discussion of the intervention. However, this is not possible for many of the interventions in the Youth Employment Toolkit.  

We have rated all interventions as ‘high’, ‘moderate’ or ‘low’ cost, based on the kinds of input that they include.  

Overall, the cost rating will be higher when inputs: 

• Are numerous. 
• Are ongoing, e.g., continuous oversight or assessment.  
• Come from multiple sources, e.g. central government, employers, education providers.  
• Include demands for expertise, e.g. analysis and monitoring of local labour markets.  
• Involve specialised equipment or settings.  
• Can’t easily be combined with other activities or services as usual. 

Other evidence resources

The Youth Employment Toolkit can be used alongside our other evidence resources: 

• Most users will need access to reliable and current data on youth employment to help frame the issue for particular groups of young people, places, and opportunities. The Youth Futures Foundation’s Data Dashboard provides quick access to the most up-to-date figures on youth employment, unemployment, inactivity and NEET rates. 
• The evidence reviews in the Youth Employment Toolkit build on the Youth Employment Evidence and Gap Map (EGM). This resource offers a visual representation of the range of evidence available globally for the impact of different interventions to support youth employment on a range of outcomes. The EGM indicates the quantity and strength of evidence, along with short summaries of the studies included and links to copies of the full text where this is available. The EGM can help you take a deep dive into the evidence for a particular intervention. 

The need for a Youth Employment Toolkit

The literature on interventions that aim to help young people into employment is extensive. However, it is highly variable in its focus and quality, and studies that use a randomised or comparison group design are comparatively few, as are systematic reviews of evaluations. The literature can be hard to navigate, and its quality, relevance, and reliability are highly variable. The Youth Employment Toolkit aims to bring together good-quality evidence, in an accessible format, accompanied by an assessment of the strengths and limitations of the available findings.  

Bringing the research together in this way also helps to identify gaps. Some potentially effective interventions have not been evaluated, or have not been evaluated well enough to give a reliable picture of their effectiveness. Where evaluations have been conducted, these don’t always provide information about everything that Toolkit users would like to know. We have done our best to flag evidence gaps, and to note limitations. 

Who is the Toolkit for? 

This Toolkit is for anyone with an interest in better youth employment outcomes, and who intends to take action to make this happen. It is designed with four main audiences in mind:  

• Policymakers in central or local government, or in other organisations and agencies that are involved in policy development. Policymakers may be most interested in the key findings, about the evidence, and useful links sections.  
• Intermediaries who work with multiple different stakeholders to improve youth employment outcomes and support young people. Intermediaries include youth organisations, grant makers (like the Youth Futures Foundation itself), and agencies that organise interventions and activities for young people.  
• Practitioners involved in delivering youth employment interventions and other programmes and activities for young people. Practitioners may be most interested in the implementation sections of the Youth Employment Toolkit.  
• Employers and employer organisations involved in planning and delivering youth employment interventions, or working in partnership to support young people.  

Limitations

The Youth Employment Toolkit should be used alongside other resources that help you to work out how best to plan interventions to support young people. You can find more information in the guidance for using the Youth Employment Toolkit.  

This Toolkit presents evidence drawn from high-quality reviews of published impact evaluations of interventions. It does not use evidence drawn primarily from, for example, practitioners’ accounts of their own experiences, or established ‘best practice’ that is not accompanied by robust evaluations. Nor does it offer a comprehensive ‘how to guide’ for any particular intervention, although it does include high-level information about how to implement the interventions effectively.  

We want Toolkit users to let us know about how you have used this resource, and about any ways in which we could make it more helpful to you in your work. Please get in touch to tell us about your use of the Toolkit, ask questions, or share your suggestions. 

Over time we hope that the Toolkit will become a site for engagement by stakeholders.

About evaluation

Evaluation involves looking at social interventions, policies and programmes in a structured and disciplined way, to understand their impacts. An evaluation includes the systematic collection and analysis of data related to a policy or intervention and its outcomes. The evaluations used in this toolkit have been assessed for their robustness and quality.  

Impact evaluations seek to estimate the causal effects of a policy, intervention or programme by comparing people who took part in it, or had the option of taking part in it, with a similar group who took part in another option or scenario. The other scenario can be services as usual.  By comparing the outcomes of individuals or groups, impact evaluations measure whether the intervention, policy or programme made a difference and by how much.  Alongside impact evaluation, high quality implementation and process analysis assesses the components and quality of the intervention and assesses to what extent it was delivered consistently in a given context.  

The ultimate aims of evaluation are to understand whether and how a policy or programme improves outcomes for the target group and, more broadly, to add to the evidence base on a particular social issue and the most effective interventions for tackling it. 

Future editions of the Toolkit

In future editions of the Toolkit, it is possible that we will gather evidence on interventions that appear to have a negative impact (i.e. where it appears that they reduce the likelihood of young people who take part getting a job). In that case, we will add ratings to reflect these kinds of impact. 

Likewise, in future editions of the Toolkit it is possible that we will include interventions where no suitable studies for inclusion in an evidence assessment and meta-analysis are available. In that case, we will add ratings to reflect this kind of evidence base.  

Please get in touch if you’d like to suggest which interventions you’d like to see in the Toolkit. We might not be able to find enough evidence to include them, or to include them in full, but we will consider all suggestions as we plan future updates to the Toolkit.