By David Taylor and Dr Ellie Ott
Dr Ellie Ott, Senior Advisor at Centre for Evidence and Implementation, and David Taylor, Research Fellow at Monash University, reflect on the use of Network Meta-Analysis to build the world’s first youth employment toolkit.
Youth Futures Foundation engaged a team from the Centre for Evidence and Implementation (United Kingdom, Australia, Singapore and Norway), Monash University (Australia), and the Institute for Employment Studies (United Kingdom) to undertake a series of rapid evidence assessments and meta-analyses to inform the development of their Youth Employment Toolkit. The Toolkit will provide policymakers and practitioners with information about evidence-informed programmes across a range of topic areas.
Our first task was to figure out which approaches to youth employment had both sufficient evidence to warrant quantitative synthesis and that were of policy-relevance in England by rapidly scoping the literature and engaging with stakeholders. This led to the selection of seven types of interventions for the first iteration of the toolkit: wage subsidies, apprenticeships, on-the-job training, off-the-job training, basic skills, life skills and coaching and mentoring.
Interventions with multiple components
It turns out that it’s not that simple to ascertain which types of interventions are effective at improving employment outcomes for youth. During the scoping phase we noticed that many of the same studies popped up across different topic areas — employment programmes don’t tend to find into neat categories! In reality, many of the programmes that had been evaluated using experimental or quasi-experimental methods had multiple ‘intervention components’ that could include, for example basic skills training, on-the-job training and coaching and mentoring. As an aside, this dovetails with the experiences of some of the young people advising the review who discussed the merits of experiences like having an apprenticeship which also included elements of life skills and off-the-job vocational training to learn about a field, the realities of working in the field, and how to work with people.
When we undertake a meta-analysis of experimental or non-experimental studies, we are aiming to estimate the true effect size of single intervention. In doing so, we include studies where the intervention we are interested in is compared to similar comparison groups, for example ‘services as usual’. When all else is equal this lets us estimate if the intervention is more effective than services as usual. The central challenge we faced is that doing a meta-analysis on a single employment programme component — basic skills, for example — could be biased as the observed impact of the programme as a whole could be down to the other components of the programme — such as off-the-job training — or the way in which multiple components interact with each other.
Network meta-analysis
To respond to these challenges, we decided to undertake a network meta-analysis. A network meta-analysis (NMA) is a methodological technique that can be used to quantitatively synthesise a diverse group of practices by generating an effect estimate for each combination of practices (Melendez-Torres et al., 2019). In its most simple form, an NMA synthesises both direct — from head-to-head comparisons — and indirect evidence — obtained via a common comparator — to allow for the comparison of multiple practices (Petropoulou et al., 2021). It also allows for an estimation of the ranking and hierarchy of each of the included interventions — which is very helpful for policymakers and programme designers (Chaimani et al., 2022).
An NMA is a powerful tool that — relative to a standard meta-analysis — allows for the inclusion of a wider range of evidence, including: studies that test multiple active practices against each other, different arms of experimental studies, and the ability to synthesise experimental and quasi-experimental study designs.
Our network meta-analysis will look at the following approaches to improve youth employment and education outcomes:
- Apprenticeships
- On-the-job training, that includes traineeships and structured internships
- Off-the-job training, such as predominantly classroom-based technical and vocational training
- Basic skills training that focuses on literacy, numeracy and digital skills,
- Life skills training that covers softer skills that equip young people with the life management skills that underpin employment
- Mentoring and coaching
To our knowledge, this is a world’s first for an evidence toolkit! We have spoken to other researchers who are using other innovative methods — including NMAs and Bayesian meta-regression — to explore related-questions around employment for different populations (i.e., adults) in different settings (i.e., low- and middle-income countries). This approach offers the exciting opportunity to synthesise and translate evidence so that it is more practical for policymakers, useful for the youth employment sector, and can be applied in a way reflects the experiences and preferences of young people.
Check back on our website in June to try out the Toolkit.
References
Chaimani, A., Caldwell, D. M., Li, T., Higgins, J. P. T., & Salanti, G. (2022). Chapter 11: Undertaking network meta-analyses. In J. P. T. Higgins, J. Thomas, J. Chandler, M. Cumpston, T. Li, M. J. Page, & V. A. Welch (Eds.), Cochrane Handbook for Systematic Reviews of Interventions: Vol. Version 6.3 (updated February 2022). Cochrane. https://training.cochrane.org/handbook
Melendez-Torres, G. J., Leijten, P., & Gardner, F. (2019). What are the optimal combinations of parenting intervention components to reduce physical child abuse recurrence? Reanalysis of a systematic review using Qualitative Comparative Analysis. Child Abuse Review, 28(3), 181–197. https://doi.org/10.1002/car.2561
Petropoulou, M., Efthimiou, O., Rücker, G., Schwarzer, G., Furukawa, T. A., Pompoli, A., Koek, H. L., Del Giovane, C., Rodondi, N., & Mavridis, D. (2021). A review of methods for addressing components of interventions in meta-analysis. PLoS ONE, 16(2 February), 5–18. https://doi.org/10.1371/journal.pone.0246631
