Supporting careers and learning: towards common standards for monitoring and evaluation in Europe

The virtual workshop invited European and international experts, researchers and stakeholders working in the lifelong guidance policy field, particularly experts in monitoring and evaluation at different levels of the system, in public policy research and policymaking, guidance providers, programme facilitators, and career counselling trainers and others.

The main objective was to exchange and generate new knowledge for supporting national efforts in monitoring and evaluation and quality assurance, in building the evidence base on individual and social outcomes of interventions and promoting solid approaches to monitoring and evaluation in career guidance. In particular, focus was on discussing current research trends and national practices in career services, and the methodological options for evaluating the outcomes and impacts of career development support services and measures for adults (18+), in a lifelong learning perspective.

The event was an opportunity for authors of six research papers to present their findings from the recently published collection of working papers and to present ongoing work by Cedefop, including a study conducted on the topic. Group discussions in breakout rooms focused on an accompanying draft framework comprising elements targeted for monitoring and linked potential indicators.  Participants debated the scope for agreement in working towards minimum standards for monitoring and evaluation that can apply nationally, across guidance sectors, particularly those serving adults.

The study presented seeks to learn which approaches combine towards achieving an integrated vision of the results of career development measures and activities. 

The workshop was restricted and by invitation only. A larger follow up event is envisioned once the final report is published, reflecting new national developments and research in the field.  

News headline on the event can be found here.

Presentations – Day 1

Session 1: System level perspective – Holistic approaches to monitoring and evaluation 

Exploring the potential of a systemic approach for monitoring and evaluating career guidance systems and services, Raimo Vuorinen and Jaana Kettunen (Finnish Institute for Educational Research, University of Jyväskylä, Finland)

Towards a general framework of evaluation of career counselling services: the Polish case, Jerzy Bielecki, Tomasz Płachecki and  Jędrzej Stasiowski, (Educational Research Institute, Warsaw, Poland)

Session 2: Monitoring person-centred approaches and rates of return

Lessons from person-centred adult educational guidance services in Ireland, Petra Elftorp (University of Limerick, Ireland) and Mary Stokes (National Centre for Guidance in Education, Ireland)

Lifelong guidance and welfare to work in Wales: linked return on investment methodology, Chris Percy (University of Derby, UK) and Deirdre Hughes (DMH Associates & Associate Fellow, University of Warwick IER, UK)

Session 3: The importance of practitioners and guidance processes

Monitoring and evaluating the German lifelong guidance programme: a proposal in the context of the professionalisation of guidance counsellors, Peter C. Weber (University of Applied Labour Studies, Mannheim, Germany)

Towards more effective career guidance processes: the CREAR digital tool for improving client-oriented interventions, Helena Kasurinen (HK Career guidance and consulting Ltd, Finland) and Mika Launikari (Laurea University of Applied Sciences, Finland)
 

Additional outcomes from the workshop

On 21 and 22 March 2022, Cedefop gathered more than 76 participants working in Europe and internationally in the field of career guidance as researchers and policy experts, practitioners and trainers, service managers, policy makers and consultants, across education, employment, youth, and social inclusion sectors and settings. The European Commission, European Commission PES Network, International Labour Organisation (ILO), the European Training Foundation (ETF), OECD, UNESCO, and Cedefop staff working in the field also participated. Priority was given to invite participants with expertise and experience both in lifelong guidance/career development as well as methodologies and research in monitoring and evaluation, especially in public policy and provider contexts.

An expert panel shared reflections to conclude the event. They responded to questions on the report presented, the papers from Day 1 and the list of indicators. The four speakers were invited to share their views from their expertise, vantage points and experience in the field, and how they view future developments and which takeaways resonated most strongly through the discussions.  Hilde Olsen, PES Network Secretariat, Pedro Moreno da Fonseca, International Labour Organisation, Teea Oja, Ministry of Economic Affairs and Employment of Finland, Alan Brown, (Emeritus) University of Warwick Institute for Employment Research and Peter Robertson, Professor working in career guidance and career development at Edinburgh Napier University.  Each focused on key messages that Cedefop could take forward. They also provided inputs needed to move further on in developing the indicator framework and finding common areas of agreement and where efforts meet most challenges.

Participants were selected to join the workshop to share knowledge and practices on monitoring the results and effectiveness of career guidance interventions and to revisit discussions and debate on how careers and learning can be enhanced though evidence-informed policy on monitoring and evaluation in career guidance. Through presenting Cedefop work in the area, the aim was to renew discussions on common indicators and methodological approaches in the field – and nationally, how to effectively capture outcomes that guidance professionals, users and policy makers can agree should be monitored based on the agreed aims of LLG.

The workshop was organised to present and discuss work in progress produced in a Cedefop project that seeks to support adult learning and careers through quality career services and to share knowledge and raise important questions faced in the field.

During the workshop, the following were presented and discussed:

• Cedefop work in progress, including innovative paper contributions made by external experts in the study’s first volume of working papers (Cedefop et al., 2022)
• findings from accompanying work that synthesises and analyses the empirical evidence collected on career development support and career guidance measures from nine EU countries, as well as highlights from the literature they reviewed
• a preliminary draft list of minimum target elements and indicators for monitoring and evaluating career guidance (informed by data collection in 2019-2021 and to be further contextualised and elaborated)

In addition to the high-quality external research papers published and presented for increasing knowledge dissemination and delivery, the workshop served as a mutual learning opportunity on the topics in focus, aimed at providing an online platform for participants to learn together while building knowledge and comparing experience. This was done through exploring and critiquing common methodological, conceptual, and practical challenges related to quality assurance in the field, as well as ensuring that approaches proposed are able to measure relevant outcomes and also be process-focused and account for meaningful career guidance and development interventions.

The work should be relevant and ultimately support national and service level capacity building in development of good policies on monitoring inputs, processes, outcomes and impacts through considering the big picture and shared challenges and aims. This means to open and deepen new perspectives from a system, service, and guidance user perspective in engaging in active research, knowledge development and sharing and tool development based on common challenges and methodologies. One key area of this work that was discussed was developing policy tools to support expanding the evidence base on career guidance results and impacts for evidence-informed policymaking adaptable to relevant levels and contexts. The list of indicators itself is not prescriptive and is more so to start discussions on common elements needed for monitoring and evaluation at different levels and to find common ground in Europe, enriched with national and international discussions on this topic.

Next steps for Cedefop

Future actions, directions and priorities will entail:

• publication of the draft set of indicators within the report on incentives to learning and careers
• continuing development and future expansion of the draft set of indicators presented at the workshop through various means and promoting development of quality frameworks in the context of standards in LLG
• supporting the development and use of effective cross-sectoral, coordinated, and coherent approaches to monitoring and evaluation in LLG  at various levels
• supporting continuing discussions on finding common standards and shared indicators in this policy field in Europe, and following national and international developments in this area 
• follow up activities (events, studies, working papers, etc.)
• enriching Cedefop’s knowledge base with new contextual evidence and feedback on this work