About the tool

The key process context factors that outline the transferability of the 25 ICT and LMI practices have been identified with the use of the Extrapolation Protocol (Ongaro, 2009[1]). This analysis has been made in the frame of the full case studies’ development process as well as through the 4-pager analysis of the 25 ICT and LMI practices.

These key process context factors have been summarised in three main constructs, namely Relevance, Context and Applicability. Within these constructs specific factors/ criteria and respective questions have been identified. The identified constructs, criteria and questions have fueled the development of the Decision-Making Tool for future transferability assessment (adapted from Buffet et al, 2011[2]).

In sum, the tool constitutes a practical framework for assessing the transferability potential of an ICT and LMI practice to a target setting. This is achieved by helping the potential users of the tool to reflect upon the following three types of transferability assessment constructs and respective factors/criteria.

  • Relevance

    The factors/ criteria and questions of the Relevance construct aim at assessing whether the practice can achieve similar results in the targeted geographical level (local/ regional/ national) and setting (goal/ objective vs. need). Assessment criteria for the Relevance construct include factors such as the existence of the LLG objectives in the target context, the magnitude of the LLG challenges in the target geographical level and setting, the comparability of the target populations, the geographical relevance and the scalability potential of the practice, etc.;
  • Contextual

    The factors/ criteria and questions of the Contextual construct aim to assess the higher-level implementation parameters of the practice in a target setting (i.e. contextualizing factors). In doing so, this construct takes into account factors such as the political and social acceptability in the target context, the availability of organisations with the expertise and capacity to implement the practice, the level of engagement and potential support of the LLG stakeholders, etc.;
  • Applicability

    The factors/ criteria and questions of the Applicability construct aim at determining whether it is possible to implement the ICT and LMI practice in the target geographical level (local/regional/national). Criteria for assessing feasibility include aspects such as the IT capacity to implement the required ICT tools, the availability of required LMI, the existence of supportive legal framework, the existence of legal prerequisites and barriers, the availability of resources (i.e. human, financial, technological), etc.

The goal of the Decision-Making Tool is not to channel its users towards specific practices or to offer a full evaluating framework of ICT and LMI practices. Instead, it aims to function as a thought-provoking tool with the potential to assist its users to reflect upon the key aspects of an ICT and LMI practice and how these correspond to the needs, norms and capacity of the target context.

As such, the overall goal of the Decision-Making tool is to present the key transferability assessment factors/criteria (i.e. Relevance, Contextual, Applicability) and to consider a broad range of respective questions that need to be asked prior to the actual adoption of an ICT and LMI practice that has been developed and implemented in a different context.


[1] Ongaro, E. (2009). “A protocol for the extrapolation of “best” practices: How to draw lessons from one experience to improve public management in another situation. European Public Sector Award 2009”, Final Symposium and Ceremony, Maastricht.

[2] Buffet, C, D Ciliska and H Thomas (2011). “It Worked There. Will it Work Here? Tool for Assessing Applicability and Transferability of Evidence. Hamilton”, On: National Collaborating Centre for Methods and Tool

As it was mentioned above, the Decision-Making Tool aims to assist LLG decision makers and managers of relevant organisations when selecting ICT and LMI tools that have been successfully applied in other contexts. However, in order for the tool to be used, the implementing authorities must have already completed the following steps:

  • Define the LLG challenges and objectives that need to be addressed;
  • Go through the inventory of ICT and LMI practices as well as through the respective 4-pager descriptions so as to identify the most relevant practices to be considered;
  • Identify additional relevant ICT and LMI practices and tools.

The abovementioned steps are necessary prior to the actual implementation of the tool that is going to assist the decision makers and managers to decide whether to adopt and introduce in their context the identified ICT ad LMI practices.

Once the ICT and LMI practices under consideration have been identified, the Decision-Making Tool can be applied. To this end, there are six steps that have to be followed:

  • Step 1. Define the people who will be involved in the implementation of the Decision-Making Tool.

    All the following steps should be executed in collaboration with this group of people.

    It is recommended this step is highly inclusive, so as to ensure the participation of all the different stakeholder groups that (i) are relevant to the potential development and implementation of the ICT and LMI practice, and (ii) hold valuable insights regarding the Relevance, Contextual and Applicability factors in the target context.

    At this point, it should be mentioned that the participation of the actual potential users of the ICT and LMI practice (e.g. students, guidance practitioners, school teachers, etc.) is imperative, since this group of people will provide important acceptability insights. Furthermore, the participation of experts from various levels of the local/ regional/ national LLG context should also be sought, since this will enhance the quality of the assessment, while at the same time offer feedback from the standpoint of the local/regional/national LLG setting.

  • Step 2. Selection of the most relevant transferability factors/ criteria (i.e. Relevance, Contextual, Applicability).

    Since not all factors/criteria are relevant for every case, the people who participate in the implementation of the tool should choose the transferability factors/ criteria and respective questions that are more important for their case. These factors have to be selected for the specific practice(s) under consideration as well as for the local/regional/national target context (e.g. political environment, organizational structure, target population features, etc.). It is important that a consensus is reached among all the stakeholders regarding the final shortlisted factors/criteria.

    It should be added that, although, the recommended use of the tool requires the assessment of a mixture of Relevance, Contextual and Applicability factors and questions, the Decision-Making Tool can also be applied for assessing these three constructs separately.

  • Step 3. Establishing the priority of each factor/ criterion - Allocation of weights.

    At this point, it should be defined whether the above mentioned selected factors/ criteria are equally important. Since, usually a set of factors/ criteria have different priority during an evaluation process, these factors can also be given different weights according to their level of impact.

    This process has to be unique for every case and its outcome should be derived after internal discussions and in consultation with the group of stakeholders who will be involved in the application of the Decision-Making Tool. This adaptation to each case is necessary since the users of the Decision-Making Tool are those who will hold the information required to prioritize the selected factors/ criteria at a particular time period and for a specific target context.

  • Step 4. Define how the different weights will be allocated among the various factors/ criteria.

    There are various methods that can be used for setting the weights of the various selected criteria. Except for quantitative techniques such as the Analytical Hierarchy Process (Saaty T. L., 1980[1]), the use of average scores and the allocation of simple numeric values, qualitative ratings can also be assigned to the selected criteria which will then correspond to a specific numeric value for the needs of the assessment (i.e. Low Priority = 1; Medium Priority = 2; High Priority = 3). These methodologies are not analysed herein since their in-depth review goes beyond the scope of this study.

    However, it is important that the final prioritization and allocation of weights to the selected assessment factors/criteria, will be based on the expert knowledge and context-based experience of the participating stakeholders.

  • Step 5. Select a scoring system for the assessment factors/ criteria.

    The users of the tool should define the scoring system that better fits their case. An easy and common way to address this step are the individual ratings of each factor/ criterion on a 1-5 Likert scale. The highest the final score, the more transferable a practice will be to the target context. In case several practices are under consideration, the priority will have to go to the practice that achieved the highest score.

    The users of the tool can also establish a threshold for the overall scoring or even for individual assessment constructs and respective factors/ criteria.

  • Step 6. Use the 4-pager descriptions as well as the full case studies of the 25 ICT and LMI practices.

    In order to define the response of their context to the selected Relevance, Contextual and Applicability assessment factors/criteria, the users of the Decision-Making Tool should reflect upon the 4-pager descriptions of the ICT and LMI case study practices that were presented in Section 3 "Transferability elements of the 25 case studies", as well as in the full versions of the respective case studies.

 

[1] Saaty T. L. (1980), “The Analytic Hierarchy Process”, New York: McGraw-Hill

The Decision-Making Tool (DMT) for future transferability assessment of ICT and LMI practices (adapted from Buffet et al, 2011), is a practical framework that is designed in three levels, namely:

i) The Constructs,

ii) The Factors / Criteria, and

iii) The Questions.

The figure below illustrates how the DMT is designed and developed.

Figure 1. The three levels of the DMT

The transferability assessment Constructs include the most important identified aspects that affect and largely define the transferability potential of a practice. Overall, the DMT suggests that the transferability of ICT and LMI practices should be viewed and assessed against the Relevance, Contextual, and Applicability aspects of a practice when this is considered in the target context. The Relevance construct of the DMT aims to assess the possibility that an ICT and LMI practice can be generalized and achieve similar results when applied in the target context.  The Contextual construct targets the high-level implementation parameters in the new setting, since it aims to define whether the key enabling environment and factors for the successful transfer of the practice are in place. Finally, the Applicability construct aims to assess the capacity of the target context as well as of the implementing organisation to apply the ICT and LMI practice in a proper way.

Following, a number of Factors/Criteria have been identified and connected to each of the abovementioned transferability Constructs.  It is suggested that these Factors/Criteria are to be used as indicators for the assessment of the Constructs since they refer to several important features of the ICT and LMI practices, including both their salient aspects as well as more specific elements such as environmental and institutional factors (e.g. local/regional/national policy framework, stakeholders, resources, etc.). Overall, these Factors/Criteria are meant to assist the users of the DMT in comparing the original and target contexts and in identifying mismatches and similarities between them.

The DMT also provides its users with several transferability assessment Questions that fall under each Factor / Criterion. These Questions are to be asked by the users of the DMT when trying to assess the respective Factors/Criteria.

The table below, outlines all the above-mentioned aspects of the DMT. The “Weight” column of the DMT is provided for the users of the tool and it offers space where they can state the weights that will allocate to each Factor / Criterion after following the Steps 3 and 4 that are described in the section “Guidelines for the application of the Decision-Making Tool”.

Construct

Factors/Criteria

Questions

Relevance

 

Answers to the following questions

Can the results be generalised?

Can we expect the same results?

 

 

Objective(s) of the practice

Is the practice targeting the same LLG priority objective(s) in the donor and in the target context?

(e.g. improving employability, Career Management Skills, supporting people at risk and disadvantaged groups, raising the skills and qualifications of young people, etc.)

Magnitude of the LLG challenge(s) addressed in the target context

 

 

 

Does the same need(s) exist? Do these LLG challenges exist in our setting?

(e.g. Tackling unemployment, facilitate knowledge exchange among practitioners, transition from school education to career selection, etc.)

Does the practice function as a one-stop-shop that addresses multiple LLG challenges?

Are there any similar initiatives that already address these LLG challenges in our context?

What is the baseline prevalence of the LLG challenges in our context? How many people in our context are affected by these challenges?

How does the above compare with the prevalence of the LLG challenges described in the donor situation? Are the problems caused by these LLG challenges as large as in the donor context? What is the difference between the two contexts?

Magnitude of the potential reach of the practice

 

 

Did the practice help the organisation in the donor context to achieve its objectives and strategic plan?

Can we reach a large proportion of our target user groups by using this practice? What will be the coverage rate for our main target groups?

Can this practice lead to increased usage rates by our current users/ customers?

Will this practice help our organisation to enhance its services, processes and technological competences so as to attract new user groups?

Does the practice have the potential to lead to higher user satisfaction rates?

Will the practice lead to increased number of users/ increased sales?

Is the practice inclusive in addressing the needs of various target groups?

Comparability of target groups

 

 

Do we target the same target groups with those targeted in the original? (e.g. school students, unemployed, adult education, career guidance practitioners, immigrants, etc.)

Is our target population comparable to the population of the original context?

Are there any differences in characteristics that could affect the effectiveness of the practice in the local setting? For instance:

  • Cultural factors (lifestyle, beliefs, preferences, behavioural factors, etc.)
  • Factors depending on their age, language, ICT literacy, educational background, etc.
  • Ethnicity, socio-economic, demographic factors, etc.

Geographical relevance

 

 

Does the practice fit our geographical focus?

Is the practice scalable?

Has it successfully been scaled from local to regional or national level?

Contextual

 

Answers to the following questions

Is the context of the target setting appropriate for the uptake of the practice?

How does the practice relate to the key enabling environment in the target context?

 

 

Political acceptability

 

 

Does the objective of the measure match with political priorities of our context?

(e.g. alignment with local/ regional/ national LLG policies or growth strategies, policies promoting ICT and LMI tools, etc.)

Does it conform to existing local/regional/national regulations and policies? Does it have contradicting effects? (e.g. LLG regulations, data privacy regulations, etc.)

Is the practice in line with the national curricula and LLG framework of the target context? It is in line with the ELGPN guidelines?

What are the government's indicators for success of practices that correspond to its LLG priorities? Has the practice yielded similar results in the donor's context?

Is the socio-economic context that has led to the development of the practice similar to this of the target context?

Social acceptability

Will the target population be interested in the practice?

Do our target groups demonstrate the needs/ challenges that are addressed by the practice?

Are the aspects of the practice in line with the local/ regional/ national norms?

Does the practice follow a needs-driven / user-centered design approach?

Is the practice inclusive in addressing the needs of various target groups? (e.g. one-stop-shop approach, single access point, etc.)

Is the LMI that is provided by the practice innovative? (e.g. real-time LMI, interoperability with other ICT practices, crowdsourcing of expert knowledge, etc.)

Does the practice use innovative ICT methods that can be attractive to users? (e.g. interactive online tools, social media usage, multimedia, etc.)

Is the practice easy to be used by the target groups? Does it require any training? Is its design user friendly? Can it be used without the assistance of a guidance practitioner? Can it be used from people with low digital skills?

Does the practice ensure personal data privacy?

Support of the LLG system's stakeholder

Can we achieve the necessary LLG stakeholders' cooperation, support and role as in the donor's context?

(e.g. education agencies, employment offices, guidance practitioners, business community, citizens, local authorities, etc.)

Can the local/ regional/ national LLG stakeholders be engaged in the development and delivery of the practice in our context? (e.g. provision of the practice, LMI provision, etc.)

Impact on other affected interest groups/ stakeholders

(since this is a "negative" statement, the lower existence of such impacts is, the higher the score of this factor should be)

Does the practice contradict the interests of any important stakeholders of the local/ regional/ national LLG system?

(e.g. National Employment Agencies, National agencies for various levels of education, etc.)

Institutional capacity and compatibility

Which organization will be responsible for the provision of the practice in the local setting?

Is the practice consistent with the mission and ideology of our organisation?

Is the practice in line with the current strategic plan of the organisation? Does the organisation have a strategy plan for the promotion of ICT and LMI tools?

Is the practice reciprocal to existing practices of the organisation? Does it overlap or contradict with existing practices?

Can we support/ achieve the same organisational structures, necessary for the implementation of the practice?

Is the implementing organisation motivated to learn and open-minded to new practices?

Are there any possible barriers due to the structure of the organisation that will implement the practice in the local/ regional/ national context?

Applicability

 

Answers to the following questions

Can the practice be implemented in the local/ regional/ national context?

Is the capacity to implement the intervention comparable?

Can it work for us?

 

Existence of supporting legal and regulatory framework

 

 

Is the practice, in the donor's context, attached to any specific LLG legislative framework? Is it the outcome of or directly connected to a local/ regional/ national LLG policy?

Is the practice compulsory by law or voluntarily used in the donor's context? How is it going to be offered in our context?

Is the necessary legislative context in place in our own setting? (e.g. policies integrating ICT tools and LMI in LLG interventions)

Is the practice a result of a local/ regional/ national/ European initiative? Do similar political initiatives exist in the target context?

Existence of legal prerequisites for the replication of the practice

(since this is a "negative" statement, the lower existence of such prerequisites is, the higher the score of this factor should be)

Is the practice open for use by other organisations or countries?

Is the practice the outcome of a public or a private organisation?

Are there any IPR, copyright, private data restrictions that affect the implementation of the practice in our context?

Does donor organisation support an open data policy?

Role and availability of required LMI aspects

Does the utilisation of the LMI by the practice serve the LLG needs and priorities of our local/ regional/ national context as well as those of the organisation that will implement the practice? (e.g. effective job matching, personalised educational advice, occupational information, etc.)

Do we have experience in offering similar LMI?

Is the LMI that is required for the development and implementation of the practice already available in our context?

Are the processes and collaborations with internal and external actors required for the development of the LMI, easily replicable in our context?

(i.e. processes and collaborations for the LMI gathering, validation, reliability, update, etc.)

Does our organisation have already established connections with the required LMI sources and stakeholders?

Do stakeholders who hold important LMI in our context, follow an open data policy?

Does the required LMI fall under any specific data privacy regulations?

Role and easiness of use of required ICT aspects

Does the organisation that will implement the practice, have experience in utilizing similar ICT tools and processes for the provision of LMI? (e.g. online counselling, online wiki, interactive tools, etc.).

Does the practice require any complex technological infrastructure?

Does the practice require any technological aspects that are protected by copyright regulations?

Is the practice built on open source software that is freely available?

Availability of resources necessary to deliver the practice

 

 

 

 

 

 

 

Which organisation will be responsible for developing and offering the practice in our context?

Does the organisation have the administrative and enforcement capacity to deliver the practice in our context?

Are the human resources adequate for the initial development and the routine application of the practice?

Are our human resources adequately trained in aspects that revolve around the required LMI and ICT usage? If not, is their training feasible?

Is the practice based on any guidance model or theoretical approach, on which our human resources have experience?

Are the financial resources adequate for the initial development and the routine application of the practice?  (i.e. considering staffing requirements, technology required, training, administrative support, promotional actions, etc.)

Is the technological infrastructure required for the development and application of the practice available? If not, is it easy to acquire such infrastructure in terms of time and costs?

Adaptability of the practice

Will our target groups accept the practice in its current format?

Is the language of the practice appropriate for our target audiences? Does it have to be translated?

Is the practice adaptable to the reality of our context?

Is the practice adaptable to the characteristics of our target groups?

Can we adopt only certain features of the practice?

Existence of other barriers and implementation risks

(since this is a "negative" statement, the lower existence of such barriers is, the higher the score of this factor should be)

 

Other local barriers and implementation risks such as:

  • Potential resistance to change from the target users and the LLG stakeholders.
  • Lack of provisions for people with inadequate digital skills.
  • Inability to apply the mitigation strategies that were used in the donor's context regarding possible obstacles for the development and application of the practice.
  • Political volatility in the local / regional/ national context.
  • Other