View examples of how countries are implementing this aspect
Validation tools and techniques refer to, and are closely dependent on, the definition and interpretation of the reference points discussed above. The following table provides an overview of the main forms of validation tools, to be used independently or in combination. Indications regarding the four main quality requirements (reliability, validity, scalability and cost) are included in the table and further discussed below. The different types will have a different degree of the four quality requirements depending on how and by whom they are applied. A mix of tools will better capture the complexities, both breadth and depth, of individual learning experiences. Any tool selected will always need to be appropriate to the context and purpose it applies to. More research on tools and instruments and how these are adapted to validation is required.
Table 1. Tools and instruments used for validation
| Types | Reliability, validity and related issues | Scalability and cost | Main relevance to stages of validation |
| Self-assessment (self-declarative) |
|
| identification |
| Fixed response/multiple choice (close-ended forms) |
|
| assessment, certification |
| Written tests (open-ended forms), including essay |
|
| assessment, certification |
| Dialogue based/interviews |
|
| identification, assessment |
| Simulation and controlled job practice |
|
| assessment, certification |
| Portfolio of evidence |
|
| identification, documentation, assessment, certification |
| Reports from others (colleagues, supervisors, clients, etc.) |
|
| identification, documentation, assessment, certification |
5.4.1. Self-assessment and self-declarative methods
In recent years, and with the increasing possibilities of ICT, several self-assessment tools have been developed. They are based on individuals’ independent identification and recording of competences (sometimes against given criteria and sometimes not). In some instances, the self-declaration might be verified by a third party. The assessment results are then recorded, for instance in a skills passport or in a CV or a similar document. Self-assessment methods are often well suited for formative assessments and are usually used for identification as a first step or as a screening tool, to decide on next steps. Guidance platforms sometimes have self-assessment tools that are carried out with counsellors.
This approach is flexible, enabling individuals to reflect on their knowledge, skills and competences at their own pace. However, these processes will normally suffer from lack of validity and reliability, due to the absence of external objective assessment and a tendency for individuals not to be fully aware of their level of skills. In practice, the validity and reliability of these methods depends on the existence of clear guidelines or standards for the individual to use, on the provision of support during the preparation phase, and on the individual’s ability to provide a realistic assessment of their own competences. Help from counsellors can increase the fairness of this method, particularly as individuals from different backgrounds may have different ways of presenting their skills and competences. Counsellors can help to moderate the importance of such biases in the reporting of individuals’ own learning.
One of the main weaknesses of the declarative method is that, on its own, it can rarely lead to a clear linking to existing qualifications or standard frameworks, particularly in the absence of guidance, and rarely leads to the award of a qualification. It also tends to be trusted less by others.
5.4.2. Fixed response/multiple choice (close-ended forms)
Most of the surveys and questionnaires, as well as competence assessment tests, tend to be close-ended forms, in which the individual has to choose one correct answer among several. Close-ended questions, if developed adequately, can offer enough room for personal expression and contextual adaptation. From a theoretical perspective, close-ended questions allow respondents to choose their answers in a continuous dimension more efficiently than if they had to elucidate the answers themselves.
Psychometric tests are based on these principles. Their development might be costly at first, given the need to elaborate the items and calibrate the tool. Development requires an adequate level of expertise, both in terms of the domain and on questionnaire development. Wording and interpretation of the questions plays a major role in creating valid and reliable tools. Once the questionnaire is developed, the unit-cost is relatively low and there are possibilities for scaling up.
These tools tend to be used for summative assessment, as they are perceived as valid and reliable. New technology and advancements in statistics are allowing for more sophisticated tools that are more efficient and reliable. Using IRT models and computer adaptive testing, for example, test length is considerably reduced.
5.4.3. Written tests (open-ended forms), including essay
This is probably the most familiar type of test. It is easy to administer and relatively easy to develop, though it might be problematic for individuals with language difficulties or negative previous experiences.
The reliability might depend on the degree of professionalisation of the assessors and their level of agreement. Different assessors might interpret answers differently. Validity might also be compromised as essay might require a different set of skills from the ones to be tested. The individual might be very knowledgeable or skilled, but might not be able to fully express it through open-ended questions; this is particularly so with migrants or others who do not speak the language. At different moments in time or situations they might also give different results.
The scalability of the method is relatively easy, and its low cost and familiarity makes it a very common tool, normally combined with other assessment methods, although evaluation can be time consuming compared with other types of tests.
5.4.4. Dialogue or conversational methods
Conversational methods of assessment can be categorised in two main types: interviews and debates (or discussions). A presentation followed by an interview/debate is also relatively common. Interviews can be used to extract further information documented through other means and probe documented knowledge, skills and competences. There might be different levels of structure, from highly structured interviews, semi-structured to unstructured. Interviews could be considered to have a supporting function, which allows for further exploration, instead of being a primary means to elicit non-formal and informal learning.
Dialogue and interviews can have an important role in themselves at various stages of validation and can be very useful tools at the time of identification of acquired competences. When they take place early in the process they can be used as a screening tool, to check whether further mechanisms to extract evidence should be applied. Interviews can be used both for summative and formative purposes.
Interviews can have a higher degree of validity than tests and examinations as they enable dialogue - offering the chance to avoid misunderstandings in the formulation of questions - and also probing. However, they can be less reliable than tests and examinations unless appropriate protocols are implemented as different interviewers (given their experience, personal characteristics, interviewing style) may affect the interview outcome. They can also be less fair than exams, as assessors can be influenced by the personal characteristics of interviewees. Assessors’ experience, communication and facilitation skills, and their thorough knowledge of the assessed learning outcomes (so that relevant and appropriate information can be extracted), are vital to the resulting validity, reliability and fairness.
Cost is relatively low and scalability possible, although they require significant amount of time, with the candidate (depending on the amount of learning that needs to be assessed) and in preparation.
5.4.5. Observations
Observation as a method means extracting evidence from candidates while they perform real-life tasks. This approach, judged by a neutral assessor, has relatively greater usage in the private sector, but is spreading to other areas as well.
The validity of observations can be high and can give access to competences difficult to capture through other means. Observations have the advantage that sets of skills can be assessed simultaneously, and measurement be valid. They are also fair, as people are not detached from their usual work environment and placed under additional stress before the assessment. Nevertheless, assessor bias may exist as personal characteristics of individuals and their workplace are revealed during the process; this may influence the assessment outcome.
Observations are not always possible due to characteristics, safety, time constraints and other factors. They may also be time-consuming, particularly if there is more than one assessor. Further, because observations are grounded in everyday practice, information obtained through them for assessment of an individual may be context-specific rather than subject to generalisations.
5.4.6. Simulation and controlled job practice
In simulation methodologies, individuals are placed in a situation close to real-life scenarios to have their competences assessed. In some cases they are used when observations are not possible. Their use, however, is constrained by several aspects, particularly costs. Some situations cannot be observed in real life, for security or other reasons: examples are reaction of aircraft pilots or bus drivers to extreme weather conditions or a motor/engine failure.
The use of simulations, in the same way as observations, scores high on validity. However, simulations can be more complex to organise and more expensive than other validation methods. Recent developments in virtual reality (VR) are making simulations cheaper. They normally require a large amount of study and job analysis to be prepared properly. The higher the level of realism of the simulation, the more effective the assessment will generally be. Simulations can solve part of the problems of observations undertaken at work as they can place individuals in various contexts, increasing assessment validity. The reliability and fairness of this method are often considered high.
5.4.7. Reports from others
Third party reports for validating non-formal and informal learning can adopt various forms. They can include reference letters (or audio/video declarations) from supervisors, employers and/or colleagues and performance appraisals by companies. The latter are quite common but are not always designed to be used outside the enterprise. The implication of this is that employees sometimes have difficulty in proving their real level of work experience, particularly where their actual performance - and thus skills and competences - exceeds that indicated by the formal job title. Employer reports can help to document the actual tasks performed.
New methodologies, like feedback 360 performance reviews, might combine input from different sources, including different people. ICT job searching platforms also use information from peers to signal competence strength.
The level of reliability and validity depends on the number of people that report and of how much it is possible to trust their judgement. The tool is scalable and not too expensive to develop, but certain categories of validation users might have difficulties in getting adequate numbers.
5.4.8. Portfolios
Portfolios are one of the most complex and frequently used methods to document evidence for validation purposes. Portfolios aim to overcome the risk of subjectivity by introducing a mix of instruments to extract evidence of individuals’ competences and can incorporate assessments by third parties. They provide the audience with comprehensive insights into the achievements and successes of the learner. There is evidence of a recent increase in the importance of portfolios and a proliferation of e-portfolios.
The portfolio method tends to be process-oriented, with much evidence that the selection process for portfolio building promotes self-assessment and focuses the candidate’s attention on quality criteria. This makes the portfolio a useful tool for formative and summative validation practices.
Portfolios can include evidence extracted through a combination of methods. It is argued that the kind of reflection and investigation associated with portfolio methods empowers people undergoing validation, which helps them obtain jobs or to choose appropriate further education. Portfolios can be developed to help disadvantaged people out of social exclusion or into employment by considering their specific characteristics.
Building a portfolio is a time-consuming exercise from the point of view of the applicant, but is popular; candidates have the possibility to show their competences in a flexible and authentic way, allowing for a combination of evidence. Assessment is often dependent on good written documentation of the individual’s skills; the method can prove difficult for some and should be supported by relevant information and guidance. The most serious risk in preparing portfolios is lack of focus that can occur when applicants prepare them alone or with little mediation from an appropriate support; counsellor aid and sufficient time for self-reflection are crucial to this method’s effectiveness and fairness. In the self-assessment against curriculum standards, guidance should be available to help explain the theoretical concepts and the transfer from theory to practice.
Some countries that provide national guidelines for validation, rather than prescribing validation methods, recommend a stage in the process which involves some form of assessment of the content of the portfolio by a third party (such as a jury) to ensure greater validity. Introducing third-party assessment does not solve all problems. Quality assurance processes should be in place to ensure consistency and transparency of third-party assessment and equality and fairness in the validation process for all candidates.
|