Malta College of Arts, Science and Technology (MCAST) internal quality review

MCAST internal quality reviews follow NCFHE's internal QA standards and the National qualifications framework (NQF). MCAST also participates in all data collection processes carried out by NFCHE.

The main inputs contributing to the creation, maintenance and responsiveness of the college's QA system are the NCFHE National QA framework-internal QA standards: these are influenced by the European standards and guidelines and the 2009 recommendation on the establishment of European quality assurance framework for vocational education and training (EQAVET) which is replaced by the 2020 recommendation on vocational education and training (VET) for sustainable competitiveness, social fairness and resilience). Several measures have been developed and implemented by MCAST over recent years as part of an overarching college QA in order to gauge and assure quality. These measures include the documentation of processes in the form of standard operating procedures and process maps (inputs, process and outputs), internal verification and lead internal verification of assessment tools and decisions, internal audits of the educational and operational processes within and across MCAST providers, and the way in which these processes impact the quality of the service delivered to MCAST clients, the students, and lecturer appraisal; they are in line with National QA framework requirements stipulated by the NCFHE.

1. Documentation

As a self-accrediting institution, the college is required to comply with the various academic and administrative national and European legislative and regulatory requirements. Over the years, MCAST has developed, a comprehensive set of academic procedures that is now an integral part of the college's quality management system.

Complementing the above, a resolution by the board of governors in 2019 identified the need for the development and dissemination of the manual of administrative procedures. The manual includes updates of the set of policies and procedures that were originally drafted in 2004, as well as a number of, new additional chapters. The manual includes 14 chapters dealing with individual administrative aspects, each chapter comprising policies, procedures and forms, so the college has an approved integrated quality management system in place.

The QA office is responsible for ensuring that, together with the deputy principals, institute directors and heads of departments, the academic and administrative processes are disseminated, regularly reviewed and updated. The deputy principals, together with the institute directors and heads of department who are the owners of these dynamic processes and procedures, also have the responsibility of ensuring that their staff are informed and compliant with the content of QA manuals.

1. Internal verification and lead internal verification

The findings from various internal audits carried out by the QA department have indicated that a number of quality practices were predominant; several observations showcased the strengths of the internal verification process underpinning its credibility and validity in assuring the quality of programmes delivered through its assessment tools and the assessment decisions taken. Several factors deemed necessary to the vocational essence of the various programmes offered by MCAST, and the robustness of the assessment/internal verification process feeding into internal and external audits, were also identified.

Overview of internal verifiers' work, crucial to assessment validity and innovation, is a theme that recurs across a wide range of literature. The discourse also inclines to such roles resolving the problems with consistency and enabling innovation. All findings consolidate the analysis of the roles of assessors/internal verifiers and lead internal verifiers, in which their maximisation is strongly conveyed, as manifested in a number of good practices identified by the trends observed in the data gathered. Nonetheless, this analytical exercise has also highlighted some factors that would benefit from attention and improvement in assessment skills and competences, particularly as these areas are considered crucial to the vocational education and training being delivered by MCAST.

Incorporation of the recommendations advocated by the findings would reflect the discourse that stems from the premise that internal verification yields greatest benefits when integrated within an explicit quality management framework.

1. Internal Audits

The main objective of such audits is to identify good practices within the providing institutions and then create a vehicle by which these are shared with sister institutions within MCAST. A corresponding and equally important goal of this exercise is to delve into areas which present risks to the smooth running of the processes and, hence, would need improvement, in turn necessitating the delineation of possible strategies for corrective action.

The scope of the engagement is to reinforce the strengths of the legacy accumulated over the past 20 years, since MCAST's inception, identify the opportunities, anticipate the threats and address the weaknesses through an open and transparent encounter with personnel, an overview of related documentation and the observation of the interactions of those involved.

The first challenge was how to go about these investigations, establishing good practices and assessing the risks. It was decided that the way forward was to have tools in place, which would assist in the identification, analysis and evaluation of the data, thereby building a solid foundation on which to formulate strategies for action in the short, medium and long term.

Secondary research applied to this exercise consisted of a comparative study of the quality frameworks applied in a number of other countries within Europe, Central Asia and Australia. This led to the creation of a QA protocol, mapped to MCAST documentation, as well as NCFHE and European QA indicators, which addresses the education and operational processes within a providing institution. This instrument targets the following process areas:

• teaching and learning;
• assessment and support services where the education processes are concerned;
• admissions and advancement;
• management strategy;
• measurement and monitoring for the operational processes.

Each process area is further sub-divided into sections. Taking teaching and learning as an example, this process area was approached through staff, programme development, delivery and review, logistics, resources and sub-contracting/external programme delivery/examination. This protocol, which can be used as a tool for internal audit and self-evaluation, separate from external audit, stipulates the standards expected within each of the education and operational processes, offers prompts as a gauge of the information required and lists the possible sources of evidence.

The methodology employed during the audit engagement is qualitative in nature, consisting of the gathering of information through initial individual meetings with the directors and deputy directors of the providing institution, followed by individual sessions with academic and administrative staff directly involved with the education and operational processes related to programme planning, design, implementation, delivery and review, as well as student progression, and focus groups with students; these look at the particular programme they were following, their history, if any, at MCAST, and their future aspirations. The exigencies of triangulation, whereby various groups within an organisation corroborate, complement, oppose or supplement the findings, are observed to the full.

The findings are sifted with the purpose of succinctly extrapolating general observations classified as those directly relevant to the provider, examples of good practice which may be disseminated depending on their applicability, and commonalities of concern within and across the providing institutions. This latter exercise is fundamental in homing in on the pertinent strengths, weaknesses, opportunities and threats to the providing institutions and MCAST as a whole.

The findings are then allotted a unique identifying code and entered into a risk assessment form according to category, in which they are quantified using three metrics: severity, frequency and the rate of detectability. Taking these metrics into account, a numeric scale of 1-5 was developed by which to measure the findings and their impact, depicted verbally in risk assessment scales. On the outcome of the latter, each identified finding listed in the risk assessment form is colour-coded, reflecting the incidence of high, moderate or low risk.

Some risks are not adversely affected by detectability while in other cases the risk is augmented, thereby aggravating the situation, depending on the risk factor for detectability. Risks are adjusted taking the worst-case scenario from either frequency or detectability to depict the influence of the latter variable on the risk. The risk assessment form is also adjusted accordingly to depict these results.

In this way, the riskier zones within and across providing institutions in MCAST are identified, assessed and evaluated based on both a qualitative and a quantitative analysis of the findings.

Applying the bottom-up approach to policy-making, all the findings are then the subject of discussion at top management level. Recommendations for implementation to disseminate good practice and address the concerns are then produced, primarily identifying the areas in urgent need of improvement and outlining the appropriate strategy for the necessary action to be taken to assure quality.

1. Appraisal

A lecturer-teaching appraisal exercise is also integral to the system. Such an appraisal involves:

• an observation visit whereby a member of the QA team will observe a lesson and give feedback to be provided on the teaching methods applied;
• reviewing teaching material being used;
• evaluation of interpersonal skills and attitudes.

MCAST's quality management framework serves to promote quality in relation to all matters affecting the educational and operational aspects of student-centred learning and programme delivery, ensuring the validity and transferability of MCAST qualifications by adhering to the key indicators of a robust system: manageability, sustainability, satisfaction, and consistency, to inspire and aspire to an innate quality culture.

1. Cyclical review

Internal cyclical review of programmes is covered by MCAST Doc 028 programme cyclical review policy and procedure in line with standard 10 of the internal QA standards in further and higher education issued by the NCFHE.

It is college policy that:

• accredited programmes are to be reviewed regularly to ensure that the programme objectives are relevant and responsive to the needs of the various internal and external stakeholders;
• review of programmes takes note of the feedback from internal and external stakeholders;
• the calendar for cyclical programme reviews is followed;
• the cyclical reviews are conducted in compliance with the requirements of the NCFHE QA framework and internal QA Standards.

Cyclical review of the programme is to be conducted:

• at the end of the first programme cycle;
• at a frequency of at least once every three years;
• or as the need arises (unscheduled).

Institute directors are responsible for the cyclical review of programmes hosted by their respective institutes and centres.

The cyclical review process needs to take note of feedback from internal and external stakeholders that are received during the previous three years of programme delivery. The stakeholders include:

• individual students and student bodies;
• industry / business / NGO stakeholders (when major changes are proposed);
• college staff, institute directors, coordinators and lecturers, quality assessors;
• internal and external academic reports/reviews, student results, dropouts, complaints, appeals.

Towards the end of each calendar year (November), a list of programmes that are due for the three-year cyclical review is forwarded to the institute directors who, based on the feedback from internal and external stakeholders, determine which programmes are eligible for:

• cyclical review (structural, minor, major, rewrite);
• withdrawal;
• no changes.

A Cyclical review proposal report is compiled for each of the programmes due for cyclical review, typically covering the following areas;

• summary of internal and external feedback;
• rationale of changes being proposed;
• list of units that require cyclical review;
• the extent of review required (minor, major, rewrite, withdrawal, no changes).

Monthly (or as required) status reports ensuring timely approval and release of MCAST programmes undergoing cyclical review are issued by the department responsible for MCAST's education and training programmes.