Methodology

Introduction

The short-term anticipation of skills trends and VET demand (STAS) grounds on a two-part econometric modelling approach, which involves both a VAR/VECM method that links vacancies and employment at the sectoral level (i.e. imposing some economic structure to the model and making use of additional information) and a univariate ARIMA approach (i.e. a purely statistical method) to model occupational employment directly. The VAR/VECM approach is applied at the sector level for EU-27 Member States (MSs) for which vacancy and employment data are both available; these series are then translated into occupation-level employment series. A third forecast is also produced, called “Naïve”, which simply follows a trimmed average of past quarter-on-quarter growth. Finally, the three models are averaged together to produce the baseline forecast.

The projections are elaborated by a Cedefop-owned software developed in Python and Stata languages during the execution of the contract No 2021-CED.375/AO/DSL/ILIVAN-PPANGEL/STAS/00.

1. Approach

The general approach is summarized in Figure 1. One pillar of the approach (depicted in the top half of the diagram) uses time-series ARIMA models of Member States’ occupational employment series at the 1-digit ISCO level. The most informative or best-performing model out of a family of models is selected to forecast the current and the next year. The form of each model (e.g. lags, differencing) varies according to the results of the model selection procedure implemented. Results by 1-digit occupations are converted into 2-digit occupations by using shares derived from the latest EU-LFS annual data available. The shares are computed annually, and then the average of the last two years [1] is applied. This is done to avoid excessively volatile shares.

Figure 1 – Approach outline.

The second pillar of the approach (in the lower half of Figure 2.1) involves VAR/VECM modelling to make use of estimated relationships between past employment and vacancies. This follows the methodology described in Lovaglio (2022). Sectoral vacancy and employment data from Eurostat at the 1-digit NACE Rev2 level by Member State are used. As vacancy series by occupation are available only for a limited set of countries, this part relies on sectoral data before applying a further step to derive occupation-level projections. The advantage of the VAR/VECM approach is the use of information the model can gather from changes in vacancies that, according to theory, are reasonably supposed to be early indicators for future employment changes. Not all pairs of series in all countries exhibit cointegration during test estimations, signifying a long-term relationship that justifies the application of a VECM model. An unrestricted VAR is estimated instead for cases of non-cointegration.

The conversion from sectoral results to occupations involves sector-by-2-digit occupation shares derived from an average of the last available periods of Eurostat annual data, similarly to the ARIMA case. This approach follows a procedure similar to the one used in the Cedefop Skills Forecast to link macroeconomic forecasts (disaggregated at sectoral level) to occupation and qualification outcomes.

2. Model selection procedure

The model selection procedure (to identify the ‘best’ model) takes the premise that a ‘good’ model should not violate important statistical or economic assumptions. Based on a number of viable models, the best-performing one is then chosen. To this end, a procedure (for both the ARIMA and VAR/VECM models) that applies a series of tests for statistical reliability and soundness was developed.

From the set of statistically sound models, those with superior out-of-sample forecast performance are preferred as the ‘best’ models. This performance is gauged by comparing forecast and outturn values for the most recent quarters in the data. The models are estimated on samples that exclude these quarters, and the prediction of the models evaluated according to root-mean-squared error (RMSE) over the chosen period. Both ARIMA and VAR/VECM models are subjected to this test to determine the best model from each part of the approach. Figure 2 summarises the model selection process, with more details provided in Section 4.

Figure 2 – Overview of model selection process.

First, both ARIMA and VAR/VECM are run to produce forecasts of employment by occupation. Then, a “Naïve” forecast is computed by applying to the latest historical data the average of the quarterly growth rates over the past eight quarters. Following the procedure suggested by Armstrong (2001), the three forecasts are then averaged together to produce the STAS baseline forecast. The rationale of the baseline forecast is to provide a conservative projection, whereby the inevitable uncertainties present in the different models will compensate each other. Indeed, the performance of the models is based on one set of recent observations, and it is not granted that the model that performed best over that period will perform equally well in the future.

Moreover, this approach allows a uniform treatment of all Member States. Figure 3 summarises the process to produce the baseline forecast. A comparison between the three models (VAR/VECM, ARIMA and Naïve) is made on the basis of the RMSE, Sign Success Ratio [2], as well as an assessment of how close the growth rates predicted by each model are to the growth rates from the latest AMECO forecast [3] on employment.

Figure 3 – Creation of the baseline forecast.

3. Data sources

The dataset for the analysis consists of quarterly sectoral employment and vacancies, and quarterly employment by occupation; all from Eurostat data. Table 1 summarises the data and includes an assessment of the potential causes of breaks, as well as the checks performed on the different series. Quarterly employment data by 1-digit occupation and 1-digit sector are downloaded from the Eurostat website, as well as annual employment by 2-digit occupation needed to fill the historical period.

Vacancy and employment data are examined for missing data and broad consistency over time before proceeding to estimation. During the model testing phase, visual inspection of the employment data confirmed that they were affected to various degree by breaks/changes owing to the COVID-19 pandemic and change in EU-LFS methodology from 2021 onwards. Data were also visually checked for possible non-stationarity and co-movements between employment and vacancies as a precursor to more formal statistical testing, as explained in Section 4.

The occupations matrices showing the share of 2-digit ISCO08 occupations by 1-digit NACE Rev2 sector and by 1-digit ISCO08 occupations are built from an ad-hoc extraction, upon request to Eurostat, of EU-LFS annual employment data. The occupations matrices are computed as an average of the shares of the years 2021 and 2022 (i.e. the last available years of data) and were applied throughout the forecast period, as explained in Section 4. Eurostat suppresses some values for reliability and confidentiality reasons, those values are filled with values from the previous period where possible, or by applying the country-level distribution by occupation to the missing part [4]. The occupation matrices are updated whenever a new annual LFS becomes available in order to keep the occupational structure of MS’s labour markets as close to the current situation as possible. A quarterly Eurostat ad-hoc extraction of employment by 2-digit occupation, which is used to perform comparisons across models and to produce the Naïve forecast, is also provided by Eurostat upon request.

Table 1 – Description of data sources.

The data constraints of the VAR/VECM approach lie in the availability of vacancy data by sector. Certain countries showed incomplete series for the number of vacancies but showed wider coverage for the number of occupied persons and the Job Vacancy Rate (JVR). In those cases, the JVR and the number of occupied people are used as a proxy to fill some of the missing vacancies data [5] and thus increase the coverage. The countries whose coverage increased substantially after this adjustment were Estonia, Spain, and Finland.

4. Implementation

While the approach uses two different families of models, both contain several similar aspects which help to ensure the statistical soundness of the approach, following the step-by-step procedure shown in Figure 4. Details on the different steps of the procedure are presented in the following sub-sections.

Figure 4 – Step-by-step procedure.

4.1. De-seasonalising, testing stationarity, testing cointegration.

The short-term anticipation model is not intended to forecast seasonal effects but annual employment outcomes independent of seasonal effects. Consistently, the first step is de-seasonalising each series before estimation. To de-seasonalise the series, the Eurostat recommended approach is applied [6]. The approach uses the X-13 algorithm, which integrates the non-parametric approach of the U.S. Census office [7] with the timeseries approach developed by the Bank of Spain (Gomez & Maravall, 1996). A log transformation is then applied to the de-seasonalised data to reduce skewness and stabilise variability.

There are several tests for stationarity in literature. The most commonly known are the Dickey-Fuller test and the augmented Dickey-Fuller test. However, the disadvantage of the Dickey-Fuller tests is their lack of statistical power, with a tendency to not reject the null hypothesis of a unit root in the face of a near-unit root process. This is exacerbated in the presence of structural breaks, even if the series is actually stationary. This issue is important in the STAS case given the potential breaks introduced by the COVID-19 pandemic and the 2021 change in the EU-LFS methodology. In developing the forecasting procedures, it was decided to follow the literature in using the Perron test (Perron, 1989), which is a unit root test that is used to assess whether a series exhibits a unit root behaviour in the presence of a structural break. It is possible to choose different specifications depending on the assumed/likely data generating process. In the STAS case the null hypothesis testes argued that the series is a unit root process with a onetime shift in level and a change in drift, versus the null of a trend-stationary process with a permanent change in the intercept and in the slope of the trend. During the validation of the STAS model 2020q2 was chosen as the break date for the pandemic. Another possibility would have been to consider 2021q1 as the break date since it coincides with the changes in the EU-LFS methodology, but it was too close to the end of the estimation period to allow for a robust testing. Variables who were found to exhibit a unit root according to the Perron test were differenced, whereas variables found to be stationary were maintained in log-levels (in the case of the VAR both variables are differenced if either one of them is found to have a unit root).

The cointegration test is specific to the VAR/VECM family of models. Two series are cointegrated if their relationship (a linear combination of the variables) is stationary. Should vacancies and employment exhibit this property, the model should be specified as a VECM model to model the adjustment mechanism back to a long-run trend. Using the Gregory-Hansen test for cointegration in the presence of structural breaks allows us to test for cointegration with breaks (i.e., with a similar rationale for the Perron test for unit roots). The test was proposed by Gregory and Hansen (1996) to allow for the implementation of specifications combining breaks in levels, trends, and slopes. In the case of the STAS, the null hypothesis of no cointegration against the alternative of cointegration with a single shift at an unknown time was tested.

4.2. ARIMA and VAR/VECM model selection

The model selection procedure starts by estimating a general model which includes four lags, a time trend, a “pandemic dummy” equal to 1 after 2020q2 and an EU-LFS break dummy equal to 1 after 2021q1 if the variable is in levels, and equal to 1 only in 2021q1 if the variable is in differences. The dummy variables are maintained in all the models tested so to check whether it is appropriate to include a time trend. More specifically, the values of the Bayesian Information Criterion (BIC) of the models estimated with or without a time trend are considered, and the model with the lowest value is chosen. Models are then tested with all the possible combinations of lags (up to a maximum of four) the ones exhibiting serial correlation are discarded. In the VAR context, the test used to detect serial correlation is a Lagrange Multiplier test, as presented in Johansen (1995). The test statistic is based on the variance-covariance matrix of the residuals from the VAR equations. The null is that there is no serial correlation at a given lag order. We test for serial correlation up to four lags (as is typical for quarterly data) and discard any model that exhibits serial correlation.

In the ARIMA model specification, an ordering of statistically well-performing models is made, selecting good performance by the Bayesian-Information criterion (BIC). All models include the dummies identifying the COVID period as well as the EU-LFS classification shift; the approach is then an ARIMAX as there are deterministic/exogenous components in the models, (i.e., the dummy variables). A set of models preselected on these criteria are estimated and evaluated in their forecasting performance. The ARIMA projections by 1-digit occupation are then converted into 2-digit occupation by using the occupation matrix described in Section 3.

In the VAR/VECM specification, models are further distinguished considering whether:

1. vacancies are significant in the employment equation (short-term lags; and the long-run equation in the presence of cointegration, such that vacancies are indeed a source of useful information when projecting employment; and
2. at least one lag (regardless of whether it relates to employment or vacancies) is significant in the vacancies equation (such that there is at least some statistical process to project vacancies into the future).

The same tests are applied to VAR and VECM. The only difference is that the estimation of the error correction term in the VECM is based on the cointegrating equation. All the models in which vacancies are significant in the employment equation, as well as lags in the vacancies, are retained. The models that do not satisfy the above criteria are retained only if they rank among the best eight models in terms of BIC. As a matter of fact, there are 15 possible lag combinations with a maximum of four lags, a model not satisfying the vacancies significancy criteria is retained only if it belongs to the top half models in terms of BIC.

The sectoral forecasts produced by the VAR/VECM models are translated into occupations by using the occupation matrix described in Section 3. More specifically, annual forecasts of employment by sector are multiplied by the shares of occupations within sectors from the latest available EU-LFS data. The shares are kept constant through the forecast period.

4.3. Comparing forecast performance

Up to eight models for each of ARIMA and VAR are retained following the above procedures and tests. The final assessment to select the best ARIMA and VAR models is made on the basis of the models’ out-of-sample forecasting performance, which is tested by removing (up to) the last eight quarters of data from the sample and then estimating the candidate model specifications on the restricted sample. The models’ predictions (up to eight quarters dynamic forecasts) are then compared to the outturn values, with predictive accuracy determined according to RMSE [8]. The ARIMA model with the lowest RMSE and the VAR model with the lowest RMSE are the two models then taken forward for the final forecast, which involves averaging the results along with a naïve forecast (see Section 5).

While not relevant to model selection, the tool also carries out tests of consistency with the AMECO forecast on employment, to gauge the extent to which the STAS results agree with it.

5. Final forecast

The selected model provides the forecast of employment (up to) eight quarters ahead from the last period of data. Annual forecasts are then obtained by averaging the quarterly forecasts each year. Alongside the two econometrically estimated forecasts described above (ARIMA and VAR), a third forecast enters the final average. This third forecast is a “naïve” forecast constructed as follows, for each country-occupation series for which there are data (where there are no data, there is no forecast):

• Calculate quarter-on-quarter growth rates for the entire series of available data.
• Trim the growth rates by removing the largest 10% and smallest 10% by value, i.e. the 20% most extreme (most positive/negative) values.
• Calculate the arithmetic mean of the remaining growth rates.
• Apply (compound) this growth rate forward from the last period of history to generate the quarterly forecast (which is then averaged to annual frequency in the same way as for the ARIMA and VAR forecasts).

The approach above was preferred to other candidates because it generates forecasts more consistent with longer-term historical trends. In contrast, an approach that only considers recent historical growth rates (e.g. for the last eight quarters) proved prone to large increases or decreases owing to the COVID-19 pandemic period and/or large departures from trend. The other alternative considered, regression on a time trend, fared comparatively poorly in the face of series with no clear single trend and also exhibited notable shifts in the level when transitioning from history to forecast. Various options to counter these effects were considered including, respectively: reduced sample sizes; and either an intercept correction or a decaying adjustment term to return to trend. All were rejected because they increased the complexity of an intentionally simple (i.e. naïve) forecast approach.

The only exceptions to the approach above are for series for which there are limited, but still at least some, quarterly data. In these cases, the quarterly historical data each year are constructed as having the same values as for the annual series, i.e. four quarters each year, with the same annual figure in each case. In these instances, the naïve forecast simply holds the value constant after the last period of history.

5.1. Alignment with an external forecast

The STAS baseline can be eventually aligned with an external forecast. The default solution is aligning the baseline to total employment growth by country provided by the latest DG-ECFIN AMECO projections. Following this procedure, the STAS baseline gains soundness and consistency with projections formulated on the basis of a different data collection (the AMECO uses national accounts data, while the employment of the STAS forecast is defined using the EU-LFS employment definition), bringing added value to the AMECO forecast by providing detailed occupational forecast by MSs for 2023-24 period.

References

Armstrong, J.S. (2001). Combining Forecasts. Kluwer.

Gomez, V. & Maravall, A. (1996). Programs SEATS and TRAMO: Instructions for the User. Bank of Spain.

Gregory, A. W. & Hansen, B. E. (1996). Residual-based tests for cointegration in models with regime shifts. Journal of Econometrics, 70(1), 99-12.

Johansen, S. (1995). Likelihood-Based Inference in Cointegrated Vector Autoregressive Models. Oxford: Oxford University Press.

Lovaglio, P.G. (2022). “Do job vacancies variations anticipate employment variations by sector? Some preliminary evidence from Italy.” Labour 36 (1), 71-93.

[1] When sufficient time has passed from the COVID-19 pandemic and the 2021 change in EU-LFS, the average could be potentially taken over more years.

[2] Sign Success Ratio measures how often a given model correctly predict whether the change in a quarter is positive or negative.

[3] AMECO is the European Commission’s database of annual macroeconomic data. Maintained by DG ECFIN, the database reports both historical macroeconomic series and the Commission’s latest forecasts. Comparisons and/or calibration of employment levels between the STAS and AMECO may thus have value as both a diagnostic and forecasting device.

[4] Missing series were found mainly in small sectors such as Household as employers, Extraterritorial organisations and NRP, therefore the results are not affected significantly by this approximation.

[5] The JVR is computed as 𝐽𝑉𝑅 = 𝑣𝑎𝑐𝑎𝑛𝑐𝑖𝑒𝑠 ⁄ (𝑣𝑎𝑐𝑎𝑛𝑐𝑖𝑒𝑠 + 𝑜𝑐𝑐𝑢𝑝𝑖𝑒𝑑). It is possible to recover vacancies with some simple algebraic manipulation if the JVR and the occupied are known.

[6] https://ec.europa.eu/eurostat/web/research-methodology/seasonal-adjustment

[7] https://www.census.gov/data/software/x13as.html

[8] Other criteria were trialled during tool development, but RMSE ended up as the preferred metric. More detailed diagnostics in the tool’s output include the sign success ratio (SSR), which measures the incidence of forecast quarters moving in the same direction as in the outturn. The SSR was ultimately rejected as it proved more likely to have models tied for first place.