A Robust Framework for Renewable Energy Policy Evaluation Using MCDA and Compromise Ranking with Stochastic Weight Identification

MCDA methods play a crucial role in decision-making processes, especially when multiple criteria must be considered [1]. Their application allows for a structured and integrated evaluation of various aspects of a decision, which is particularly important in complex decision problems. By leveraging expert knowledge, these methods can provide reliable assessments and determine the significance of selected criteria [2]. Over the years, many new approaches have emerged for evaluating decision alternatives and their corresponding criteria.

One such approach is the SPOTIS method [3], which enables an expert to define an Expected Solution Point (ESP), modeling the decision-making grid. This method provides a clear reference point against which different alternatives can be evaluated, offering a way to balance diverse criteria by focusing on a preferred solution space. Another innovative approach is RANCOM [4], which facilitates the evaluation of criteria by comparing them in pairs using a three-level scale. This pairwise comparison helps to rank the criteria based on their relative importance, ensuring more nuanced decision-making. Additionally, the Best-Worst Method (BWM) [5] has gained popularity. The decision-maker identifies the most and least important criteria in this approach, and weights are derived based on these extremes. This method simplifies the weighting process and reduces inconsistencies arising from subjective judgments while still providing accurate and balanced results.

These advancements in MCDA provide decision-makers with a wider range of tools that are flexible and adaptable to various decision contexts. They enable experts to incorporate both qualitative and quantitative data, making the decision-making process more robust, especially in fields like resource management [6], [7], energy policy [8], [9], and environmental sustainability [10], [11]. The continual development of these methods ensures that decision-makers can address increasingly complex challenges with greater precision and insight.

One of the most critical areas where MCDA is widely applied is the energy sector, particularly in the context of sustainable development. Managing energy resources sustainably is a significant challenge for countries, as energy policy decisions must balance conflicting criteria like economic feasibility, environmental impact, and social factors. Determining a country's sustainability in this area is essential for shaping policies and evaluating effectiveness. Ribeiro et al. [12] demonstrated MCDA's use in evaluating electricity production scenarios in Portugal, ranking them based on 13 economic, environmental, and social criteria. The study highlighted the tension between cost-effective solutions like coal and more sustainable but expensive options like renewable energy, with experts divided on energy costs versus long-term sustainability. Historically, the conflict between economic growth and environmental goals spurred MCDA's development in the energy sector. Diakoulaki et al. [13] noted that energy market liberalization added complexity, requiring planners to integrate multi-stakeholder perspectives and uncertainties. This broader application of MCDA helps address challenges like competition, sustainability, and renewable energy integration. Manoj et al. [14] used multiple MCDA techniques, including MOORA, TOPSIS, and VIKOR, to select optimal hybrid renewable energy systems (HRES) based on cost, emissions, and renewability. Their study highlighted MCDA's ability to balance economic and sustainability goals in energy planning.

There are many publications also in the literature that attempt to assess the energy sustainability of individual countries. Mainali et al. [15] used the energy sustainability index to assess the energy sustainability of countries such as China, India, South Africa, Sri Lanka, Bangladesh, and Ghana. Phillis et al. [16] used the PROMETHEE approach to assess 43 European countries' energy sustainability. Sahabuddin and Khan [17] used seven different MCDA approaches to assess the sustainability of the electricity generation sector. Siksnelyte-Butkiene et al. [18] developed a methodology for a comparative assessment of selected Northern European countries using the TOPSIS approach. In turn, Kouikoglou et al. [19] created a methodology for assessing energy sustainability at the national level based on the fuzzy analytical model SAFE and the TOPSIS multi-criteria method. Various multi-criteria methods are often used in these studies, leading to different rankings depending on the method used. However, due to the various approaches, assessing which ranking best reflects the actual situation is complex. In addition, assessing sustainability requires the consideration of many criteria, the relevance of which is often challenging to determine.

In the publication by Więckowski et al. [20], a framework for sensitivity analysis related to the assessment of renewable energy management in EU countries was proposed. The authors focused on identifying the significance of criteria and the stability of rankings using the Monte Carlo algorithm. Four MCDA methods were employed in the ranking comparisons, but the study did not address how to construct the most stable ranking from the results obtained. Therefore, this paper focuses on creating a robust compromise ranking (using the Copeland method [21]) that will be as close as possible to the rankings from the four methods. Additionally, we will determine the importance of the criteria expressed through weights derived from the Stochastic Identification of Weights (SITW) approach [22]. This will provide insight into which criteria were the most significant in the four submodels presented in the publication by Więckowski et al. [20]. By applying these approaches, we aim to enhance the understanding of the stability and reliability of the renewable energy management rankings for EU countries while offering a clearer perspective on the critical factors influencing these rankings. This study will provide a more nuanced interpretation of the original analysis and contribute to developing decision-making processes in renewable energy policy.

The sections of the paper are divided as follows: Section 2 describes the methodology, which proposes a framework for the study related to the construction of a consensus ranking and the identification of the relevance of criteria. Section 3 presents the results of the study for evaluating the energy policies of European Union countries. Section 4 discusses the results and interprets them in the context of existing literature and energy policies. Section 5 summarizes the main conclusions of the research.

In this study, we develop a comprehensive framework for the stable ranking and assessment of criterion importance in the context of renewable energy policy management across European Union (EU) countries. The data used in this research is derived from the work of Więckowski et al. [20], which presents four submodels, each with a distinct decision matrix. These matrices encompass 16 alternatives (EU countries) and 9 criteria, which measure various aspects of energy consumption and generation, including coal, fossil fuels, gas, hydropower, nuclear energy, oil, renewable sources, solar, and wind power. Each criterion is evaluated per capita or as a percentage share of total energy consumption or generation.

In the original study, Więckowski et al. [20] employed four MCDA methods—MARCOS, TOPSIS, SPOTIS, and MOORA—on these decision matrices. However, the authors focused exclusively on the rankings generated using the MARCOS method, without considering an integrated approach that accounts for all four methods. To address this limitation, the present study develops a compromise ranking using Copeland’s method, synthesizing the rankings produced by all four MCDA techniques for each submodel. Consequently, this approach will yield four compromise rankings, one corresponding to each submodel.

In addition, we use the SITW method to determine the importance of each criterion within the submodels. The SITW approach produces more stable and robust weight values for each criterion, reflecting their relative importance across the four submodels. This step is essential to ensure that the criteria weights are not solely influenced by a single MCDA method but are based on a more holistic assessment.

The final analysis will involve two key components: (1) the generation of four compromise rankings (one per submodel) based on the integration of the four MCDA methods using Copeland’s method, and (2) the derivation of four weight vectors, one for each submodel, using the SITW method. To evaluate the reliability and consistency of these outcomes, we will conduct a comparative analysis using ranking similarity coefficients (such as the weighted Spearman and Weighted Similarity correlation coefficients) and weight vector similarity measures. This analysis will provide insight into the degree of concordance between the rankings generated by different methods and identify the most critical criteria across the submodels.

By integrating multiple MCDA methods and employing advanced techniques for weight identification and ranking comparison, this methodology provides a more robust and comprehensive evaluation of renewable energy policies across EU countries. This approach also facilitates the assessment of ranking stability, which is essential for formulating reliable, data-driven policy recommendations in sustainable energy management. As a result, the framework illustrated in Figure 1 has been developed. The following sections detail the algorithms and mechanisms that constitute the foundation of this framework.

This section outlines the Copeland algorithm, a widely used method for creating compromise rankings in multi-criteria decision-making through pairwise comparisons. The Copeland method evaluates alternatives against one another, assigning points based on the number of "wins" (better performance) or "losses" (worse performance) in these comparisons. Each alternative's net score is calculated by subtracting losses from wins, forming the basis for the final ranking. This approach mitigates biases inherent in single-ranking methods, providing a balanced and robust compromise ranking. It effectively synthesizes results from multiple decision-making models, making it a practical tool for decision-makers in various fields, including energy policy evaluation. The Copeland approach consists of the following steps:

Step 1. Calculation of the Wins Score. The initial step involves calculating the wins score for each alternative. The wins score, denoted as $W_i $ for alternative $ i $, is determined by summing the ranks assigned to that alternative based on various evaluation methods. This can be expressed mathematically as:

where, $r_{ij} $ represents the rank of alternative $i $ in method $ j$ and $ m$ is the total number of evaluation methods.

Step 2. Calculation of the Losses Score. Next, the losses score for each alternative is calculated by subtracting the wins score from the majority wins score $ M $. This yields the losses score $ L_i $ for each alternative $ i$:

where, \( M \) is the highest possible wins score (the score for the alternative that wins in all comparisons).

Step 3. Determination of the Final Scores and Ranking. The final score \( F_i \) for each alternative is obtained by taking the difference between the wins score and the losses score:

The alternative with the highest final score \( F_i \) is then identified as the best option.

This section introduces the SITW method, which utilizes a stochastic optimization technique known as Particle Swarm Optimization (PSO). Originally proposed by Kizielewicz and Sałabun [22], the SITW method addresses the issue of insufficient input data regarding weights, which are crucial for various MCDA methodologies. By employing stochastic optimization techniques, the SITW method enables the determination of optimal weights, thereby enhancing the re-identification of MCDA models. This study focuses specifically on the PSO approach, outlining the SITW method through the following steps:

Step 1: Select a Dataset

The dataset should contain:

· \( C = [C_1, C_2, \dots, C_n] \): a matrix of criteria, where each column represents the criteria vector for one decision option,

· \( T = [T_1, T_2, \dots, T_n] \): a vector of criteria types, which indicates whether a criterion should be maximized or minimized,

· \( R = [R_1, R_2, \dots, R_n] \): a vector of actual rankings for the decision options.

Step 2: PSO

We apply the PSO algorithm, where the position \( X_i \) and velocity \( V_i \) of each particle \( i \) in the swarm are updated in the search space to find the optimal weight vector \( w \). The updates are governed by the following formulas:

where, \( V_i(t) \) is the velocity of particle \( i \) at iteration \( t \), \( X_i(t) \) is the position of particle \( i \) at iteration \( t \), \( P_i \) is the personal best position of particle \( i \), \( G \) is the global best position, \( w \) is the inertia factor, \( c_1 \) and \( c_2 \) are acceleration coefficients for the personal and global best positions, respectively, \( r_1 \) and \( r_2 \) are random numbers drawn from the range \( [0, 1] \).

Step 3: Fitness Function

The fitness function is designed to maximize the weighted Spearman rank correlation coefficient \( r_W \), which measures the agreement between the model's ranking and the actual ranking \( R \). The formula for \( r_W \) is given by:

where, \( x_i \) is the rank of the \( i \)-th option according to the model (determined by the function \( \text{base}(C, w, T) \), in this case is TOPSIS method), \( y_i \) is the actual rank of the \( i \)-th option (from the vector \( R \)), \( n \) is the number of decision options, \( r_W \) is the value of the weighted Spearman rank correlation coefficient.

Additionally, three constraints are imposed on the weight vector \( w \):

1. The sum of the weights must equal 1:

\[ \sum_{i=1}^{n} w_i = 1 \]

2. Each weight \( w_i \) must lie within the range \( [0, 1] \):

\[ 0 \leq w_i \leq 1 \quad \forall i \in \{1, 2, \dots, n\} \]

These constraints ensure that the weights represent a valid distribution across the criteria, are positive, and do not exceed 1.

Step 4: Optimization Process

The goal of the optimization process is to find the weight vector \( w \) that maximizes \( r_w \), ensuring that the model's ranking aligns as closely as possible with the actual ranking. The optimal weight vector \( G \) is determined by the PSO algorithm as follows:

By following these steps, the SITW method utilizing PSO efficiently determines the optimal criteria weights, maximizing the fit between the MCDA model and the actual ranking data while ensuring the sum of the weights is equal to 1, and all weights are within the interval \([0, 1]\).

As part of our analysis, we focused on using submodel data from the publication of Więckowski et al. [20], which provided a variety of rankings and decision matrices. These data provided the foundation for evaluating various alternatives in the context of European energy policy. We evaluated this using four different multi-criteria decision-making methods: MARCOS, TOPSIS, SPOTIS, and MOORA. Each of these methods has a different approach to analysis, allowing us to obtain differentiated results, which were then used to develop compromise rankings. These results represent an important step toward identifying the most effective energy strategies for EU countries.

The compromise rankings, shown in Table 1, result from using different methods to evaluate alternatives. Each table shows the countries' positions for each method, allowing an in-depth analysis of their consistency and differences. For example, the ranking order was broadly consistent for the MARCOS, TOPSIS, and SPOTIS methods, suggesting a degree of stability in the results. In the compromise ranking, it was observed that specific alternatives, such as $A_1$, $A_3$, and $A_{16}$, received consistently high rankings, indicating their dominance in the analysis.

The results of the ranking correlation matrices for the four submodels presented in subgraphs (a), (b), (c) and (d) of Figure 2, based on the weighted Spearman coefficient, underscore the key role of the compromise ranking constructed using the Copeland method. Integrating results from different multi-criteria methods (MARCOS, TOPSIS, SPOTIS, MOORA) is particularly important when evaluating complex problems, such as selecting optimal strategies and technologies in the renewable energy sources (RES) sector. In the context of energy policy management in EU countries, evaluation criteria include a wide range of variables, such as cost-effectiveness, environmental impact, and long-term stability of energy supply. The Copeland method integrates different evaluation approaches and is crucial in achieving a balanced and stable decision-making outcome.

An analysis of the correlation matrix shows that the compromise ranking is consistent with the results of the other methods in all submodels, demonstrating its effectiveness. In submodel $1_1$, the correlations between the compromise ranking and the other methods range from 0.99 to 1.00, showing almost complete agreement. Such results are significant in the RES sector, where complex technologies and diverse investment strategies require accurate evaluation based on multiple criteria. With its ability to integrate results from different methods, the compromise ranking ensures investment decisions based on broad consensus, which minimizes risk and maximizes economic and environmental benefits.

Similar concordance was observed in submodels $1_2$ and $2_1$, where correlations range from 0.99 to 1.00. This indicates a strong link between the compromise ranking and the other methods, confirming the stability of the evaluations of alternatives such as the RES technologies analyzed in these submodels. In submodel $2_2$, where correlations are slightly lower (0.97-0.98), the compromise ranking still reflects the overall pattern of results, which is crucial for making decisions related to energy transition and achieving sustainability goals.

After the compromise rankings for each submodel were obtained and analyzed, we determined the significance of the criteria, i.e., the weights for each submodel. Criterion relevance reflects the impact of each criterion on the final ranking score, which is crucial for assessing which criteria are most relevant in the context of renewable energy management in EU countries. The SITW approach was used to determine criteria weights in submodels. SITW allows optimal weights to be assigned to criteria based on a decision matrix and a compromise ranking. In this approach, the goal was to select weights such that the baseline method (in this case, TOPSIS) with fixed weights generates a ranking as close as possible to the compromise ranking, maximizing the value of the weighted Spearman correlation between the rankings. The weight optimization process used the PSO technique, an efficient stochastic method used to solve optimization problems. For each submodel, a distinct weighting procedure was carried out, where the decision matrix and compromise ranking served as input to the SITW algorithm. The result of this process is the sets of weights, which are presented in Table 2. This table shows the determined weights for the nine criteria for the four submodels, making it possible to identify which criteria have the most significant impact on scores in a given submodel.

After analyzing the rankings obtained from the weights determined by the SITW method, a detailed comparison was made with the compromise rankings to assess the level of agreement between these approaches. The results, illustrated in subgraphs (a), (b), (c) and (d) of Figure 3, reveal that the rankings derived from SITW weights align closely with the compromise rankings, particularly in the upper tier of the rankings. The high values of the weighted similarity coefficient ($WS$) and the weighted Spearman coefficient ($r_w$) indicate that the discrepancies between the two ranking systems are minimal, with negligible influence on the final ranking outcome.

In the case of submodel $1_1$, the analysis shows perfect alignment between the SITW-derived and compromise rankings, with both $r_w$ and $WS$ achieving values of 1.00000, indicating a complete match. This demonstrates that the SITW approach successfully captures the priority structure of the compromise ranking without any deviation. Similarly, in submodels $1_2$ and $2_1$, the SITW rankings exhibit very high agreement with the compromise rankings, with $r_w$ values of 0.99792 and 0.99879, respectively. This high concordance suggests that differences between the two ranking systems are minimal and are primarily confined to the lower-ranked alternatives, which have less impact on the overall decision-making process.

For submodel $2_2$, although the $r_w$ value is slightly lower at 0.98227, the agreement remains strong, with $WS = 0.99952$, further reinforcing the conclusion that the SITW method produces rankings that closely mirror the compromise ranking. The small deviations that occur are isolated to the lower positions in the ranking, which typically have less strategic importance compared to the top-ranked alternatives. These results suggest that the differences, while present, do not meaningfully alter the conclusions drawn from the rankings.

Overall, the analysis demonstrates that the SITW method is highly effective in reproducing the compromise rankings, especially for the most crucial alternatives at the top of the list. The strong agreement across all submodels, particularly in the top positions, confirms that the SITW approach provides a reliable method for ranking re-identification, ensuring that the key priorities of the decision-making process remain intact. The minor variations that arise in the lower ranks have little effect on the overall decision outcome, making SITW a robust tool for MCDA in complex contexts such as renewable energy management.

The weights obtained for each submodel show similarities and differences, especially in the context of specific criteria, as shown in Figure 4. The weights of $w_5$, $w_7$, $w_8$, and $w_9$ are relatively similar across all submodels, suggesting that these criteria have similar weights regardless of submodel specificity. For example, $w_5$ oscillates from 0.11097 to 0.12732, and $w_9$ from 0.11614 to 0.12751, indicating that the weight of these criteria is stable across scenarios. In contrast, more significant weight differences can be seen for the criteria $w_2$, $w_3$, and $w_7$. In particular, criterion $w_2$ in submodels $1_2$ (0.14719) and $2_2$ (0.18664) is significantly higher compared to submodels $1_1$ (0.07713) and $2_1$ (0.13215), which may indicate different priorities for this criterion in these submodels. Also, the weight of $w_3$ in submodel $2_1$ (0.02834) is significantly lower than in the other submodels.

Analysis of the weight similarity matrix using the weight similarity coefficient ($WSC_2$) [23] shows that the submodels generally have a high correspondence of weights, as evidenced by similarity coefficients in the range of 0.80-1.00. Meanwhile, subgraph (b) of Figure 4 shows the weight similarity matrix itself. The highest concordance, at 0.87, is observed between submodels $1_1$ and $1_2$, suggesting that the criterion weights in these two submodels are most similar to each other, even though they differ in terms of $w_2$ weights. In contrast, the lowest concordance (0.80) was observed between the $1_2$ and $2_2$ submodels, indicating some differences in the distribution of weights between them, particularly for $w_2$, $w_3$ and $w_7$. Nevertheless, all coefficients exceeding 0.80 indicate overall consistency between submodels, suggesting that although there are differences in the distribution of weights, their overall structure is similar.

The results of this study provide important insights into the application of MCDA methods for assessing renewable energy policies in EU countries. By integrating four MCDA methods—MARCOS, TOPSIS, SPOTIS, and MOORA—and using Copeland's method to create a compromise ranking, we have addressed a fundamental limitation of previous studies, which often rely on a single ranking method. Our approach ensures that the final ranking of alternatives (EU countries) is more robust and stable, as it synthesizes multiple perspectives on renewable energy performance.

One of the significant findings of this research is the high degree of concordance between the rankings generated by different MCDA methods. Despite using four distinct techniques, the rankings in all submodels showed minimal variation, especially for the top-ranked countries. This result suggests that the leading EU countries in renewable energy policy management, as assessed by our criteria, perform consistently well regardless of the method used. This stability in ranking is significant because it reinforces the reliability of the analysis and indicates that policy recommendations based on these rankings would likely be robust to methodological changes. Furthermore, the Copeland compromise ranking effectively balances the insights from all four methods, providing a solution that reflects the consensus of the different ranking approaches.

Applying the SITW approach further enhances the reliability of the analysis by generating stable and consistent criterion weights across submodels. The weights derived using SITW reflect the relative importance of criteria such as renewable energy generation, environmental impact, and economic efficiency, which are crucial for evaluating energy policies. The results show that specific criteria, particularly those related to renewable energy sources like wind and solar power, are consistently ranked highly important across all submodels. This finding aligns with current trends in the EU's energy policy, emphasizing the transition towards renewable energy and reducing carbon emissions.

However, some variations in criterion weights between submodels suggest that specific countries may prioritize different aspects of renewable energy management depending on their unique economic, geographic, and environmental contexts. For example, submodels that emphasized fossil fuel reduction showed greater weight on criteria such as gas and oil consumption, while others placed more importance on renewable energy shares. These differences highlight the need for tailored energy policies considering each EU country's specific conditions and priorities rather than a one-size-fits-all approach.

The high correlation values between the compromise rankings and the rankings generated by the individual MCDA methods, particularly the weighted Spearman coefficients, confirm the effectiveness of the compromise approach. Even in submodels where slight deviations occurred in the lower ranks, these discrepancies did not significantly impact the overall ranking structure. This robustness is crucial in decision-making processes, where small fluctuations in rankings can lead to different policy outcomes, mainly when the rankings involve large-scale national energy strategies.

While the methodology and results presented in this study offer a comprehensive assessment of renewable energy policies in the EU, several limitations must be acknowledged. First, the analysis is based on a finite set of criteria and alternatives, which may not capture the full complexity of energy policy challenges in different countries. Future research could expand the number of criteria to include social and technological factors, such as public acceptance of renewable energy technologies and the pace of technological innovation. Additionally, while this study focuses on static policy performance evaluation, renewable energy policies are dynamic and evolve over time. A temporal analysis that tracks changes in country rankings over several years could provide deeper insights into the effectiveness of policy interventions.

Furthermore, the reliance on decision matrices derived from existing data may limit the generalizability of the results to regions outside the EU. Extending this framework to other geographic regions, such as developing countries or regions with different energy infrastructures, would help test the methodology's robustness and provide a broader perspective on global renewable energy management.

In conclusion, this study demonstrates the effectiveness of integrating multiple MCDA methods and employing the SITW approach to provide a stable and comprehensive evaluation framework for renewable energy policies across EU countries. The use of Copeland’s method for compromise ranking ensured that the final rankings were robust, reflecting the strengths of each MCDA method without biasing the results toward any single approach. Additionally, the SITW method allowed for the determination of stable and optimal criterion weights, ensuring a balanced representation of the most critical factors influencing energy policy performance. This framework offers policymakers a reliable tool for navigating complex decision-making processes in the renewable energy sector, where multiple, often conflicting criteria must be balanced.

Future research should explore the application of this framework to other regions and sectors to assess its adaptability and robustness in different contexts. Expanding the set of criteria to include social, technological, and market-based factors could provide a more holistic view of energy sustainability. Additionally, incorporating temporal analysis to track changes in rankings and criteria importance over time would offer valuable insights into the dynamic nature of energy policies. Such developments would ensure that the framework remains relevant in addressing the rapidly evolving challenges in sustainable energy management and policy formulation.