Evaluation of Citizen Satisfaction with E-Services: An Analysis Using the E-GovQual Model and Importance-Performance Analysis

E-government is defined as the use of computers and the internet by the government to deliver information and services to citizens, businesses, and other stakeholders [1]. Over the last decade, e-government has received increasing attention as many governments recognize the importance of using information and communication technologies to ensure effective and transparent governance [2]. However, success in the implementation of e-government services cannot be measured absolutely by increasing the utilization rate of the services offered by the state to users (society/citizens). Information systems in general show that ultimate success depends on continued use rather than initial use. And continuous use can be achieved if users are satisfied with the system used [2], [3], [4]. Evaluating the success of online e-government services from various user perspectives is still not widely done. Therefore, this is considered a major gap in the currently dominant e-government online service maturity model [5].

User satisfaction is the result of a subjective evaluation that includes cognitive and emotional aspects, including the feelings and emotions of information system users. Meanwhile, e-government online services involve many stakeholders, each of whom has different interests and goals and will influence various interests that play a central role in the success of online e-government services. User satisfaction in the context of e-government has been the focus of various studies that propose different frameworks and approaches. Efforts to identify key factors that influence user satisfaction with the effectiveness of e-government services have evolved over the years since e-government was introduced as a new method of providing services in the public and government sectors [6], [7], [8].

The success of e-government services is a complex concept and its measurement must consider multidimensional factors [2]. This policy was created to improve the quality of service to the community through better implementation of e-government provided by the government. This is in accordance with where service quality can be realized through fulfilling customer needs and desires, as well as delivery decisions to match customer expectations [9]. In this respect, many media tools are used by the government to convey information to the public. One way to provide e-government services to the public is in the form of a government website [10]. Therefore, governments try to improve electronic services (e-services) by making it easier for people to access information, and services [11].

The main purpose of this research is to comprehensively analyze the student satisfaction level regarding the electronic services provided by Anadolu University through its website and Anadolu mobile application. In this context, the research includes a detailed evaluation by using the E-Government Quality Model (E-GovQual) and IPA methods to determine the quality, effectiveness, and user satisfaction regarding e-services. For this purpose, primary data was collected by surveying 1,000 students (undergraduate and graduate) actively receiving formal education in a total of 9 faculties and one graduate institute. Thus, by conducting relevant analyses in the light of both primary and secondary data, a comparative perspective was developed by evaluating user satisfaction with Anadolu University e-services. The differences in user satisfaction levels between these different academic units were comparatively discussed.

The findings of the research provide information that might contribute to the improvement of the overall quality of online services at Anadolu University. Therefore, the strategic goals and recommendations of the study are vital for the development of e-services, increasing user satisfaction, adoption of e-governance, and making transparency and participation an important element in e-service provision for Anadolu University, as well as other universities and public institutions.

Service is the behavior of producers to fulfill the needs and desires of people to achieve consumer satisfaction. This behavior can occur during, before, and after a transaction occurs [12]. From this definition, it can be concluded that service is an action carried out by a company towards consumers that has a positive or negative impact on both parties. Service quality is a measure of how well the level of service quality provided meets customer expectations [9]. On the other hand, public service is defined as providing services (serving) the needs of people or society who have an interest in the organization itself with the basic rules and procedures that have been determined [13].

In this modern world, public services are not only limited to face-to-face contact, but also utilize the internet as a means and provision of services needed by the community. E-government is a government system process that utilizes ICT (Information, Communication, and Technology). ICT is the use of information and communication technology that is fast, easy, and at a lower cost [14]. E-government is a relationship with the use of information technology used by government organizations that can form relationships with society, businesses, and other institutions in government [15].

Almarabeh and Abu Ali [16] define e-government as the use by government institutions of information technology in the form of the Internet and mobile computing, which can change relations with society and business. The World Bank [17] defines e-government as an effort to benefit from and use telemetrics to increase the efficiency and effectiveness of the state, to better provide various services to the public, to provide a wider public with access to information, and to make the state administration more responsible, accountable, and transparent to the public.

There are four indicators of success in e-government, namely:

1. Availability of data and information in the data center;

2. Availability of data and information for regional promotional needs;

3. Availability of e-government applications as support for public services;

4. Availability of public dialogue applications to improve government communication with the public, between government agencies, via e-mail, SMS, or teleconference.

E-government applications have various goals:

1. To facilitate people’s access to public services;

2. To facilitate the state’s provision of public services to society;

3. To create a network system that facilitates communication and coordination among state institutions, as well as among the state, society, and the private sector, with the Internet, and facilitates the public’s participation in policy-making processes;

4. To implement e-government services using information system applications and data processing systems that facilitate the transparent dissemination of information to the public.

According to Oliver [18], satisfaction is a summary of the psychological state that emerges as a result of the combination of emotional aspects related to expectations with the consumer’s previous feelings about the consumption experience. Besides the categorical classification of transactional and cumulative satisfaction based on the natural view, it can also be classified into cognitive satisfaction and emotional satisfaction [19]. In this research, both cognitive satisfaction and emotional satisfaction will be adopted (post-adoption), which refers to how users think and feel about the service after using it.

In the literature, there are several theoretical models and frameworks for evaluating the success of e-government services for different purposes and perspectives. Welch and Hinnant [20] define satisfaction with e-government services as “an individual’s overall assessment of the effects (positive or negative) of e-government operated by the government”. This is because satisfaction with e-government services is a multi-dimensional construct consisting of a variety of different values such as efficiency, accountability, fairness, and effectiveness.

Several studies place user satisfaction as a dependent variable or independent variable. In e-government satisfaction studies, user satisfaction was the fourth most frequently used independent variable (21 studies), and only 95 studies considered user satisfaction as a dependent variable [21]. Vaezi et al. [22], who conducted research on user satisfaction with various research models in the Information Systems study domain, identified that research related to user satisfaction is divided into 2 research orientations, namely, “process-oriented satisfaction” and “result-oriented satisfaction”. Satisfaction in process orientation aims to study the factors that influence or shape user satisfaction. The results orientation is to study the consequences or impact of user satisfaction.

Osman et al. [23] proposed a new conceptual model to measure the success of e-government online services from the perspective of different stakeholders. They conducted a bibliographic study to identify factors that influence user satisfaction with e-government online services, which is influenced by various factors and dimensions that are grouped into 4 main constructs: costs, opportunities, and benefits, namely the “COBRA Model”. The development of this model is based on the IS-Success model theory, Social Cognitive Theory (SCT), and the SWOT (Strengths, Weaknesses, Opportunities, and Threats) Analysis Model. The COBRA variable in quantitative structure analysis consists of 49 indicators, including cost (13 variables), benefits (18 variables), risk (5 variables), and opportunity (13 variables). Data analysis uses SEM (Structure Equalization Modeling) to build relationships between identified variables related to variables and user satisfaction. The measurement range of this method was developed, tested, improved, and validated on a sample of the use of e-government online services in Türkiye. The research results confirm that the COBRA method is an interesting approach to evaluate the success of e-government online services from the citizen’s perspective and can be generalized to other perspectives and management environments.

Noor et al. [24] conducted a user satisfaction survey on a type of e-government service in a government financial system in Malaysia. The results of this research show that e-government user satisfaction is measured by 4 constructs, namely individual impact, organizational impact, system quality, and information quality. Individual impact is used to measure the extent to which the e-government system affects the skill and effectiveness of users. Organizational impact measures the extent to which e-government services influence improvements in organizational outcomes and capabilities. Information quality is a measure of the quality of the output obtained from the e-government system. System quality is the performance measure of the e-government system in terms of engineering and design.

In the article “E-service Quality Measurement: The Example of Public Universities Providing Distance Education Services” written by Çelik and Perçin [25], 30 universities providing distance education were evaluated in terms of e-service quality. The e-services provided by the universities were evaluated with surveys conducted on both students and faculty members, and the most successful university was Sakarya University, while the least successful university was Fırat University [25]. Similarly, in the article “Digital Transformation in the Provision of Public Services: A Study on the Internet of Things” written by Göçoğlu [26], it was evaluated that the services provided by public institutions have undergone digital transformation with the development of Internet technologies. He discussed that developing internet technologies has had an impact on every aspect of life, public administration has also been affected by these effects, and many services previously provided by going to public institutions have started to be provided in a virtual environment within the scope of digital transformation.

Tamtürk’s [27] article “Electronic Document Management System in Public Administration” revealed the benefits that an electronic document management system might provide to the public sector. According to Tamtürk [27], since the 1990s, especially with the development of computer technologies, the private sector in Türkiye has slowly started using technology and e-services, but these breakthroughs in the public sector have been made at a later date. Therefore, the use of “e-government” applications in the provision of public services is very important and provides many benefits.

In the literature, the number of studies on the effectiveness, efficiency, reliability, and participation level of e-services in Türkiye is very limited. If the deficiencies of the services are not revealed, and policies are not determined, the level of user satisfaction with these numerous services might decrease. In addition, applications and services that ignore the requirements of the age and user needs will not be able to show the necessary effect. Therefore, the evaluation of effectiveness, reliability, efficiency, and participation level of current e-services, and strategic goals and recommendations are significant for most public institutions to increase service quality and citizens’ satisfaction.

This part of the study includes the research question, case selection, data collection, data analysis, and hypotheses of the research.

In light of the above information, measuring the satisfaction level of students of Anadolu University with several e-services is important in terms of values such as quality, effectiveness, transparency, and participation. Therefore, in this research, the research question “What are the factors that affect the satisfaction levels of citizens towards e-services?” is examined.

Türkiye has been developing online services since the last decade and has succeeded in the majority of the public services, like health, transportation, and education. Nearly every public institution in Türkiye has a website and an application that can be used to obtain information and use e-services. Anadolu University is one of these institutions that implements e-government policies on its website (https://www.anadolu.edu.tr) and mobile application (Anadolu Mobil). Both formal and open education students can benefit from e-services such as announcements on the online learning platform (Mergen), lecture videos (E-Campus), grades and attendance (ANASİS), E-Mail, E-Cafeteria, and E-Library. Anadolu University is the educational institution with the highest number of students in Türkiye, with 1 million 141 thousand 930 active students (formal and open education) from both Türkiye and abroad as of the 2023-2024 Spring Semester (Anadolu University website, January 2024 data). The university has 12 faculties, 3 of which are open and distance education, 3 colleges, one of which is the State Conservatory, 3 vocational colleges, 2 institutes (Graduate Education Institute and Disabled Research Institute), and 25 research centers.

Previously, the Anadolu University website has never been measured comprehensively concerning the quality of e-government services. So the factors related to e-government on the website are unknown. In the literature, there are mainly four criteria that are widely used in analyzing the quality of e-government services. These criteria, namely efficiency, trust, reliability, and citizen support, are also used as 4 dimensions of E-GovQual that have been validated by Papadomichelaki and Mentzas [11]. Therefore, analysis of the Anadolu University e-services by these criteria is expected to reveal service quality and student satisfaction from these services.

To comprehensively evaluate student satisfaction at Anadolu University, a pilot survey was conducted with 100 students receiving formal education at Anadolu University. They were selected by probability sampling and random sampling techniques that are mainly used where all elements of the population have the same chance of being selected. A 5-point Likert scale ranging from 1 (lowest) to 5 (highest) was used in the survey. A score of 1 generally indicates low satisfaction or strong dissatisfaction, while a score of 5 indicates the highest level of satisfaction. This scaling system allowed participants to express their feelings and thoughts in more detail, enabling a more detailed and effective analysis of their perceptions. Depending on the preliminary results of the pilot study, the survey questions were edited and then applied to 1,000 students at the campus.

The universe of the research covers a very diverse student community consisting of approximately 20,423 formal students, both local and international students, who are actively studying in formal education, covering 9 (nine) faculties and 1 (one) graduate institute at Anadolu University. Therefore, a sample size of approximately 1,000 students was selected for the research, 100 from each faculty and one institute. This sample size represents 5% of the total student population. Participants were randomly selected regardless of age and gender, and the only criteria were to be an active student at Anadolu University and to have regular access to one of the e-services (Anasis, Mergen, library, cafeteria and other operations) on the website or mobile application at least once a month. This inclusive approach aims to capture a wide range of perspectives and experiences, and to provide a more comprehensive understanding of the problems and satisfaction in website and mobile application use. By including both local and international students from various academic units, the research aims to address the unique needs and preferences of the student body and contribute to the development of a more effective and tailored digital experience. In order to fill out the surveys, the researchers reached volunteer participants directly through face-to-face meetings. The desired number of 1,000 students was successfully reached.

To analyze the obtained data, the E-GovQual model and IPA analysis method are used in this research. In this direction, calculations were made using SmartPLS to calculate every variable and the SPSS program to test the hypotheses. Then, analyses were made to determine the importance levels of the perceived performance of e-government services and the real expectations of the students, and the overlap between the performance of Anadolu University e-services and the expectations of the students was determined.

The effect of each e-service variable in the E-GovQual model was analyzed as Charaselt [28] did to find the factors affecting the quality of e-services. Similarly, in the research conducted by Napitupulu [29], an e-service evaluation framework was developed by combining the E-GovQual model and IPA analysis method to determine the quality of e-government services in Indonesia and to identify the issues that need to be improved. Therefore, our research also benefited from these methods and analysis techniques.

Within the scope of the research, a comparative analysis of e-services provided by different faculties and institutes within Anadolu University via the website and mobile application was provided. The comparison between faculties revealed detailed information on the differences in student satisfaction. Thus, this research helps to determine the strengths, opportunities and risks of the university’s website and mobile application in general, and the faculties and institutes in particular.

According to Papadomichelaki and Mentza [11], there are four variables that need to be applied to e-services (Figure 1). Firstly, “efficiency” refers to whether government websites can provide ease for the public in accessing any information needed in the government portal, enabling the collection of necessary and quality information. This variable describes the ease, quality, and accessibility of the e-services. The indicators of the efficiency variable are:

· Clear and easy-to-follow structure

· Effective search engine

· Well-organized application and website map

· Meeting users’ expectations

· Accurate and detailed information

· New information

· Adequacy of instructions for filling out forms

· Information in English for foreign students (internationalization)

Secondly, as the second variable, “trust” refers to doubts about website services in protecting personal or private information. For instance, a website can provide trust in the privacy of users by protecting the personal information provided against third parties. This variable is used to assess people’s trust in e-services in terms of their protection from dangers and risks in the online service process. The indicators of this variable are as follows:

· Username and password security

· Personal data authentication

· Personal data security

· Use of personal data for clearly defined purposes

“Reliability” is the third variable that refers to the data loading time of the website due to maintenance on the internal system, which made some services inaccessible. A website can provide good services according to accessibility and availability, so that online services are available all day long with high access speed. This variable is useful to determine citizens’ views of e-services regarding the accuracy and timeliness of service delivery. This term includes technical functions related to accessibility, usability and service accuracy. The indicators of this variable are as follows:

· Quick download of forms

· Easy access when needed

· Success of the service when first accessed

· Just-in-time service delivery

· Fast download of website and application tabs

· Compatibility of the browser system

Behind all that, there needs to be “citizen support” in terms of overcoming errors or loading. The existence of a special department formed to process shared data helps the community in searching for information or transacting. This variable is used to measure the extent of assistance provided by public institutions to help citizens search for information or perform other actions. The indicators of this variable are as follows:

· Public employees’ sensitivity to user issues

· Public employees’ prompt response to user questions

· Public employees have sufficient knowledge to answer user questions

· Public employees’ being reliable and confident in providing services

Calculations to measure efficiency, reliability, trust, and citizen support of e-services were carried out with SmartPLS 4, to calculate the outer model analysis, inner model analysis, and hypothesis testing. Then an analysis of the suitability between perceived performance and actual public expectations of e-government services is carried out, as well as a gap analysis to determine the value of the gap between performance and expectations of the Anadolu University website with the expected quality value of the website.

Depending on this model, the following hypotheses were developed:

H1. High level of efficiency is likely to increase citizens’ satisfaction with e-government services.

H2. High level of trust is likely to increase citizens’ satisfaction with e-government services.

H3. High level of reliability is likely to increase citizens’ satisfaction with e-government services.

H4. High level of citizen support is likely to increase citizens’ satisfaction with e-government services.

Factor analysis is a statistical technique used to provide a simple description through several factors. It has the principle of simplifying data descriptions by reducing the number of variables or dimensions. Factor analysis is a procedure for identifying variables from their equations that are shown by high correlation values. Variables with high correlation form a group of factors. Factor analysis is used for factor validation testing. In this analysis, the large correlation between the factors that form the variables was also observed. There are several explanations regarding factor analysis [30]:

1. Correlation between variables must be strong enough, such as above 0.5.

2. Partial correlation refers to the relationship between two variables while controlling for the influence of other smaller variables, assuming they remain constant.

3. Testing the correlation matrix involves examining the presence of significant correlations between variables within the matrix.

4. Meet the normality of the variables or factors that occur.

In factor analysis, there are two approaches:

1. Exploratory Factor Analysis means that factors, which are formed in the form of latent variables, are random. In other words, exploratory factor analysis is valid where researchers do not yet have the knowledge or theory to compile the factor structure that is formed, and a new theory will be formed.

2. Confirmatory factor analysis refers to a number of factors that will be formed and variables will be included for each factor that has been formed. It is a multivariate analysis method used to confirm the measurement model that is built is as hypothesized. This analysis has latent variables, or variables that cannot be formed, and indicator variables that can be observed and measured directly.

Similarly, IPA is a simple and useful technique for identifying attributes of service providers that need improvement or attributes that need to be reduced in priority [31]. IPA was first introduced by Martial and James in 1977 to measure the relationship between consumer perceptions and priorities for improving product or service quality, known as quadrant analysis [32]. There are two types of methods for displaying IPA data. The first technique is to place the quadrant intersection line at the average value for the level of satisfaction and priority of handling to know the general distribution of data in the quadrant, while the second method is to align the quadrant intersection line at the average value of the observation results on the satisfaction level axis and the treatment priority axis to know specifically in which quadrant these factors are located [29]. For this reason, researchers frequently use the IPA method to reveal information about service factors that affect users’ satisfaction and loyalty and the factors that users prefer.

IPA can give an idea about the quality of e-services of the relevant institutions based on the indicators in the E-GovQual Quality Model. This is because IPA has advantages over other methods by showing the features that need to be increased or decreased to maintain user satisfaction. Additionally, analysis results are relatively easy to interpret, because IPA helps to develop new methods according to the importance and application levels of relevant factors that are useful in developing e-services via websites and mobile applications.

In order to reveal the findings of the study, analyses and tests were made at different stages.

First of all, analysis is carried out with a measurement model at the first-order construction stage where the analysis of this measurement model consists of four stages of testing: i. individual item reliability, ii. internal consistency reliability, iii. average variance extracted (AVE), and iv. discriminant validity [33], [34]. Measurement models are carried out to determine the relationship between constructs (variables) and their indicators. The following results of the model measurement analysis are explained in four stages.

Stage I: Individual Item Reliability Test

This test is done by looking at the standardized loading factor value. This value illustrates the magnitude of the correlation between measurement items (indicators) and their constructs. A loading factor value above 0.6 is said to be ideal, which means that the indicator can be said to be valid as an indicator that can measure the construct or variable [35]. The results of this test show one indicator, namely EF8, having a value below 0.6, so it's clear we need to delete EF8 from the other test to ensure the reliability and validity of the results (Table 1).

Stage II. Internal Consistency Reliability Test

At this stage, testing is carried out using the Composite Reliability (CR) value with a threshold of 0.7 [34], [35]. CR is better at measuring internal consistency compared to Cronbach’s alpha in the SEM model because CR does not assume all indicators are the same in a variable. According to Table 2, it can be seen that all CR values are above 0.7, so it can be said that there are no problems in the CR test in general.

Stage III. AVE Test

The next convergent validity test is to measure the AVE values of the variables. This value describes the amount of variance or diversity of manifest variables that can be contained by latent variables. The AVE value of at least 0.5 indicates a measure of convergent validity [34], [35]. The data in Table 3 shows that all AVE values exceed 0.5, indicating that there are no issues with the AVE test.

Stage IV. Discriminant Validity Test

This test is done through cross-loading and then comparing it with the AVE root value. Another measure of discriminant validity is that the AVE root value must be higher than the correlation of other constructs [34]. Measuring cross-loading between indicators is done by comparing the correlation of the indicator with its construct and other block constructs. If the correlation between the indicator and its construct is higher than the correlation with other block constructs, this indicates that the construct predicts the size of its block better than other blocks [34], [35]. Discriminant Validity Test Results by SmartPLS (Appendix Table A) show that the loading value on the target construct is greater than the loading value with other constructs, this can be seen from the numbers highlighted by a darker color. The examination of the Fornell-Lacker’s cross-loading value is done by looking at the root value of AVE, which must be higher than the correlation between constructs with other constructs.

Table 4 shows that the AVE root value is higher than the correlation between constructs with other constructs. Therefore, based on the results of the two-stage cross-loading examination, it is known that there is no problem in the Discriminant Validity Test.

Figure 2 shows the results of the analysis after going through the outer model analysis. Based on the four stages that have been carried out in the previous model measurement analysis (outer model), it can be seen that the model that will be proposed in this study already has good characteristics statistically, under the requirements at each stage in the model measurement (individual item reliability, internal consistency reliability, AVE, and discriminant validity). Therefore, it can be concluded from the results of the model measurements that the model meets the requirements to proceed to the structural model testing stage (inner model).

The structural model analysis was carried out at the second-order construction stage with six stages of testing, consisting of testing the path coefficient ($\beta$), coefficient of determination (R$^2$), T-test using the bootstrapping method, effect size (f$^2$), predictive relevance (Q$^2$), and relative impact (q$^2$). However, the research model used is in the form of a second order, and it can be carried out with a first-order construct significance test/T-test with outer loading, a path coefficients test with bootstrapping, and also added with a coefficient of determination (R$^2$) test as in the research.

Stage I. Path coefficient ($\beta$)

This test is carried out by looking at the threshold value, which is above 0.1, where the path can be stated to have an influence in the model if the results of the path coefficient test value are above 0.1.

The results of the 4 paths in this research model are accepted and significant because the path coefficient value is above 0.1 with efficiency (EF) with E-Government Service Quality (EGSQ) (EF -$> $ EGSQ), Trust (TR) with E-Government Service Quality (EGSQ) (TR -$> $ EGSQ), Reliability (RL) with E-Government Service Quality (EGSQ) (RL -$> $ EGSQ), and Citizen Support (CS) with E-Government Service Quality (EGSQ) (CS -$> $ EGSQ), as shown in Table 5.

In the second order, this test was also carried out using the bootstrapping method on SmartPLS 4. From the results of the path coefficients study, it can be seen in Table 6 that all first-order constructs have a significant effect on the second-order construct of EGSQ, where the T-statistic value is above 1.96, this indicates that all first-order constructs are dimensional constructs that form the EGSQ construct.

Stage II. Coefficient of Determination (R$^2$) Test

This test is conducted to explain the variance of each endogenous target variable (variables that are considered to be influenced by other variables in the model) with a measurement standard of around 0.670 considered strong, a range of 0.333 considered moderate, and 0.190 considered weak. The following are the results of the R-Square calculation using SmartPLS as follows:

Through the table, the results of the coefficient of determination test can be seen, where R$^2$ of EGSQ has a value of 1000 (100%). This means that the exogenous variables EF, TR, RL, and CS strongly explain (100%) the variance of EGSQ, as indicated in Table 7.

Stage III. T-Test

At this stage of testing, the bootstrapping method in SmartPLS 4 was used by testing two-tailed with a significance level of 5%. Through data analysis results of the T-test (Appendix Table B), it can be seen that all the results are above 1.96, stating that all construct indicators in the model are valid.

Based on the research model, namely E-GovQual, and the results of the previous pilot test, the variables used in this study consist of four variables that are reduced to 20 indicators whose coding is described in Table 8.

The data used in this stage that was used for validity and reliability testing comes from 1000 questionnaires that have met the requirements that are declared valid and in accordance with the criteria and are declared complete. Testing for validity and reliability is divided into two parts, namely based on user perception assessment and user expectation assessment in using the Anadolu University website or platform.

The results of the reliability and validity tests on user perception assessments show that the Cronbach Alpha value is 0.909 from the 20 existing indicators, this value is greater than the existing provisions of 0.6, which shows that the instrument above can be said to be reliable [36].

The validity value of each question can be seen by paying attention to the Corrected Item-Total Correlation values that were obtained by using the Pearson Product Moment correlation technique, namely by comparing items with the total item score (r$_{\text{count}}$ value) compared to the r$_{\text{table}}$ value. With a significant value of $<$ 0.05 ($\alpha$ = 5%), if the calculated r-value $>$ r table value, then the item can be said to be valid [37]. The value of the r table in this study is 0.062 (Appendix Table C). Based on the test results, it shows that all indicators for assessing the expected quality of website services are valid, this can be seen from the calculated r value, which is greater than the r table. Based on the internal coefficient relationship, it can be said that, on average, the indicators used in this study have a strong level of coefficient relationship; this can be seen from the calculated r value, which is, on average, at an internal coefficient of 0.60 - 0.799. From the results of testing the validity and reliability of the assessment of expectations above, it can be concluded that all indicators used as questions are declared valid and reliable.

After conducting validity and reliability tests, a Paired Sample T-test analysis was conducted to determine whether or not there was a difference in the average between the two groups of the same sample. If the probability value or Sig. (2.tailed) $<$ 0.05, it can be stated that there is a significant difference between the two paired samples. Conversely, if the probability value or Sig. (2-tailed $>$ 0.05, there is no significant difference between the two. Testing is done by looking at the Sig. (2.tailed) value [38]. After conducting a Paired samples T-test, it can be seen that the outputs have a Sig value (2-tailed) of 0.000, meaning $<$ 0.5, so it can be concluded that there is a gap or disparity between perception and expectations regarding the quality of website services (Appendix Table D).

This analysis is used to determine the achievement of the quality of e-service performance in the form of a website provided by Anadolu University based on student assessments. The measurement is done by calculating the comparison between the reality of the service received or the user’s perception with the expectations of the service desired by the user. The results obtained are in the form of the level of conformity of the achievement of the quality of e-service in the form of a website at Anadolu University.

The value of the level of conformity obtained from the comparison between the reality of the services received and the expectations of the community describes the quality of service performance, which is assessed in the form of a percentage of conformity. The following are the results of the level of conformity between the quality of e-service in the form of a website at Anadolu University for each dimension of E-GovQual:

Table 9 shows the average level of conformity of each indicator in the four E-GovQual variables. From the table, it can be seen that all average values of the level of conformity of the 4 dimensions are worth 79.15%. These results indicate that overall, the system or service being evaluated is quite good at meeting user expectations, although there is still some room for improvement. This average shows that most of the evaluated dimensions have provided quite adequate results, but some areas still need to be improved to be more in line with user expectations.

The dimensions of efficiency (EF) and trust (TR) obtained a level of conformity of 83% and 83.6%, which indicates that users feel that the performance in terms of efficiency is very good and almost completely understand their expectations. Although there is room to improve performance in terms of speed or smoothness of the process, so that the system or service is more optimal. And some aspects of trust that need to be improved, such as transparency or security elements, to further feel the increase in user security and trust. On the other hand, Reliability (RL) and Citizen Support (CS) recorded a level of conformity of 75.9% and 75.1%. This indicates that users may experience instability or technical problems that cause the system or service to be unreliable. And improving response speed and providing more effective solutions can greatly improve customer satisfaction.

At this analysis stage, it is carried out to determine the level of gap or difference between student expectations and perceived performance or user perception of the quality of e-services in the form of websites or platforms. This analysis was conducted on the Anadolu University website in order to determine the gaps that occur in the e-services contained on the website. The gap analysis was conducted based on the existing E-GovQual model.

Gap analysis in this study was conducted in three areas: analysis by service item, analysis by service dimension, and overall average analysis. This approach enabled the identification of gaps across each service/attribute item, dimension, and the overall evaluation within the four dimensions of E-GovQual. The results of the Performance (Importance) gap calculations for each item, per dimension, and the overall average are presented in Table 10.

The table shows the average level of gaps for each indicator in the four E-GovQual variables. From the table, it can be seen that all gap values for each indicator are negative, indicating that the services provided are not in accordance with user expectations. A negative gap indicates room for improvement. If the gap is negative, it means that performance does not meet user expectations. The larger the negative gap, the greater the difference between expectations and reality, indicating that there is a problem or deficiency that needs to be addressed immediately. The average gap shows a value of -0.925, indicating that overall, the performance of the system or service is lower by about 0.925 points than user expectations. Reliability has a larger gap than other dimensions, namely with a value of -1.11, indicating that this system or service has significant problems in terms of reliability or consistency.

Although the comparison of the average performance and importance gap values for the four dimensions in the research results yields negative gap values, a gap result of less than -1 indicates good service quality, while a result greater than -1 suggests that the quality of service provided is not satisfactory. Therefore, it can be concluded that the quality of e-services at Anadolu University is considered good.

The next part is the analysis process is carried out with IPA using quadrant analysis, the results of which are mapped into a Cartesian diagram with the x-axis explaining perception and the y-axis for expectations. The following are the results of the quadrant analysis obtained from processing data from the distribution of questionnaires.

Based on Figure 3, the following can be seen:

Quadrant A

In this quadrant, the variables have a high level of importance, but a low level of performance. Therefore, these variables are the main priority for improvement.

There are 7 main priority factors for improvement, namely

1) EF3: Regularity of layout and on e-service sites

2) EF7: Clear assistance and instructions for filling out forms in e-services

3) RL3: Success of the first display when accessing e-services

4) RL4: Timeliness of services provided by e-services

5) CS1: Responsiveness of university staff in solving e-service problems

6) CS3: Knowledge of university staff in answering questions about e-service

7) CS4: Staff ability to foster trust

Quadrant B

This quadrant includes variables with high levels of importance and performance. These variables are the main strengths that must be maintained. There are 9 factors that must be maintained, namely:

1) EF1: Ease and clarity of e-service structure to follow

2) EF2: Effectiveness of search engines in e-service to find information

3) EF4: Suitability of e-service to user needs

4) TR1: Security of username and password when using e-service

5) TR2: Use of personal data is limited to what is needed to access e-service

6) TR3: Security in data storage

7) TR4: Use of data only for specific, clear purposes

8) RL1: Speed of file or content download process in e-service

9) RL2: Availability of e-services

10) RL5: Smoothness of e-service on various devices such as computers and mobile phones

Quadrant C

In this quadrant, the variables have low levels of importance and performance. These variables are not a top priority for service development.

There is 1 factor with low priority, namely:

1) CS2: Speed of response of university staff to e-service problems

Quadrant D

This quadrant includes variables with low importance but high performance. These variables allow for more resource allocation than they should. There are 2 redundant variables, namely:

1) EF5: Accuracy and detail of the information displayed

2) EF6: Timeliness of information provided in e-service

This stage is carried out to analyze respondents’ answers to the questions in the questionnaire. The respondent data was successfully obtained by the researcher within 6 weeks (27 May 2024–7 June 2024; then there was a 3-month gap due to the summer holidays, then continued on 14 October to 8 November 2024) and was 1000 data. The demographic information includes Gender, Faculty and Department, Grade, Level of education, Nationality, Mother language, Frequency of using the website, and which 3 are most frequently accessed.

Gender

1000 respondents are involved in this study, the majority were female, with a total of 638 respondents (64%), and male respondents with a total of 362 (36%).

Department

Concerning the 1000 respondent data used in this study, there are 100 respondents from each faculty in Anadolu. In total there are 9 faculties, namely the Faculty of Pharmacy, Faculty of Education, Faculty of Science Literature, Faculty of Fine Arts, Faculty of Law, Faculty of Economics and Administrative Sciences, Faculty of Communication, Faculty of Tourism, and Faculty of Health, and 100 respondents were taken from the Institute.

Education Level

There were 900 responses from the undergraduate level from 9 faculties. The rest were 99 respondents with master’s level and 1 respondent at doctoral level who are in the Institute.

Grade

The number of respondents is based on their class/school year. It was found that 269 respondents or around 29% were first-year students, who were the most respondents. Then continued with students in the second year, which was 250 respondents or around 26%. Followed by students in the third year, which is not much different from before, which is 249 respondents, or around 22%. While in the fourth-year students, there are only 216 respondents, or around 21%. Finally, the questionnaire was filled out by fifth-year students, as many as 16 respondents, or around 2%.

Nationality

854 respondents are Turkish citizens (84%), the remaining 155 (16%) are international students who come to study, these foreign students come from Uzbekistan, Syria, Russia, Indonesia, Azerbaijan, Kazakhstan, Turkmenistan, Serbia, Belgium, and Afghanistan.

Language

Demographics and Language Percentage show the same numbers as the nationality percentage. That is, there are 16% of students who do not use Turkish as their daily language.

Frequency Access

Regarding 1000 respondents, the access frequency is divided into five categories; 69 respondents (6.9%) indicated that they access e-services “Very Rare” (once, or more a month), then 75 respondents (7.5%) said “Rare” (once in a week), 244 respondents (22.4%) said “Normal” (twice or more in a week). Then 350 respondents (35%) said “Often” (once every day) option, and the rest of the 282 respondents (28.2%) said “Very Often” (twice, or more every day).

Most Frequently Access

The 3 e-services most frequently used by students are MERGEN, ANASIS, and E-Cafeteria. The MERGEN application has the largest number of users, namely with 953 respondents (33%), indicating that this application is very popular among students. ANASIS follows with 847 respondents (29%). Meanwhile, E-Cafeteria received a score of 823 respondents’ (27%) votes. These three applications show a high level of student engagement with platforms that support their academic and daily needs.

On the other hand, applications such as E-Library, E-Mail, and E-Magazine have a much lower number of users. E-Library was only chosen by 96 respondents (2%), while E-Mail and E-Magazine were chosen by 147 respondents (5%) and 24 respondents (less than 1%). This shows that although these e-services are important, they are not as popular as MARGEN, ANASIS, and E-Cafeteria in terms of user frequency.

In addition, 110 respondents decided to choose 3 e-services that they used or chose not to use any applications, which may reflect that some students do not need to access these applications or have not found their maximum benefits.

In this study, an analysis was conducted with 2 stages for the research model, namely the analysis of the outer model/measurement model and the inner model/structural model. The measurement model was conducted to test the reliability and validity of the outer model through indicator reliability, internal consistency reliability, convergent validity, and discriminant validity.

Meanwhile, testing the structural model (inner model) through the path coefficient, coefficient of determination, T-test through the bootstrapping method, effect size, predictive relevance, and relative impact using blindfolding testing [34], [35].

Although the indicators of each questionnaire question come from the original variable E-GovQual, researchers must still ensure that each variable and indicator are related to each other. Therefore, several tests were conducted on SmartPLS and SPSS. This is because the research model used is in the form of a second order, the structural model test (inner model) is carried out with a first order construct significance test/T-test with outer loading, a path coefficients test with bootstrapping, and also added with a coefficient of determination test as in the analysis results.

Based on the results of the measurement model analysis that has been carried out by this study, it has met the requirements, and has good statistical characteristics, and is worthy of being continued to the model structure analysis stage. This model structure analysis was carried out to test the inner model of the research conducted.

It is important to note that one indicator was removed, namely EF8 (Use of Web Language), in this research model because it did not meet the outer loading value standard. The elimination of the indicator, according to the researcher, was caused by the use of 2 languages, namely Turkish and English, which were not too much noticed by students. This is because most of the respondents were students with Turkish citizenship, which reached 84.5%, while foreign students were only 15.5%. In addition, most of them did not provide answers, so the data became a missing point and could not be used in further analysis.

Based on the discussions above, researchers need to review and develop the research instrument, either through input/suggestions from experts such as lecturers, previous researchers, or others in order to obtain a more appropriate research model. Although researchers have tried to do their best, of course there are still many things that are beyond the control of researchers when implementing it in the field.

In this part, the interpretation and discussions are presented based on the results of the 3 stages of model structure analysis that have been carried out by the researcher. The three stages of analysis are the coefficient of determination (R$^2$) test, the path coefficient (B) and the T-test with the bootstrapping method. The following is an explanation of the hypothesis that has been formulated based on the results of the analysis that has been carried out.

H1. High level of efficiency is likely to increase citizens’ satisfaction with e-government services.

Based on the results of the model structure test conducted, the results of the path coefficient test show that the efficiency (EF) variable has a significant influence on the quality of e-government services, and shows that the EF variable in the first order construct is a dimension that forms the EGSQ construct [39].

Based on the results of the regression analysis, the t-statistic value for the Efficiency variable is 4.32, with a significance value of 0.012 (Sig. $<$ 0.05). This shows that the efficiency dimension has a significant influence on service quality. This positive and significant t-statistic value indicates that an increase in the efficiency dimension will improve the overall quality of government services.

The average score on the efficiency dimension shows a relatively high value, but there are still some aspects with gaps between expectations and performance. For example, variables such as EF5 and EF7 have quite large gaps, indicating the need for improvement in this area. These factors can strengthen or weaken user perceptions of EGSQ if not handled properly.

This is consistent with previous research by Papadomichelaki and Mentza [11] and Charaselt [28], which states that the efficiency variable has an influence on EGSQ. Based on the results of previous research, the researcher concluded that hypothesis 1 states that Efficiency (EF) has a significant influence on the quality of e-government services accepted.

H2. High level of trust is likely to increase citizens’ satisfaction with e-government services.

Based on the results of the structural model testing conducted, the results of the path coefficient test show that the Trust (TR) variable has a significant influence on the quality of e-government services and show that the TR variable in the first-order construct is a dimensional construct for forming the EGSQ construct.

Based on the results of the regression analysis, the t-statistic value for the Trust variable is 3.89, with a significance value of 0.021 (Sig. $<$ 0.05). This shows that Trust has a significant influence on service quality. A positive t-statistic value indicates that an increase in the Trust dimension will directly increase the perception of e-government services.

This is the same as the previous research conducted by Papadomichelaki and Mentza [11] and Charaselt [28], which stated that the trust variable has an influence on EGSQ. Based on the statistical value of 3.89, the hypothesis states that trust (TR) has a significant influence on the quality of e-government services accepted.

H3. High level of reliability is likely to increase citizens’ satisfaction with e-government services.

Based on the results of the model structure test conducted, the results of the path coefficient test show that the reliability variable (RL) has a significant influence on the quality of e-government services, and shows that the RL variable in the first-order construct is a dimensional construct that forms the EGSQ construct [39].

Based on the results of the regression analysis, the t-statistic value for the Reliability variable is 4.05, with a significance value of 0.015 (Sig. $<$ 0.05). This shows that Reliability has a significant influence on the quality of e-government services. A positive t-statistic value indicates that the more reliable a service is, the better the perception of service quality felt by users.

Some variables that support this influence are RL1 and RL2, which show quite good performance. However, variables such as RL3 and RL4 record a fairly large gap between performance and user expectations, indicating a need for improvement in these aspects.

This is the same as previous research conducted by Papadomichelaki and Mentza [11] and Charaselt [28], which stated that the reliability variable influences E-Government Service Quality. The researchers argue that based on the statistical value of 4.05, the hypothesis stating that Reliability (RL) has a significant influence on the quality of e-government services is accepted.

H4. High level of citizen support is likely to increase citizens’ satisfaction with e-government services.

Based on the results of the structural model testing conducted, the results of the path coefficient test show that the Citizen Support (CS) variable has a significant influence on the quality of e-government services, and shows that the CS variable in the first order construct is a dimensional construct that forms the e-government construct [39].

Then based on the results of the regression analysis, the t-statistic value for the Citizen Support (CS) variable is 3.72, with a significance value of 0.028 (Sig. $<$ 0.05) this indicates that Citizen Support has a significant influence on the quality of e-government services. A positive t-statistic value indicates that an increase in customer support will increase the perception of overall service quality.

Some important variables in this dimension include CS1 and CS4, which have a significant impact on user satisfaction levels. However, variables such as CS2 and CS3 record a larger gap between performance and expectations, indicating that there is a need for improvement in this area.

This is the same as the previous research conducted by Papadomichelaki and Mentza [11] and Charaselt [28] which stated that the citizen support variable has an influence on EGSQ. Based on the t-statistic value of 3.72, the hypothesis stating that Citizen Support (CS) has a significant influence on the quality of e-government services is accepted.

Based on the analysis of 20 variables in the dimensions of Efficiency (EF), Trust (TR), Reliability (RL), and Citizen Support (CS) used to evaluate Anadolu University’s e-service, the average presentation of the overall suitability value is 79.15%. This result shows that in general, e-government services provided by Anadolu University, such as student information systems (ANASIS), MERGEN, and E-Library, are quite good at meeting user expectations. However, there is room for improvement in several important aspects to further enhance the quality of service.

Some variables that show high suitability are EF1 (Ease and clarity of service structure), TR1 (Security of username and password), and TR2 (Limited Use of Personal Data). These variables have a suitability level above 85%, which indicates that users are satisfied with the clarity of the service structure, security of personal data access, and ease of operating e-government systems such as ANASIS to manage student information.

However, several variables with low suitability values require more attention, variables such as RL3 (success of the first display when accessing the service, RL4 (timeliness of service), and CS2 (speed of university staff response) have a significant gap between expectations and performance, with suitability values below 75%. This indicates that users experience problems in accessibility and consistency of services when accessing features such as E-Library, or getting support from university staff through the electronic complaint system.

In terms of dimensions, Efficiency and Trust have relatively high suitability values compared to Reliability and Citizen Support. The main focus of improvement should be directed at the aspects of service reliability and staff responsiveness, especially in providing faster and more responsive solutions to user needs. This improvement is very important to improve the user experience when accessing e-services, such as registering for courses, viewing exam schedules, or taking appointments at Cafeteria.

By improving the quality of services on key variables that require attention, Anadolu University can strengthen its position as an effective and efficient e-government service provider, providing a better user experience for students and other academic communities.

The results of the gap analysis show the difference between user expectations (importance) and perceived performance (performance) for the 20 measured variables. Overall, the average gap is 0.925, indicating that the performance of e-government services provided by Anadolu University has not fully met user expectations. This negative gap indicates that although users consider e-government services important, their performance has not met their expectations.

Several variables that have the largest gaps, such as RL3 (Success of first appearance when accessing the service), RL4 (Timeliness of service), and CS2 (Speed of university staff response), show that there is a fairly large gap between expectations and performance. These variables reflect the areas that need the most attention.

However, some variables such as EF1 (Ease and clarity of service structure), TR1 (Security of username and password), and EF2 (Effectiveness of search engine in E-service) show a relatively small gap between expectations and performance. This indicates that these elements have been well managed by Anadolu University’s e-government system, so that users feel quite satisfied with the ease of access and security offered.

Overall, this gap analysis highlights that Anadolu University has succeeded in providing efficient and secure services, but still needs to focus on the reliability and responsiveness of services, especially in the areas of reliability and Citizen Support. To improve user experience, focusing on improving service speed, timeliness, and staff response is essential. By reducing the gap between expectations and performance, the quality of e-government services at Anadolu University can be significantly improved.

Based on the results of the IPA analysis in the figure, it can be described in the classification of service factors based on user satisfaction seen from the results of perception and expectation values using IPA or quadrant analysis as follows:

Quadrant A (Main Priority)

This quadrant contains variables with high importance but low performance. The variables in this quadrant are the main focus for improvement because they have a major influence on the overall quality of e-government services. Variables in this quadrant indicate areas where user expectations are very high, but the service has not met those expectations. If not addressed immediately, it will reduce user trust and experience of the service.

Examples of variables in this quadrant are EF3, RL3, RL4, and CS1. These variables reflect problems perceived by users, such as inconsistent service displays or slow response times from university staff. If not addressed immediately, these issues can reduce user trust in the e-government system.

To improve performance, it is necessary to optimize the information technology system, such as increasing server capacity to speed up service access and improve the interface to be more responsive. In addition, university staff must be given special training on how to handle user complaints more quickly and effectively through an automated system or integrated chatbot.

Quadrant B (Maintain Achievements)

This quadrant includes variables with high importance and high performance. Variables in this quadrant are the main strengths of e-government services that must be maintained to maintain user satisfaction. Variables in this quadrant show what has been managed well and contributed greatly to user satisfaction. The focus is on maintaining or further improving service standards.

Variables in this quadrant are EF1, EF2, TR1, TR3, RL2, and RL5. Users find the service easy to access, safe, and efficient, such as on the ANASIS and E-Library platforms. It is important to maintain this high performance, for example, by ensuring that the service structure remains easy to understand and data security remains a priority.

Quadrant C (Low Priority)

This quadrant includes variables with a higher level of importance and performance. These variables do not require much attention because their influence on the quality of e-government services is not too significant. This area has a low priority for improvement because its impact is small on the user experience.

Examples of variables in this quadrant are CS2. Although performance in this area is still low, users do not consider it a very important aspect. Therefore, the focus of improvement in this area can be postponed. Unless there is an excess data source.

Although a low priority, simple efforts such as Establishing a regular schedule to update information on online services can improve user perceptions. For variables such as CS3, the researchers suggest that universities can provide staff training on key features of e-services, so that they can provide better explanations to users when needed.

Quadrant D (Excessive)

This quadrant includes variables with low importance but high performance. These variables may receive excessive training or resource allocation compared to their impact on user satisfaction.

The variables in this quadrant are EF5 and EF6. Although the performance of this service is very good, users do not pay much attention to these aspects. For example, the layout of the site is very organized, but users prioritize access speed. Resources in this area can be diverted to improve variables in quadrant A.

The study aims to evaluate the level of student satisfaction with Anadolu University’s e-services. The model used is the E-GovQual model to measure service quality based on four main dimensions, namely EF, TR, RL, and CS. The IPA method is applied to map 20 variables into four quadrants, namely Top Priority, Maintain Achievement, Low Priority, and Excessive, in order to determine the priority of improvement.

Data collection was conducted through a direct survey involving 1000 respondents from 9 faculties and 1 institute. Respondents actively used services such as ANASIS, MERGEN, and Cafeteria. The analysis involved calculating the level of conformity (gap analysis) and evaluating using t-statistic values to measure the significance of the relationship between expectations and performance.

Based on the results of the analysis, it was found that the overall service quality has a level of conformity of 79.15%. This shows that most services have met user expectations, but there is still room for improvement in several dimensions that show significant gaps. The Efficiency and Trust dimensions are the main strengths of Anadolu University’s e-government services, with an average level of conformity above 90%. Meanwhile, the Reliability and Citizen Support dimensions show significant gaps that need to be fixed immediately, especially in the variables in Quadrant A (Top Priority) such as EF3 (Regularity of layout), EF7 (Help and instructions in filling out the form), RL3 (First display look on website), RL4 (Service time), CS1 (Staff responsiveness in solving problems), CS3 (Staff knowledge of service), and CS4 (Stagg ability to increase trust).

By improving service reliability through technology optimization and providing staff training to improve responsiveness, Anadolu University can improve its overall service. An effort that will not only increase user satisfaction but also strengthen their trust in the e-services provided by the university.