Integration of eucs variables into delone and mclean models for e- government evaluation: Conceptual models

Article history: Received 5 July 2019 Revised 20 July 2019 Accepted 21 July 2019 Published 7 January 2020 This research was based on the modification of the DeLone and McLean information systems models by adding end-user computing satisfaction variables to determine the success factors for e-government systems. This model was adopted due to the aim of this study to investigate the factors responsible for the successful implementation of e-government by bringing it closer to public value. However, while the DeLone and McLean models focus more on the information system approach, the model proposed was on the premise that system quality (SQ), information quality (IQ), content (CO) and format (FO) are determinants of e-government system user satisfaction. Furthermore, the net benefits through a five-dimensional public value determinants were used to evaluate e-government websites from a community perspective. Responses from 150 communities were analyzed by smart PLS 3.0 using structural equation models to examine the relationship between the constructs of the proposed model. This study contributes to the research gap in adopting DeLone and McLean's model in the e-government due to the limitation in its validation for different contexts. The results support the effect of content variables on user satisfaction and simultaneously prove that it is possible to explain net benefits, with an r-squared value of 69.1%, using the variables in the proposed model. The five dimensions of public value adopted all proved to have a positive influence with a confidence level of 95%. The level of construct significance identified is able to help in the formulation of strategies to improve e-government services.


Introduction
Nowadays, the application of Information System and Information Technology (IS/IT) has improved in several fields including industry, education, automotive, and even services provided by government agencies through the utilization of electronic services. When a Government decides to use IS/IT to provide effective and efficient services to its users, they need a guarantee of the quality of the IS/IT product that is indeed good and satisfying. This can be evaluated based on several measuring variables such as project construction, special construction of information security, transparency, and information capability [1]. An example of this is the use of customer satisfaction as an indicator of information capability produced from electronic services. Therefore, to create effective and prime e-government services, the local government need to be focused on every time without limits. It was, however, reported by Bagus, Eka, Wijaya, and Ayu [2] that the accessibility of e-government websites, especially in the Provinces of Indonesia has many problems as observed from 23 out of the total 34 provinces having more than 1,000 errors. This affects the effectiveness of the services and satisfaction level in the community.
The implementation of e-government services requires quite a large investment due to the influence of factors such as the complexity of services, regulations, and the use of computer technology infrastructure. Besides, each city government implementing e-government is required to pass the development stage of e-government, with the maturity level of personalization which serves as a linear and consistent guide to transformation in the use of the technology starting with presence, semantic adoption, user modeling, procedure planning, and middleware layer [3]. However, the community/service is required to start with the stage of presence, urban information and interaction, transaction, and the last, e-democracy [4].
To ensure the successful implementation of e-government, these development stages need to be conducted consistently up to the top or final position through the use of appropriate strategy in line with the objectives. Moreover, an evaluation model to cover both aspects of technology usage and community approach at once is required but, until now, most of the maturity assessment models developed focus only on the capability of government agencies in using technology. Therefore, there is a need for the ones able to assess the maturity of e-government implementation focusing on the community [5]. In this study, an evaluation model covering both aspects of the use of technology and community value.
Evaluation helps technology service providers easily discover factors responsible for the success of e-government implementation and Balikpapan City was chosen as the object of this research due to the lack of assessment of the two aspects since the system was implemented in the city. A study of the effectiveness of the system in the city has been previously conducted by Yunita [6], but the model used tends towards its net benefits for the users using satisfaction as the indicator. The researcher thought a study was needed to identify the factors responsible for the successful implementation of e-government in the city from the perceptive of the use of technology and public value. This was supported by the findings showing development strategy and challenges of industrial cities in developing countries to transform economic activities due to decrease in production and global oil prices by changing public policies and regulations [7] followed by modernization of infrastructures and services to improve life quality of local residents.
This study aimed to propose an evaluation method to measure and determine the factors responsible for the successful implementation of e-government system adjusted to achieve its objectives and meet community expectations. The results are expected to be useful in increasing the success of the system within the Government Institution environment.

Research Method
The information system model first formulated by DeLone and McLean in 1992 identified six factors influencing its successful implementation, including system and information quality and their effect on the use and user satisfaction which also influences each other as well as on individual and organizational impacts. The model was updated by DeLone and McLean by responding to the strengths and weaknesses observed by several researchers and this led to the grouping of the variables of individual and organizational impacts as net benefits. These variables indicate the importance of the services as a contributor to IS success (shown in Figure 1).  [8] This updated model is not only used to measure the success of IS but also for the implementation of e-government. For example, it was used by Agrifoglio, Metallo, and Lepore [9] for e-court applications. This study made use of six variables which have been confirmed to be valid, however, further investigation related to the use of the D&M model in different e-government contexts, especially to assess the relationship between ICT and performance of government services by using a content management system is required. A study was conducted by Rana [10] through the development of a conceptual model to accept e-government in the public sector. The results showed the success level of the system through the integration of IS models by using intention to use and user satisfaction as the main measures through the use of 419 communities in several Indian cities to validate the model. Moreover, D&M model combined with the construct of Seddon which involved adding perceived ease of use and perceived risk. That proposed model focused on the level of technology use but the proposed causality construct was valid in measuring the success level of technology-based public services. In 2015, a study was conducted to measure the effectiveness of the system based on public value by Scott, DeLone, and Golden [11], and it started with the broad definition of net benefits. The researchers [11] proposed to integrate the theory of public value into D&M model by creating 30 items of scales to evaluate the e-government website. The concept provided a new understanding of the ways to evaluate government service activities by combining dimensions of efficiency, effectiveness, and social value. The net benefit measurement model [11] was adopted into a model later proposed as a basis to measure community value approach.
Several studies have proposed models which adapted aspects of government services and electronic-based service quality [12] They emphasized more on evaluating the service quality of the egovernment system from the perspective of the users by using the importance-performance analysis method. Moreover, the measurement of the levels of community satisfaction and trust in e-government services has been conducted using a modified D&M model [13]. The variable added for further validation was the relationship between costs related to the use of services to determine its effects on user trust and satisfaction and the results showed information and system qualities have positive impacts. Another study conducted by Weerakkody, et al [13] also proposed to understand user satisfaction as a measure of e-government services. However, the previous study also tested the system from the community perspective by adapting the D&M model which involved the addition of variables/constructs such as subjective norms, attitude, and trust [10] through a series of hypothesis tests. This was conducted through the use of a questionnaire to obtain primary data from the samples in several cities. The results showed the relevance between user satisfaction and attitude with their strong influence on the system. This study is a continuation of the successful implementation validation of DeLone & McLean [8] model in different contexts, in this case, e-government. The aim was to evaluate the success factors of egovernment based on the public value approach developed by using the D&M success model [8]. The variables used were system quality, information quality, use, and user satisfaction modified by adding content and format variables from the end-user computing satisfaction model [14]. However, the net benefits adopted from the public value model were also applied (Scott et al., 2015) by using variables of convenience, communication, information retrieval, well-informedness, trust, and participation. It was important to evaluate user satisfaction for quality due to its significant influence in adopting and spreading technology in a community [13]. Therefore, the measurement of the user satisfaction level proposed in this model also paid attention to aspects of the public value of net benefits and the use of technology due to the potential of community satisfaction to improve public services through the utilization of electronic facilities [15].

Research Methodology
This study was exploratory and the analysis of the cause-effect relationship conducted through the use of primary and secondary data collection methods. The primary data were obtained by distributing questionnaires in Balikpapan City while the secondary data was by interviewing related parties. The variables include the combination of IS D&M and EUCS success models. Moreover, the research produced empirical proof and descriptively presented the relationship between variables based on the quantitative analysis conducted on the results obtained using Likert scales in the questionnaire. Furthermore, the Structural Equation Model (SEM) Partial Least Square (PLS) was used for analysis with the construct tested initially for validity and reliability. The results obtained in numerical form was used to describe the relationship between the exogenous (influencing) and endogenous (influenced)variables.

The subject of the study
The main targets of this research were active users of e-government services in Balikpapan City. The questionnaires involving closed questions on the use of e-government website services (www.balikpapan.go.id) were distributed to 150 respondents. Each statement was compiled based on the indicators of each variable of the proposed evaluation model after they have been tested for validity and reliability.

Scheme of the study
The stages involved in this study are shown in Figure 2. It started with the literature study stage to identify the relevance of previous studies, followed by the design, developing a questionnaire based on indicators on each variable, and identifying respondents. This was followed by data analysis of the outer model in two stages of validity and reliability tests while the last stage involved the application of the model through structural model testing (inner model).

Data analysis
The first test conducted was the validity of the measurement instruments to determine the accuracy of the question indicators in the questionnaire. It was conducted twice for the discriminant and convergent and the rule of thumb of the convergent validity parameters are shown in Table 1. The second was the reliability test to determine the internal consistency of the measuring instruments at different times with the same symptoms. It is possible to conduct this test using Cronbach's alpha and composite reliability tests and the parameter is shown in Table 2.
The cause of the relationship between variables was measured using structural model testing and the hypothesis with the bootstrapping method. Structural model testing was used to determine the relationship between the constructs and their significance values as shown in the t-statistics. The hypotheses were tested at a confidence level of 95 percent or alpha of 5 percent when the T-Statistics value was greater than the T-Table value of 1.96. However, the coefficient (β) value of the independent variable (positive or negative) can be seen in the value of the path coefficient [16].

Structure of the conceptual model
Based on the information obtained from the literature studied and the objectives of this study, the structure of the conceptual model is illustrated in Figure 3. Moreover, the IS success model of DeLone & McLean [8] was modified by adding two variables of EUCS and implemented in this research [14]. Furthermore, the public value approach adopted the use of net benefits proposed by [11].

Hypothesis model E-government and updated D&M is success model
The IS success model has been validated by several previous researchers, [17]and [18], and has been adopted in identifying the factors contributing to the success of e-government implementation with results proven to be valid [19]. Moreover, an internet-based application was used by the users to search and make transactions, therefore, there is a need to provide systems with good qualities for information, system, and services. The hypotheses proposed are shown as follow: H1: Information quality is positively related to user satisfaction. H2: System quality is positively related to user satisfaction. H3: Service quality is positively related to user satisfaction.

H4:
User satisfaction is positively related to net benefits.

Format and content
Some studies proposed user satisfaction as a benchmark of success to evaluate the e-government system implemented [20] due to its high importance in determining the intention to use electronic services. However, user satisfaction, especially with the use of a computer, technology is influenced by several related factors. For example, [21]and [22] proved format factor to have a positive influence on the satisfaction of information system users such that the system is required to have an interesting display and easy to be understood. Another factor is the content variable [23], Hong et al [24] which indicates the production of the precise, appropriate, and relevant information required by the community. Therefore, the following hypotheses were formulated. H5: System format is positively related to user satisfaction.

Information Quality
System Quality

User Satisfaction Net Benefits
Service Quality

Updated D&M IS Success Model
Public Value Approach H6: System content is positively related to user satisfaction.

E-government and public value of net benefits
The study conducted by Scott et al [11] proposed the use of Theory of Public Value to expand the D&M IS Success Model by creating 30 items of E-government net benefits scales. This model shows it is possible to measure the effectiveness of the system from the community perspective based on the net benefits influenced by users' satisfaction. This was supported by another study which proved a strong relationship between the evaluation of the community satisfaction based on use and influenced by several dimensions [25]. Through the observation and analysis of previous research works conducted on this factor, five out of the total nine dimensions including well-informedness, communication, information retrieval, trust, and participation were used Mohajerani et al [26]. This is illustrated as shown in Figure 4.

Result and Discussion
The data analysis was conducted using Partial Least Square (PLS) based on structural equation modeling technique with the recommended minimum number of samples varying between around 30 -100 [16] or 10 samples in each path contained in the study model [27]. This study used a sample of 150 respondents with 6 paths making the minimum number of samples to be fulfilled to be 60 samples.
The measurement or outer model involved testing convergent and discriminant validity as well as internal construct consistency. As shown in Table 3, all the indicators fulfilled the convergent validity test due to the fact that their loading factors were above 0.60. Moreover, the AVE value of each construct was determined to show the ratio of its variants toward total variants among the construct items. The results obtained showed all the constructs to have values greater than 0.50 as shown in Table 3, thereby, fulfilling requirements of convergent validity. Furthermore, composite reliability test conducted showed all variables to have value > 0.60 and it was concluded that they are all reliable.
The discriminant validity test was also conducted to show the differences between the construct used from others as observed from the values obtained from cross-loading, or by comparing the AVE root for each of them with their correlation. The main principle of the test is to ensure different construct measurers are not highly correlated [16]. Table 4 shows all the root values of the AVE variable to be diagonally greater compared to the correlation values of others and this made them fulfill the discriminant validity requirements in the comparison test of AVE roots with the correlation value of latent variables.
The results of the structural model evaluation aimed to predict the relationship between variables by using R2 (R-square) for the dependent variables, the value of the path coefficient or t-values of each path to test the significance among constructs. In Table 5, the R-square value obtained for the dependent variable net benefits was 0.691 while user satisfaction was 0.759 classifying them into moderate and strong category respectively [28]. The path coefficients of the conceptual model proposed were through the bootstrapping procedure of PLS as shown in Figure 5. Moreover, the hypotheses were tested by comparing the T-statistic value with the T-table and the positive or negative influence of independent variables on the dependent ones was based on the coefficient (β) value. The results showed H1 hypothesis was not supported with t value = 0.976, path = 0.329, and p > 0.05, therefore, this means the

Net Benefits
Trust Participation information quality did not significantly influence user satisfaction. The system quality was found to significantly influenced user satisfaction and shows that H2 was supported with t value = 5.134, path = 0.000, and p < 0.05. Similarly, H3 was also found to be supported with t value = 4.156, path = 0.000, and p < 0.05 meaning that service quality significantly influenced user satisfaction. Moreover, H4 was supported as observed in t value = 24.184, path = 0.000, and p < 0.05 and this means user satisfaction had a significant influence on net benefits. However, one of the external variables of the EUCS model, H5, was not supported as seen in t value = 1.292, path = 0.197, and p > 0.05 and this concludes that format did not significantly influence user satisfaction while content was observed to have significant influence which means H6 was supported with t value = 2.061, path = 0.040, and p < 0.05.  The conceptual model proposed was modified with the variable path, especially the net benefits, adjusted to the dimensions of the people's perspective [11]. Thus, it is possible to observe differences in the influences of the information system success measured through the dependent variable of user satisfaction directly toward net benefits based on each dimension of the people's perspective value, as shown in Figure 6.  Furthermore, the results showed overall direct positive influences of user satisfaction on net benefit dimensions of the community's perspective. The values of R-squares produced in the second model trial revealed four factors including well-informedness with 50.6%, information retrieval with 58.4%, participation with 54.6%, and trust with 57.3% are classified in the moderate category and this shows that the community's perspective dimension could be explained by the independent variables of IS D&M success. Moreover, only communication was classified in the weak category with an R-square value of 41.7% and this means it could be explained by independent variables derived from the success model of IS D&M with the remaining 58.3% by others outside this study model. The significance level of the relationship path for user satisfaction on IS success with each variable of community's perspective

Conclusions
The results showed the format derived from the EUCS model has a positive effect on IS user satisfaction with 0.197 at a confidence level of 95%. Moreover, the R-square value of user satisfaction was found to be 75.9% indicating the possibility of measuring the success of the e-government system implemented using this concept as an independent variable. It was also discovered that from the community's perspective, the success of the system is influenced by that of IS implementation, which means the ability of government agencies to utilize information system/technology into an electronic service requires careful planning and organization. The measurement model proposed was highly required to evaluate the implementation of e-government services from the perspective of the use of technology and public value. Based on the model testing, a modified second concept was recommended for a detailed evaluation of each dimension from the community's perspective. The success level was also found to be influenced by community satisfaction in utilizing technology for communication and information retrieval to increase public trust and participation towards e-democracy. However, the model proposed in this study was discovered to be in the initial stage and needs further improvement in the future. This, therefore, requires the addition of variables with positive relationships outside the model to produce a better evaluation model to cover the technology and community values. This is expected to explain the relationship between the dependent variables by examining the values of R-square and path coefficient. Furthermore, a greater number of respondents are also required for the model to be developed as well as in its application to evaluating a particular public electronic service.