Does Performance Audit Recognise Improvements in Procurement Activities Leading to Enhanced Value for Money?

The vision of many Supreme Audit Institutions established to perform an audit, including performance audits associates to the positive impact they should have on the audited entities or public organizations through auditing. This impact is related to the performance improvement by achieving the tree E’s (economy, efficiency, and effectiveness) nonetheless, the role of performance audit in performance accountability is not less.

Lately, studies on performance audit are also focused on the role of audits as one of the instruments of enhancing the accountability of audited institutions and as a drivers of important management practices changes within these institutions (Morin, 2004, 2008; Reichborn-Kjennerud, 2013a). Performance audits have a greater purpose than just improving the operation of a public institution. Implementation of standardised public procurement regulations diminishes irregularities and preserves value for money thus impacting the economic growth. Standardised regulations reduce the risks of potential political influence and corruption scandals. The economic component of public procurement also contains the basic principle of public procurement “value for money”, in which case we realise that while carrying through a publicly funded procurement activity, we should strive to achieve the required outcome at the lowest possible cost (economy) (Raymond, 2008). However, practice shows that irregularities in procurement processes are frequent. These irregularities are related to the lack of coordination, transparency or competition in managing procurement processes (Hamza, Gerbi, and Ali; 2017).

Although procurement performance audits are popular throughout the modern world, in Kosovo, these audits are new, nevertheless, special importance is given. The National Audit Office (KNAO) within the Republic of Kosovo, which is also mandated to carry out performance audits, plays a vital role in improving the situation and reducing shortcomings in public procurement. Since 2010, performance audits have become an integral part of the KNAO and added value to its activities in addition to financial audits. Over time, performance audits have become increasingly focused, being divided into a procurement-focused performance audits or value-for-money audits. By conducting performance audits, the KNAO applied common practices in this area, same as many other Supreme Audit Institutions (SAIs). Regardless of the specific wording that specific country applies, performance audits assess how the Government is managing funds in terms of economy, efficiency, and effectiveness.

The Kosovo National Audit Office through procurement-oriented performance audits continuously identified shortcomings in the procurement processes undertaken by Public Institutions in Kosovo, focusing on the value achieved for the money spent. Shortcomings vary from the absence of clear procurement strategies continuing with inadequate identification of needs and procurement planning as well as drafting technical specifications not fully professional up to poor management of procurement contracts.

Given the growing importance of this type of audit and since the goal is to meet one of the three E’s (economy, efficiency, effectiveness) it is more than reasonable to raise the question of how well did performance audit carried out by the Kosovo National Audit Office perform and what are the impacts as perceived by procurement officers of audited institutions. This paper aims to provide answers to the following questions: What is the impact of performance audit on improving how procurement activities are developed? What factors influence the positive perception of performance audit and make it easier to undertake changes? Does performance audit result in changes within audit entities with a focus on enhanced/added value for money?

The main contribution of our research is to draw attention to the impact of performance audit in countries with high levels of corruption and irregularities, such as Kosovo. In addition, this research reveals audit entities key persons’ perceptions, on the factors that influence the undertaking of changes that derive from performance audits. To serve the purpose of the research, this paper is structured as follows: literature review, methodology used, analysis of the results of the survey conducted, and the conclusions given.

Researches focusing on the impact and effects of performance audit on organizational change in public institutions has been considered deficient by many authors. In this regard, the authors Grönlund, Svärdsten and Öhman (2011) argue that although analyses and studies on the impact of performance audit were conducted, few of them researched the current content of performance audits without indicating which changes from those performance audit reports were undertaken. Similarly, Loocke and Put, (2011) point out that the number of empirical studies aiming at the real outcome of performance audits is almost symbolic. Nonetheless, if we research this field more, we might notice that recently the number of researches when it comes to the impact of performance audit within public administration of different countries (Reichborn-Kjennerud and Johnsen, 2018; Alwardat and Basheikh, 2017; Raudla et al., 2016) or the research such as Effectiveness of Follow-Up on Performance Audit Recommendations (Umor, Zakaria and Sulaiman, 2020; 2016) is increasing.

Due to the difficulty of determining the real effect from the impact of performance audit, most authors rely on surveys conducted with officials and representatives of the audited institutions and their perceptions when it comes to the impact. Accordingly, one of the researches on the perception of performance audits undertaken by Reichborn-Kjennerud (2013b) revealed that performance audits by audited entities in Norway are perceived as useful. The author of this study suggested that the degree to which audited entities see audit reports as useful relates to or depends on the entities’ assessment of the report quality. According to her, the extent to which the auditors’ comments were considered during an audit process, as well as the changes undertaken following the performance audit contribute to this perception. Meanwhile, in a different study, Reichborn-Kjennerud and Johnsen (2018) demonstrated how the requirement of accountability by audit entities led to factual changes within the audited entities. However, different results were achieved in other countries. Consequently, Raudla et al., 2016 in research covering Supreme Audit Institution in Estonia found that, audit entities did not perceive performance audit reports as useful due to insignificant changes that occurred in the audited institutions.

Performance audits also depend on the role played by the auditors themselves and the audit institutions which face various challenges in applying new methodologies or practices in performance audits (Morin, 2016). In this regard, Morin’s research explained that, the influence of the French SAI was limited, the audits of the abovementioned institution had contributed more to the “attractive appearance” of reports and institutions than to the real or factual impact. (Morin, 2016). However, to have real effects, the auditor’s reports must be accepted by civil servants as high standard reports, when those reports have some impact on the process and when the latter agrees with the auditors’/SAI’s approach (Reichborn-Kjennerud, 2013b, 2014a, 2014b). Morover, Alwardat (2010) show that the view of audited institution about auditors’ expertise is also important. While, Justesen and Skaerbaek, (2010) find that keeping the audited entities politically accountable impose concrete changes.

Raymond (2008) in his research argues that Value for Money (VfM) is one of essential principles when it comes to public procurement. Therefore, every purchase or project should only be executed if there is value for money created (Burger and Hawkesworth, 2011). So, the main objective of a state public procurement system is to provide efficiency and value for money when it comes to the use of public funds considering that is has many implications for the whole economy.

VfM does not mean only choosing the lowest bid price but choosing the lowest life costs of that project, service or product. There are some challenges and opportunities to achieve VfM. The limitations or challenges are related to reliability of information, lack of comparability indicators, limited and untrained human resources etc. On the other hand, there are opportunities on enhancing VfM. Estache and Iimi, (2011) Purchasing models that claim to have competitive bidding and public-private partnerships can reduce unit costs and increase the efficiency of public funds. Transparency and accountability are other important factors that affect system efficiency. They are essential to alleviate poverty, promote economic growth and improve social welfare (Estache and Iimi, 2011). Also e-procurement is seen as an important way of higher level of efficiency (Croom and Brandon-Jones, 2007).

Like any other policy area, public procurement is governed by a specific framework that includes a set of rules that must be adhered to by each contracting authority. The legislative framework, however, does not provide those authorities with all means to ensure effective public spending. To do this, in addition to legal provisions and manuals, procurement officers must rely on information that is available from a variety of sources to tailor specific strategies that ensure effective outcomes. This relates to the market research before initiating a procurement procedure. Indeed, cost-effective through sound public procurement practices provide the opportunity to improve the quantity and quality of products and services as well as to ensure their timely delivery (OECD, 2018).

Since the focus of this research was to find the impact of performance audit on value for money, in other words in procurement, procurement officers and officials involved in executing projects (purchases) within the entities audited by the Kosovo National Audit Office were initially identified to send them a questionnaire as a method of data collection for this research. 261 public officials (civil servants/middle-level managers and Heads of the organizations) have been identified (those involved in one of the steps of the procurement cycle within the entities that have been subject to performance audits carried out by the Kosovo National Audit Office). After sending the questionnaire via e-mail to 261 respondents, a total of 86 responses were received whilst 175 others did not respond, having a response rate of 33%. The small number of responses relates to the period selected for research. The selected period is 2015-2020. There was a risk of not having the same employees working in the same position or institution or even if they had not changed position, most likely did not recall the performance audit thoroughly and did not respond.

To find out which institutions have undergone a procurement-focused performance audit, each performance audit report published by KNAO in the mentioned period has been analysed. As a result, 28 institutions were identified (15 local level institutions and 13 central level institutions).

To answer the questions raised in this research, a questionnaire containing 28 questions was designed having an earlier questionnaire designed by Kristin Reichborn-Kjennerud as a starting point which was prepared to answer her research question on the impact of SAIs. Furthermore, the abovementioned author relied on the questionnaire of Professor Danielle Morin from HEC Montréal, who had also researched this field, namely that of the impact of performance audit. In addition, the questionnaire is also based on the literature available for performance audit and the experience of one of the authors of this research as an auditor in performance department for a certain period of time (four years in Performance Audit Department and 9 years in Corporate Services within the Kosovo National Audit Office). The design of our questionnaire differs from that of the abovementioned authors because the latter contained questions on a procurement-focused performance audit.

For this research, the period from the moment when performance audits started applying until now has been selected. The selection of respondents was made since the public institution where they work was subject to a procurement-oriented performance audit.

Specific questions in the questionnaire were designed to answer the research questions. Initially, to see respondents’ perceptions on the impact that performance audit had on improving how procurement activities were developed, two questions were posed: What changes have been undertaken in your workplace since the performance audit? and to what extent do you see these changes as improvements?

As shown in the graph below, since the performance audit has been carried out, several changes have been undertaken in public institutions, mainly changes in internal control and risk management (changes in 31.4% of cases) which means that the management of these entities undertook changes in strengthening internal control and the way identified risks are managed to be in line with their objectives and strategies. Other significant changes undertaken by the entities after the performance audit was finalised are “changes in strategy, planning and/or management-oriented towards objectives and results” (changes in 30.2% of cases) which shows that the management of the audited entities has reaffirmed areas for improvement (identified by performance audits) and taken action to strengthen the organization and achieve the desired objectives and outcome. And, last but not least “changes in laws and regulations” (with changes in 20.9% of cases) prove that the entities have taken these actions to facilitate the organization/entity to achieve the objectives as quickly and in the best way as possible.

In order to see how much did these changes had an effect, as illustrated in Table below, we have also posed the question mentioned above, from which answers we can conclude that “The changes in internal control and risk management” (15 respondents rated with quite a lot or 17%) as the step/change which has contributed the most to the improvement. The step/change which was least (according to the respondents) seen as improvement was “dismissals and changes in the workplace” (with 38 respondents or 44%).

Undertaking a series of the actions listed above after the performance audit in the relevant entities is completed cannot lead to anything else but affect improving how procurement activities are developed, especially the changes undertaken in, laws and regulations including changes in a strategic and planning aspect which result in better preparation and planning of the procurement process making other steps easier to follow, and as a final result there is no other way but to have enhanced value for money in those purchases no matter how small they may be.

Further on, to find the factors which have an influence on the positive perception of performance audit and make it easier to make changes, we relied on the data deriving from the question posed in the questionnaire: To what extent do you agree or disagree with the following statements regarding the performance audits performed by the Kosovo National Audit Office (KNAO)?

As shown in Table 2 below, 38% of respondents fully agree that performance audits contribute to improvement within the entity, and 48% of respondents or 41 of them disagree at all and state that “Instead of contributing to improvements “KNAO worsens working conditions in audited entities”.

Furthermore, 35% of respondents think that KNAO is an important institution promoting transparency and preventing fraud. Last but not least is one of the factors that surely has an impact on enhancing perceived usefulness from performance audit, including the fact that according to respondents, KNAO does not worsen working conditions, which means it contributes to improvements. All this approach creates the easiest way to undertake change.

Out of approximately 50 performance audit reports published by the Kosovo NAO so far, around 12 procurement-focused and project management performance audits have been identified. These performance audit reports cover the period of 2015-2020. Institutions subject to these audits were: Ministry of Health (4 performance audits), Central Procurement Agency (2 audits), Ministry of Infrastructure (2 audits), Ministry of Education, Science and Technology (2 audits), Municipality of Prishtina (2 audits). Many other institutions (e.g., Ministry of Internal Affairs, other municipalities) are involved in one (1) performance audit.

Not all recommendations exclusively cover the public procurement cycle, nonetheless, the lion-share of recommendations covered some of the steps of the procurement process. The analysis of these reports shows that a total of 56 recommendations were given covering the steps (cycle) of procurement. Public procurement-oriented performance audit reports finalised and published since 2015 until the first part of 2021 are around 9. It is worth noting that audit findings and recommendations which were not related to procurement were not considered for analysis in this research.

Some of the main findings of the audits conducted are as follows: lack of a (clearly articulated) procurement strategy (without procurement strategies, procurement activities cannot be planned and objectives cannot be achieved), followed by inappropriate assessment of needs (without knowing the real needs, the result may be higher costs, poor competition), non-compliance in all steps from the stock situation, the required quantity of items following with the quantity which is distributed (such a situation results from poor monitoring). The other finding in procurement is related to ineffective management of contracts while processes in this area had room for improvement (as a result of lack of contract management plans).

In order to see the correlation between the findings in the performance audit reports and the perceptions of the officials within the institutions that were subject to audit, the hypothesis is raised that proper market research as a step in the procurement process directly affects the increase in value for money in procurement process impacted by performance audits.

Value for money means purchasing a product or service at a lower cost and meeting the required specifications. Since it is quite hard to measure direct value for money and since the purpose of our research is to look into the impact of performance audits on value for money then the dependent variable is the one representing the impact of audits focused on the procurement process and is constructed based on the question: Could you point out the extent to which the audited entity has made changes based on the evaluation results in the performance audit report? This variable is ordinal as respondents responded in options 1-very little to option 5-quite a lot.

The independent variable is constructed by considering the questions: Do you collect information on market characteristics and recent market developments and document them?; Do you collect information on potential suppliers, their products, their prices and costs; Do Procurement Officers within your institution discuss with other procurement officers within public sector including clients who purchased similar products or services to better understand the market and its participants; If the respondent answered positively to one of these questions, then it is considered that the institution applies market research before each purchase otherwise the institution does not apply market research at all.

Ordered Logistic Regression (OLM) was used to test this hypothesis. This model is used when the dependent variable is expressed in an ordinal scale since Likert-type questionnaires were used in this research (Gujarati, 2011, p. 180). These models indicate the probability that the respondent belongs to one category.

The results show a statistically significant correlation between performance audit and market research as a vital step in the procurement process. This is confirmed by the regression coefficient which is positive (.9527777) and statistically significant, where its value p = 0.021 is less than 0.05. Based on this result, we accept the hypothesis raised and we can conclude that proper market research is a step that must be applied to increase or add value for money in the procurement process and this is related to performance audits considering significant changes undertaken by the audited entities.

As specified in the introduction to this paper, impact through audit within audited entities or public organizations is also embedded in the vision of many Supreme Audit Institutions. Through this paper, we have strived to recognise the impact of performance audits focused on procurement and carried out by the Kosovo National Audit Office to improve or enhance value for money, which means saving already scarce public funds.

We managed to demonstrate that performance audits generally have a positive impact. From the analysis of audit reports carried out by Kosovo National Audit Office for the period 2015-2020 - value for money or procurement audits as well as whether the external audit or specifically a procurement-focused performance audit has affected in enhancing value for money in these institutions, it turns out that most of the institutions that were subject to audits encountered obstacles in enhancing value for money in their purchases through procurement since two of the main steps of the procurement cycle identified throughout the research (identification of needs and procurement planning - market research and contract management) were not met to the desired level.

Furthermore, from the administered questionnaire, for the questions related to market research, the results show that there is limited knowledge by the officials whose duty is to provide good planning for market research methods. Whilst, concerning the changes undertaken as a result of the audit, and that some of those actions are quite significant, it shows that the management of the audited entities has re-confirmed the areas for improvement identified by the performance audits and has taken actions to strengthen the organization and achieve the objectives and the desired outcome.

On the other hand, the results of the hypothesis show that proper market research is a step that needs to be applied to enhance/add value for money in the procurement process and that performance audits in institutions in Kosovo have had a positive effect in undertaking changes which lead to enhanced/added value for money.