Measuring results of performance audit

This paper has been prepared for the International Seminar on Effective Performance Auditing: "Mission Impossible?" in den Haag, May 2002.

Introduction

Audit is not an end in itself. However, it is an indispensable part of a system aimed at disclosing deviations from accepted standards and violations of the principles of legality, efficiency, effectiveness and economy; so that actions may be taken.

When audit is considered to have a corrective function, it is of course obvious to ask: What are the results of our audit reports? Do the reports have any effect on public sector procedures and products? How do we – auditors – ensure that our work meets the expected results? and how does it affect the audit work when no effect is seen?

Today, it is not enough to justify audit work by postulating that the audit is a preventive function in itself. Any Parliament who appropriates resources for a Supreme Audit Institutions expects value for money.

For that reason, the NAOD has tried to document the results of the performance audit examinations during the past 4 years. This paper describes the methodology used and discusses the problems in relation to this documentation.

The purpose of measuring effects

The NAOD measures the effects of the reports by recording the changes that takes place in the ministries, institutions agencies etc. The purposes of these measurements are

• to obtain an impression of the improvements and measures of effectiveness that have been initiated on the basis of an audit report,
  
• to ensure that the subjects and methodology of the reports are chosen within the areas that are expected to have an effect on the state administration, and
  
• to focus on the procedures of following-up on reports, so that the individual audit case may be followed and closed as soon as possible to the benefit of both NAOD and other involved parties.

Planning and implementing performance audit

The purpose of a report is to inform the Parliament’s Public Accounts Committee of all significant financial and/or important matters. The NAOD presents 15-18 reports annually.

When choosing subjects for the NAOD-examinations emphasis is placed on materiality, risk, topicality and visibility. The subjects may concern all significant financial as well as other important matters.

For areas of great financial importance, examinations should be carried out within reasonable intervals. This may concern; the overall financial management in an area, basic activities or the examinations may influence one or more sub-areas.

Risk is primarily focused on whether significant errors and deficiencies may occur. Risk assessment is based on auditor’s experience of the audit area. When assessing risk, it also has to be taken into consideration whether the NAOD may be liable to criticism for not having carried out performance audit examinations.

If there are areas or questions of particular interest to the public, an audit examination may also be initiated. Topicality is determined on the basis an actual assessment of the value of the audit result in the individual institution, other institutions in a similar situation, and political and media awareness.

Visibility is encouraged by planning and carrying out audits in close communication with the institutions.

When planning the annual year’s performance audit examinations all the areas of the state accounts are reviewed. Every third year a total analysis is carried out, whereas only major changes are reported in the intermediate years. Reviewing an area is based on the following aspects:

• financial situation
• previous reports and important memoranda to the PAC
• the audit form
• organisation – structure and complexity
• political/public awareness and topicality
• cross-ministerial problems
• remarks from the financial audit
• possibility for major expenditure programmes

Before making a report, a preliminary examination is made in order to obtain sufficient information about an area, so that an adequate decision may be made of whether or not to initiate an examination.

In the preliminary examination, questions, methodology and expected effects are considered. The institutions, which are included in the preliminary examination, are informed of the preliminary examination, the purpose of it etc. The preliminary examination results in an audit-planning memorandum describing the background of the preliminary examination. The audit objective shall be presented and known problems in the area shall be described. Furthermore, a short summary of the conclusions of the preliminary examination shall also be included and expected results and effects of the report as well.

Hereafter, the examination is carried out and as part of the quality control, the report is forwarded to the ministries for hearing. Once the minister’s answer is forwarded to the NAOD, the factual comments are incorporated in the report, whereas differences of opinions between the institution and the NAOD are considered. Hereafter, the NAOD works out the final report.

Reporting the performance audit

The NAOD presents its report to the PAC and the PAC com­ments on them if necessary; hereafter the reports are submitted to the Parliament and the mi­nister concerned. The minister must respond to the reports within four months, describing intended actions. The minister has to respond to a report because the ministers are subject to the Act of Mini­sterial Responsibility.

The following chart illustrates the relations between the NAOD, the PAC, the mini­stries and the Parliament in the contradictory process in Denmark.

Once the Auditor General has received a minister’s reply to a report, the Auditor General has one month to consider the reply and to report his views to the PAC in a follow-up memorandum.

The NAOD follows a case by way of a follow-up memoranda until a case is considered closed – in other words, until the institution has shown intent to act on the identified recommendations. Hereafter, the financial audit department of the NAOD follows whether the initiatives are implemented. The ministers’ responses and the Auditor General's replies are reproduced in the PAC’s final report on the state accounts. According to the Constitution, the final report on the state accounts forms the basis for Parliament’s approval of the state accounts.

Baseline for measuring performance audit

The National Audit Office of Denmark has carried out measurements of various reports. The measurements have been made public in the annual report. The concept is self-evaluation and the measurement concerns the reports that have been presented for the Parliament’s Public Accounts Committee in the year prior to the presentation of the annual account. This period has been chosen, as the effects of the reports in question are obtained almost at the same time the annual report is made.

Measuring effects takes its point of departure in the objective of a report. When reviewing the follow-up memoranda it is possible to measure the effects that have been obtained in the first four months after presentation of a report. It is not expected that all effects have been achieved at this point in time, but it is expected that the institutions consider the recommendations given in the reports and that they respond to the minister accounting for carried out and on-going work. In other words, it is expected that the ministry has taken actions based on the identified recommendations.

The National Audit Office of Denmark groups effects in the following categories:

To improve information to the Parliament for instance by way of improving the validity of the information to be used for the finance act, orders and information to the Parliament

Obtain authorisation by means of special legislation and the budget etc.

Improved management information by means of making tools for managing so that the management may obtain faster and better information and analyses when making the decisions.

Improved management basis by means of guidelines, policies, management, targets, strategies and performance based contracts.

Improved account and control procedures by means of establishing controller units, working out audit instructions or control and accounting procedures.

Improved use of computers for instance by means of approval and control procedures.

Improved basis for efficiency and efficiency analyses by means of data for production and use of resources and examination of the obtained effects.

With the NAOD vision in mind, the categories were chosen to improve an effective state administration. The reports have been chosen independently of the content, that is, without consideration for the specific purposes of the individual reports.

Categorising the effects is merely based on an assessment as some of the categories are very closely linked. In other words, an affect is only counted in one category even though the effect may cover other categories.

The concept of effect has been divided according to the extent that it has been met. The effects may thus be: completely met, partially met and not met yet. Completely met effect reflects that the work has been implemented, for instance, that an instruction for an institution has been reviewed significantly. A partially met effect reflects that the work is ongoing, for instance, that a group has been established with the aim of working out a guideline for e.g. targets and performance based contracts. An effect that has not been met reflects the unresolved work, which might have been postponed due to lack of resources.

The effect measurement limits itself from measuring a number of identified potentials for instance:

• whether launching an examination may have an effect, as disclosed problems may be solved in the period of examination. The presentation of the report may subsequently be the reason to the effect,
  
• whether the initiatives of the institutions subsequently resulted in savings/increased revenue/limits waste,
  
• whether the report has effects on other institutions’ administration, and
  
• whether the satisfaction with the service increased for citizens.

These potential improvements are a significant part of the effects which are a consequence of the NAOD’s audit products, however, they are difficult to measure. Therefore, these effects have not been included in this self-evaluation.

Measuring results of performance audit

Measuring the effects of the reports have primarily been carried out by quantitative methodology. However, a more descriptive and qualitative methodology has also been applied as supplement.

The quantitative measurement of effects

The most recent effect measurement has been carried out on the basis of 18 reports presented in 2000. This period has been chosen, as the effects of the 2000-reports have been acquired in 2001. Categories and grouping of effects are shown in table 1.

Table 1. The effects in the 18 reports presented in 2000 have been grouped in expected effect complete, partial and no effect

Categories of effects	Expected effects	Follow-up memoranda Obtained effects		No effects
		Complete	Partial
	----- Number -----
To improve information to the Parliament	7	7	0	0
Obtain authorisation	1	1	0	0
Improved management information	5	0	4	1
Improved management	26	15	9	2
Improved account and control procedures	27	12	11	4
Improved use of computers	1	0	0	1
Improved basis for efficiency and efficiency analyses	10	0	9	1
In all	77	35	33	9
2000-reports in all, %	100	45	43	12
1999-reports in all , %	100	21	51	28

Table 1 shows that the NAOD expected an effect 77 times in 18 reports. 1/3 of the effects were in the category “Improved management” and 1/3 in the category “Improved accounting and control procedures”. Measuring the 2000-reports showed that 45% of the effects were completely met, whereas 43% were partially met and 12% had p.t. no effect.

When measuring the 1999-reports – 21% of the effects had been completely met, 51% partially met and 21% had no effect. It seems as if the effects of the 2000-reports have been achieved faster compared to the results of the 1999-reports.

This may be ascribed to the fact that the NAOD has improved its communication process with the institutions, for instance, making the expected effects more clear and straightforward. However, contingencies may also influence the result.

Effects in the categories “improved management” and “improved accounting and control procedures” are often to be found, as 53 of the 68 effects have been categorised in these two categories. Effects are rarely found in the categories “Obtaining authorisation” and “improved use of computer” – only 2 effects have been categorised here. The additional categories include between 5-10 effects each.

The number of expected effects in the reports varies from 0-8. The variation is large because the purposes of the various reports of the NAOD are very different. For instance, illumination of a sale compared to illumination of a policy in various ministries.

In 7 of the 18 reports, all the effects had been met of all the identified improvements areas. This is a significant difference compared to the 1999-measure­ments where only one of the reports was completely met.

This is partly because some of these problems concentrate on already implemented actions, for instance, construction of a building. Other reasons may be that the institutions, when making the reports, focus on following-up on the critical problems and therefore the institutions have been well under way when the report was presented.

Lessons learned show that effects in the categories “significantly improved information to the Parliament” and “obtaining authorisation” have quickly been implemented. It takes longer time to obtain effect in the categories “significantly improved basis for efficiency and effectiveness analysis” and “significantly improved management information”. This is, among other things, because implementation of a new steering system is very time-consuming.

The descriptive analysis

The NAOD has categorised five effects in the report “Home Service Control”; one in the category “improved accounting and control procedures” and two in “improved information to the Parliament” and two in “improved management”. After the report was presented, the Danish Commerce and Companies Agency ensured that the control measures were improved and better organised. The Danish Commerce and Companies Agency also ensured an improvement of the quality of the administration in the home service companies. Based on the report, the Ministry of Business and Industry worked out a socio-economic analysis of the financial consequences of the home service arrangement to be used when assessing the effect of the arrangement. If the socio-economic effect analysis is disregarded, the case was finalised just after presenting a follow-up memoranda. An explanation to this may be that the examination was implemented in a close and beneficial dialogue with the Danish Commerce and Companies Agency, so that the agency on a day-to-day basis was able to initiate new procedures, control procedures and targeted examinations.

In the report on “subsidy to free schools”, 8 effects were grouped into three categories: two in the category “improved management” and four in “improved accounting and control procedures” and two in “improved basis for efficiency and effectiveness analysis”. The report resulted directly in improved management, and improved the basis for efficiency and effectiveness analysis. At some schools, registration was introduced and the Ministry of Education carried out a review of selected circumstances in the schools and established the basis for determining the level of subsidies. Hereafter, the ministry launched some initiatives and changes, but the effect hereof will not be seen until later.

In the report about “DR (the Danish Broadcast corporation) and TV2’s efficiency and Public Service Company” no improvements were outlined and no major problems were disclosed in the preliminary phase. The NAOD wanted, however, to make a report in this area, which attracted great media interest, because a new political perennial media policy agreement had to be entered.

Discussion on the concept of analysing effects

Evaluating effects or results can be assessed from two different perspectives. Firstly, when an assessment is made of the effects of the performance audit reports the basic assumptions of the measurements should be realistic. Secondly, it is important to consider exactly which criteria to choose when measuring effects. The criteria for this evaluation are the well-known concepts from social science: validity and reliability. However, the criterion of relevance is also used in the present discussion.

The evaluation of the effects is based on three basic assumptions, see table 2. The principal assumption is that there is a causal relationship between the audit observation as a determine variable and the effect as a response variable realised by the administration. The measurements of these effects are an interpretation more than it is conceived to be a causal relationship in a social science sense. In order to get a more precise understanding of the effect of the audit results several case studies are required. However, studies that extensive are far beyond the possibility of the NAOD at present time. It should also be added that it is important to be aware that other factors may also influence the results.

Table 2. Basic principles in the measurement of audit results

Principles of effect measurement	The choice of the NAOD	Disadvantages
Causality between report and effect	There is a causal relationship between the reports of the NAOD and the effects in the institutions, and that this relationship in confirmed by the institutions.	Is this subjective opinion correct? Or is it two mutually independent measurements – a measurement of the expected effects of the reports and a measurement of actions in institutions?
Accordance between the effect categories and the vision, targets and strategies of the NAOD	The NAOD has chosen the categories according to the products the NAOD usually focuses on, and these are in accordance with the vision, targets and strategies of the NAOD.	Still significant categories are still missing, for instance, the NAOD reports that contribute to: savings, limits waste, increases income, increases citizen satisfaction citizen with services, during the process of a report has improved the effort
Self-evaluation	The NAOD has based its effect measurement concept on self-evaluation. In other words, an independent team in the NAOD measures the effects of their colleagues’ reports, hereby obtaining a certain degree of objectivity.	Is it possible that self-evaluation may be the basis for an objective measurement of effects, or will measurements of this kind always include some kind of inherent subjectivity?

The evaluation of the effects should be useful in the decision-making of the NAOD. The results of the effect measurements are not available until after the audit has been planned, and consequently the results may be obsolete and strategically irrelevant because the operational audit objectives for performance audit could have changed. Even though the effect measured is implemented in a valid and reliable way.

Even though the evaluation methodology is described carefully and the assessment criteria are defined with meticulous care, the results are per definition biased; the results of the effect measurement may never be as reliable as assessments from an independent evaluator.

In Denmark, the measurements of the effects are first and foremost based on ministers’ delivering their replies to the PAC in where the measures and considerations which the report has given rise to are presented. The information contained in this statement is of course a subjective presentation of the solutions and measures to the problems brought forward in the audit report. This influences the validity and the reliability of the information. But when the statements have to be published it should be expected that the information is sustainable and resistant to critical review.

It should be emphasised that it is connected with considerable difficulties to construct categories of effects, which are not overlapping and exclude each other see table 3.

Each effect has only been recorded one time in one category (one to one relationship) even though an effect could be classified in several categories (one to several relationships).

Table 3. Methodology for measuring effects in the NAOD

Methodology	The choice of the NAOD	Disadvantages
Measurements after 4 months – follow-up memoranda	The applied data material has been made 4 months after the report has been presented on the basis of a follow-up memoranda about the minister’s account for on-going and future work. The NAOD realises that 4 months is not a very long time for the minister to follow- up on the reports, and the NAOD has therefore divided the effects into the categories “completely met” and “partially met”.	It may be questioned whether the statement of the minister rests on merely a statement of intent and in fact very little on implemented follow-up.
The categories do not clearly prevent reciprocal exclusion and the effects are only counted one time in one category.	This is not a conscious choice but it is founded in the NAOD’s experience with effect measurements. Criticism is sought accommodated by future developments of the effect measurement concept.	The objectivity is reduced as the measurement is based on an assessment

It is a professional obligation of any Supreme Audit Office to be aware of all the difficulties in measuring the effects of the audit observations and to acknowledge the difficulties in using the methodologies. It is the only way to improve measuring the effects of performance audit reports.

Future developments

In 2002, the NAOD will adapt and further develop the concept for measuring effects in various areas.

First of all, the NAOD will adapt the definition of the effect categories additionally, as the effect categories do not sufficiently exclude each other. In some cases, the effects of the reports may be placed in several categories, and consequently the most appropriate category is chosen. This choice is based on an estimate and therefore not objective. Furthermore, it should be considered whether some categories should be ruled out as these only rarely obtain any effect.

Secondly, the NAOD will initiate a second measurement of the already measured reports two years after the follow-up memoranda have been presented. Today, the concept of effect measurement covers a period of 4 months after the report has been presented and this period is very short. Therefore, a re-measurement has been decided in order to measure the long-termed effects of the reports. The NAOD has chosen the timeframe of two years after the follow-up memoranda have been presented as the effects no longer may be claimed to be of the report. Measuring after more than two years after the follow-up memoranda have been presented will allow other factors to have influenced the result.

Thirdly, the NAOD will increase focus on effect measurements in the audit planning memoranda. At present time, the audit planning memoranda includes the expected results and effects which have to be included in the report. The intention is to increase the number of expected effects in the audit planning memoranda and to describe these as clearly as possible, so that the expected effects may influence the reports even more.

Finally, the NAOD will measure the long term effect every year by means of a case study of an institution or an area that has been subject to a number of audit actions for instance a report or several. The concept for measuring the long termed effects has not been established yet.

Conclusions

NAOD has tried to look nearer home by starting to evaluate the effects of its own activities. During this process, NAOD has recognized that there are several problems in establishing such a system. In this paper, the most important inconveniences by such an evaluation system are presented and discussed. Recognizing the fact that it is difficult to get a valid and reliable answer to the question about the effect of the performance audit, the NAOD has taken complementary initiatives to improve the quality of the performance audit results.

Firstly, the NAOD reports have been evaluated by well-known social scientists from Danish universities. Secondly, a consultant has made qualitative interviews with senior management in the state administration, our primary clients and principal user of the audit reports – after the PAC.

All three elements added together have improved the decision-making by the top management in the NAOD and established a pragmatic learning process in the organization.