Planning of choice of methodes for performance audit

Lessons learned of in the National Audit Office of Denmark by Rolf Elm-Larsen¹⁾

Table of contents

I. Introduction

II. The Danish approach to performance audit

A. Conceptual system

B. The case study: National Working Environment Authority

III. Economy or prioritisation of resources

A. Conceptual system

B. The case study: Prioritisation of resources of the National Working Environment Authority

IV. Efficiency

A. Conceptual system

B. The case study: Efficiency of the National Working Environment Authority

V. Effectiveness

A. Conceptual system

B. The case study: The outcomes of the activities of the National Working Environment Authority

VI. Conclusion

Literature

I. Introduction

1. Auditing standards for performance audit focus on the planning, particularly the selection of areas according to the principles of materiality and risk²⁾. However, the subsequent stages of the planning process, where the analytical approach and the methodology and pertaining techniques are chosen, have not attracted much interest. These subsequent elements of the planning process are crucial since this is where the choices are made relating to the quality of the information generated by the audit study. If the incorrect methods are chosen, the auditor’s conclusions will be just as irrelevant as if he selects areas or subjects that are immaterial and present little risk to a decision-maker. If the worst comes to the worst, the wrong choice of methods may lead to the auditor drawing erroneous conclusions from a correct data material. The choice of methods thus involves a considerable audit risk.

2. This article deals with the problems associated with that part of the planning phase in which the audit approach and audit methodology are decided. This part of the planning is crucial in relation to the selected subjects. The identification of an area or a subject as material and/or risky is insufficient in itself if the subsequent planning phase does not result in the choice of a relevant audit approach that generates the information that is of importance to the decision-maker. To get this information created it is crucial to choose correct methods and techniques.

3. This article is in two parts. The first part is a presentation of how the National Audit Office of Denmark (NAOD) has developed a concept for performance audit that ensures stringent coherence between the audit approach and the choice of audit methodology and techniques.

The second part of the article is a case study illustrating the theoretical problems presented. The example chosen is the NAOD report on the National Working Environment Authority’s management of its inspection function.

4. The article also reflects the development strategy of the NAOD concerning performance audit. The NAOD has emphasised further development of methodology and techniques relating to the concepts of economy, efficiency and effectiveness and the associated management aspect.

II. The Danish approach to performance audit

A. Conceptual system

5.Figure 1 illustrates the conceptual starting point of performance audit of the NAOD.

Figure 1.A model of performance audit linking the basic concept in performance audit to different types of governance

The figure illustrates the model on which performance audit as carried out by the NAOD is based.It makes clear the correlation between central concepts within performance auditing, management levels and the environment of the model.

The foundation of the model is that all public activity is a productive process which consumes resources/factors of production for producing both tangible and intangible products or services.An alternative interpretation of the model is that it reproduces the value creation chain of public activity.This is also a general feature of other models of performance audit.³⁾

The crux of the model is the production process, which transforms input in the form of physical resources into output:products and services.The resources are made available to the entity through the financial resources that appear from budgets/financial statements.The output must satisfy the performance targets set out or have an impact, which is designated by a term from evaluation research:outcome. For part of the public sector economy, the intention is not immediately to generate economic activity, but rather directly to generate an impact.Thus, transfer income aims at generating a redistribution of the incomes or an impact on the income distribution of society.

Production processes can be managed by the use of one or more types of managerial tasks.The essential object of budgetary management is the implementation of and follow-up on budgets and financial statements.When the management concentrates on the composition and dimensioning of resources for inclusion in the production process, resource management is applied.When the management focus is on the output or the activity, the management process is called activity management.The last management task to be performed is performance management, the object of which is the achievement of the objectives by the activities or their correct impact.

6. By focusing on the connection between the individual elements of the production and financial management process, the auditor assesses whether sound financial management has been taken into consideration.

If the auditor performs a comparison between the physical resources procured (input) and budget figures/financial statements, the auditor can evaluate the ability of the entity to husband its resources or exercise economy.

The relationship between factors of production included in the production and the volume of the production measured in physical units describes the ability of the resources to generate an output and is called efficiency.The output may be either products or services, such as teaching activities, casework or the like.

Output related to outcome is an assessment of the ability of the production to realise the targets or results set out or to achieve the desired impact.This is a measure of the effectiveness of the output.Effectiveness can also be measured as a success rate in the form of the number of production units that satisfy the quality requirements specified.

Cost effectiveness is measured as the ability of the resources to generate an outcome or achieve the performance targets set out or achieve the desired impacts in the environment.

7. The conceptual system of the model allows decomposition to be carried out.The cost effect is generated by three sequential elements, viz., economy, efficiency and effectiveness.In other words, the total cost effectiveness is expressed as the product of the factors of economy, efficiency and effectiveness.

The correlation between economy and efficiency is shown graphically in Figure 2, which corresponds to Farrel’s measurements⁴⁾.

Figure 2.

 

It appears from this that the total efficiency is constituted by the product of technical efficiency (0B/0A) and allocative efficiency (0C/0B).

Figure 3 illustrates the correlation between production and costs.

Figure 3.

The movement from point C₀ to C₁ expresses the potential for improvement in total efficiency, which means that the relation C₁/C₀ also expresses the relation 0C/0A in Figure 2.

However, the costs can be reduced further from C₁ to C₂ if the potential for improving effectiveness is exploited, that is, if the production is targeted at the outcomes and thus reduced from Y₀ to Y₁. The relation C₂/C₁ is thus an expression of the effectiveness provided that no quality failures have occurred in connection with the change in production.

In general, the following equation expresses cost effectiveness:

correspondingto

costeffectiveness = effectiveness * efficiency * economy

8. Decomposition makes it possible to diagnose any problem sites in the process.An entity may thus have an optimum production, but not necessarily the production with the ideal impact.Conversely, an entity may well have low productivity of products and services, but with just the right impact.

A major advantage of this model is that it is based on well-established economic theories.That makes it a robust tool providing clear audit evidence.

9. The model presupposes the existence of an outcome chain illustrating how the consumption of resources and an effort lead to the desired activity and outcome.The outcome chain clarifies the existing cause-and-effect relationships in the administration and assists in determining a number of measuring points in the process so failing outcomes can be identified.The outcome chain also implies that, in terms of time, consumption of resources comes before output, and outcome follows output.

The cause-and-effect relationship of the outcome chain also means that the auditor must pay attention to external influences not attributable to the effort made.The calculation of efficiency and effectiveness should then be adjusted for extraneous variables getting mixed in with the measurement of the efficiency or the effectiveness.

10. The model of Figure 1 shows the ways in which performance audit can be carried out in practice.There is an analytical division of the critical financial auditing into three aspects (economy, efficiency and effectiveness) for practical and systematic reasons.

By examining how resources have been gained the auditor can assess economy.A comparison of the production and the factors of production spent allows efficiency to be evaluated.Finally, effectiveness can be assessed by considering whether the public entity did in fact achieve its declared goal (goal achievement) and at what consumption of resources (cost effectiveness).With the model as a starting point, the auditor can also assess how the management of the public entity has chosen to manage the entity.

According to the Danish way of carrying out performance audit, and for practical and systematic reasons, the critical financial factors are divided into the following three aspects, which are well-known from all definitions of performance audit:

Economy:Have products and services been acquired in the most economical manner in view of quality, quantity, etc.?

Efficiency:Is there an optimum relationship between resource application and yield?

Effectiveness:To what extent have the goals been reached – the outcomes triggered – that were intended with the particular grant?Is there an optimum relationship between outcome and the consumption of resources?

B. Thecase study: National Working Environment Authority

11. The Danish Working Environment Act stipulates that the object of the National Working Environment Authority is to contribute to the creation of a safe, healthy and individual developing working environment at Danish workplaces. This is achieved through inspection of the individual workplaces in Denmark. The National Working Environment Authority is an institution under the Danish Ministry of Labour and has a central unit and 15 districts. In 1996, the Minister of Labour presented an action programme for a clean working environment by year 2005, which formulated a vision for the preventive working environment effort.

12. The problem to be examined by the NAOD study was that the number of accidents at work had not dropped considerably but had stagnated despite a declared objective of reducing the number of accidents at work, cf. Figure 4. The question was how the National Working Environment Authority could contribute to a reduction in the number of accidents at work and occupational diseases through its inspection activity.

Figure 4.Development in the number of industrial injuries reported to the National Working Environment Authority 1990-1999⁵⁾

13. A calculation made in the report of the National Working Environment Authority, “The Bill for Industrial Injuries” (1994), estimates that the implications to the national economy of accidents at work and occupational diseases approximate DKK 23 billion a year in direct costs (such as hospitalisation, pensions, lost earnings). This corresponds to 2.4 per cent of the GDP.

14. The model applied for analysing the National Working Environment Authority appears from Figure 5.

Figure 5. Model for analysing the inspection activities of the National Working Environment Authority

The figure illustrates how the activities of the National Working Environment Authority can be decomposed into activity-determined production in the form of field time and a result in the form of a reaction, such as orders, guidance or instruction to the enterprises.

15. Table 1 shows the development in the total number of man-years, field time, field time spent on meetings/ instruction, number of orders and number of guidance sessions in the general field in the period 1996-1999.

Table 1. The development in the total number of man-years, total field hours, number of
reactions from the National Working Environment Authority in the period 1996-1999

Year	Man-years, total	Hours Field hours	Efficiency	Orders	Guidance sessions	Reactions, total	Number of reactions per field hour
1996...............	332.9	83,395	250.5	9,236	6,696	15,932	0.19
1997...............	363.4	89,459	246.2	10,759	8,337	19,096	0.21
1998...............	384.3	103,014	268.1	10,165	9,419	19,584	0.19
1999...............	384.5	102,133	265.6	12,350	7,754	20,104	0.20

The table shows that the growth in field hours (22 to 24 per cent) is larger than in man-years (15 per cent). This results in an increase in efficiency of 6 per cent measured as field hours per man-year.

The increase in field hours did not result in more reactions from the National Working Environment Authority. On one hand, the number of orders rose by 34 per cent, while the number of guidance sessions only rose by 16 per cent. An unweighted aggregation of the two types of reactions also shows that the rise in the number of reactions in relation to the number of field hours is very small. Thus every fifth field hour results in a reaction. The rate of success measured as reactions per field hour thus did not improve essentially during the period.

An interpretation of these facts points towards improved managerial utilisation of staff resources for the production of more field hours, while results per field hour achieved were unchanged for the period 1996-1999. There has thus been no professional development towards finding the high-risk workplaces.

16. This example illustrates the importance of decomposition in performance audit. In the case study, the method gives an indication of the need to focus more on the professional processes in the National Working Environment Authority rather than the managerial processes. Decomposition also allows performance audit to point to places for potential improvement and its nature.

III. Economy or prioritisation of resources⁶⁾

A. Conceptual system

17. In performance audit the NAOD examines whether products and services have been acquired in the most economical manner in view of quality, quantity, etc. (the economy aspect).

The NAOD assesses whether the dispositions in question accord with the general norms for suitable and economical administration required in the use of public funds in view of the nature and size of the entity.

The auditor may consider the various types of costs that can be broken down by fields of responsibility and cost units in the financial statements. The examination can be made through a direct assessment of whether the price paid for acquired goods was reasonable compared with the quality. Sometimes, it may be practical to apply an indirect form of assessment and consider the procedure forming the basis of the disposition or the transaction.

The auditor may thus assess whether the procedure followed, for example, in procurement is suitable to ensure economically appropriate transactions.

In economy audits attention should be directed at the organisation and staff of the entity, including suitable distribution of responsibility, authority and assignments, and the reasonableness of staff costs in relation to the size and tasks of the entity.

Normally, no “key figures and ratios” are prepared to illustrate the economy aspect, but it is assessed whether the dispositions are based on common sense, and whether the considerations of economy reasonably to be expected in connection with the expenditure of public funds are made.

B. The case study: Prioritisation of resources of the National Working Environment Authority

18. The study of the National Working Environment Authority mainly deals with the question whether the individual districts disposed of requisite and adequate staff resources. This was crucial because the primary resource of the districts for performing the inspection task is the staff. The study does not consider the actual procurement of the resources, including recruitment procedure, staff policy, etc.

19. The Board of Management of the National Working Environment Authority is responsible for the overall coordination and management of the resources to ensure that the districts attain the results that contribute to the goal achievement of the National Working Environment Authority. Correspondingly, the district manager is responsible for the district’s performance of the activities planned and its achievement of the results planned.

Staff resources are allocated between the districts on the basis of a prescribed number of staff calculated by means of a district key, which is determined centrally and is the most important management tool of the Board of Management. The district key is to ensure equal distribution of staff resources in relation to activities.

The district manager lays down the final distribution in the district between administrative staff and inspectors and between various professions among the inspectors. The staff composition of the individual districts may thus vary.

20. The district key is determined on the basis of a table from Statistics Denmark's Central Business Register of the number of enterprises in each district broken down by size and industries. Furthermore, the industries have been weighted on the basis of the seven vision fields and in accordance with the experience of the National Working Environment Authority according to which the manufacturing industries, particularly the building and construction industry, are more time-consuming.

The calculations underlying the district key had not been updated since 1996, and no calculations had been made of the sensitivity of the district key to changes in variables (such as number of enterprises and distribution of industries).

21. The NAOD study showed that fundamentally the district key is a good tool for the management of staff resources by the Board of Management. The district key comprises relevant criteria for allocation of staff resources in the districts, such as the number of enterprises in each district by size and industry, but it needs to be recalculated and possibly adjusted.

22. The analysis of the resource management of the National Working Environment Authority demonstrated the importance of assessing the distribution of resources between individual organisational units in a management aspect. Only via evaluation of the management process can the auditor satisfy himself that the Management allocates resources to the fields requiring them.

IV. Efficiency

A. Conceptual system

23. The audit methodology of the NAOD in analysing efficiency consists of three elements.

The first part of the audit analysis is calculation of the efficiency ratio between input and output. This efficiency ratio can be elaborated by using different indexes depending on which efficiency (productivity) functions that are assumed. The ratio is an indication of the existence of an efficiency problem that ought to be analysed in detail.

The second part of the analysis consists in benchmarking, using Data Envelopment Analysis (DEA) where the efficiency problem is decomposed into elements such as allocative efficiency (the right mix of inputs), output efficiency (the right combination of outputs), scale efficiency (does the entity have the right size).

Thirdly, the NAOD uses regression analysis for calculating potential savings and for determining to what extent the production function is in favour of scale efficiency.

The case study only uses the efficiency ratio and the DEA method. The latter method requires a more detailed description in this paper.

Description of the DEA method

24. The DEA method compares the relationships between resource input and production (i.e., efficiency) in and between the individual entities.

The DEA method makes it possible to analyse the efficiency of a number of entities all performing the same activities. The strength of the method is that it does not presuppose a prior balancing of the activities in areas where no prices exist, as do the more conventional efficiency analyses. The method itself weights various activities so that the highest possible efficiency is calculated for the entities analysed.

The DEA method can examine the efficiency between the individual districts within the same year (cross-sectional analysis) and examine the efficiency development in a single district from year to year (time series analysis).

Cross-sectional analysis

25. The DEA method does not calculate the efficiency itself, but rather the relative efficiency (meaning that the districts are measured in relation to the most efficient ones). If a district is among the most efficient ones, its relative efficiency will be 1 (100%). If, on the other hand, a district is among the less efficient ones, its relative efficiency will be less than 1 (<100%). So if the relative efficiency is 0.9, the potential for improvement is 10 per cent.

Relative efficiency can be divided into a number of elements that allow various causes of low efficiency to be analysed. The NAOD analysis of the efficiency in 1999 divided relative efficiency into scale efficiency and managerial efficiency. Scale efficiency is a measure of whether some districts are too small to exploit any advantages of scale, or, conversely, whether some are too large to be able to organise the work without too many intermediaries. Managerial efficiency is a measure of the extent to which the individual districts utilise the resources in the optimum way. The method is illustrated in Figure 6.

Figure 6. Calculation of relative efficiency

The figure illustrates a district producing at the point A. The efficient frontiers have been set out on the basis of the most efficient districts, that is, districts having a relative efficiency of 1. Total efficiency is measured as the distance from the point A to the total efficient frontier, while managerial efficiency is measured as the distance between A and the managerial efficient frontier. Scale efficiency is calculated as the difference between total efficiency and managerial efficiency.

Time series analysis

26. For assessment of the development of efficiency over time (total efficiency), it is also possible to divide efficiency changes into two factors, technological change (changes of technology) and catch-up effect (changes in relative efficiency). Positive technological change is caused by higher efficiency in the most efficient districts, while a positive catch-up effect is caused by higher efficiency in the less efficient districts. The method is illustrated in Figure 7.

Figure 7. Calculation of total efficiency

Figure 7 shows a district which produces at point A during period 1 and at point B during period 2. The development in efficiency over time is calculated as the difference between the slopes of the two dotted lines. The difference in the slopes may partly be caused by the total efficient frontiers of the two periods differ, and partly that the distances from points A and B to the total efficient frontiers have changed.

When the slope of the total efficient frontier becomes steeper, this reflects a technical potential for higher production at the same cost (technological change). As the total efficient frontier is calculated on the basis of the most efficient districts, technological change becomes an expression of the efficiency development of the most efficient districts. A change in the distances from points A and B to the total efficient frontiers is an expression of a change in the total relative efficiency (the catch-up effect). The catch-up effect thus reflects whether the poorest districts have caught up on or have lost ground in relation to the best districts.

B. The case study: Efficiency of the National Working Environment Authority

27. One of the purposes of the NAOD study was to analyse and assess the efficiency of the National Working Environment Authority on the basis of the data available as well as the extent to which efficiency data are included in management.

As the basis of this assessment, the NAOD examined:

· whether the efficiency of the National Working Environment Authority is documented, and whether efficiency analyses are used as management information, and

· whether there are marked efficiency differences between the districts of the National Working Environment Authority and over the period 1996-1999.

Efficiency ratio

28. The National Working Environment Authority has computed efficiency as the unit costs for selected activities in its activity accounts, cf. Table 2.

Table 2. Unit costs for main activities (1999 price level)

Unit costs	1996	1997	1998	1999
	----- DKK -----
Case in offices.........................	5,650	4,503	4,434	4,779
Inspection hour in districts....	1,208	1,260	823	877
General inspection..................	2,648	2,658	1,693	2,132
Statutory inspection.................	1,429	1,466	1,403	1,452
Product register, case............	-	374	390	416

29. It appears from Table 2 that the unit costs for a general inspection rose markedly from 1998 to 1999. The National Working Environment Authority explains this with the introduction of the adapted inspection in 1999, where each basic visit took a long time, among other reasons because there were many training visits with two inspectors.

However, these efficiency measurements were not applied as management information in relation to the management of the National Working Environment Authority.

30. An analysis of the efficiency development in the individual districts of the National Working Environment Authority by means of the DEA method showed that two districts had exploited their resources by 86 per cent and thus had a potential for efficiency improvement of 14 per cent, cf. Table 3.

Table 3. Relative efficiency of districts in 1999

District No.	Field time as production			Reactions as production
	Relative	Managerial	Scale	Relative	Managerial	Scale
	----- Efficiency -----
12.................	0.96	0.98	0.98	0.76	0.81	0.94
11.................	0.96	0.98	0.98	0.85	1.00	0.85
14.................	0.97	0.97	1.00	1.00	1.00	1.00
7....................	0.96	0.98	0.98	1.00	1.00	1.00
3....................	1.00	1.00	1.00	0.96	0.96	1.00
s6.................	0.93	0.95	0.98	0.55	0.61	0.90
15.................	0.86	1.00	0.86	1.00	1.00	1.00
13.................	0.86	0.87	0.99	0.89	0.95	0.94
5....................	0.91	0.92	0.99	1.00	1.00	1.00
9....................	0.89	0.90	0.98	0.89	0.89	1.00
4....................	0.88	0.88	1.00	0.98	1.00	0.98
8....................	0.93	0.93	1.00	0.99	0.99	1.00
1....................	0.98	1.00	0.98	0.91	1.00	0.91
2....................	0.98	0.98	1.00	1.00	1.00	1.00
10.................	0.96	0.98	0.98	0.92	1.00	0.92
Total relative efficiency.....	0.94	0.95	0.98	0.93	0.96	0.90

31. If efficiency is measured as field time, only one district produces efficiently. However, two other districts produce managerially efficiently, while four others have the right size to produce efficiently.

But if an efficiency measure consisting of the number of reactions is applied, there are five efficient units. This reflects the fact that several districts are equally good at utilising the field time at their disposal for the production of reactions.

32. Calculations of the development in total efficiency as well as the catch-up effect and the change of technology are set out in Table 4.

Table 4. Development in total efficiency, catch-up effect and change of technology in the
period 1996-1999 for the general field

	Total efficiency	Catch-up- effect1)	Change of technology
	----- Efficiency measure 1 -----
1996-1997.......................................	0.99	0.97	1.01
1997-1998.......................................	1.10	1.12	0.98
1998-1999.......................................	1.00	1.11	0.90
	----- Efficiency measure 2 -----
1996-1997.......................................	0.94	0.95	0.99
1997-1998.......................................	1.09	1.09	0.99
1998-1999.......................................	0.91	1.03	0.89
1) The catch-up effect can be calculated on the basis of the total relative efficiency from Table 3.

33. The total efficiency for efficiency measure 1 (where field time has been applied as a measure of production) shows as follows:

· Total efficiency rose by approximately 9 per cent from 1996 to 1999 for product 1.

· The approximately 9 per cent rise is due to higher efficiency among the less efficient districts from 1997 to 1998.

· From 1998 to 1999, the efficiency differences between the districts evened out without resulting in improved efficiency.

Efficiency measure 2, where the number of reactions was used as a measure of the production, has seen the following development:

· Total efficiency fell by 6 per cent from 1996 to 1999.

· The fall covers a decrease in production from 1996 to 1997 and from 1998 to 1999 and an increase from 1997 to 1998.

· The rise in efficiency from 1997 to 1998 is due to the higher efficiency of the least efficient districts.

· In the period 1998-1999, efficiency evened out between the districts in the form of reduced efficiency among the most efficient districts, among other reasons.

34. Figure 8 shows the negative changes of technology illustrated for the efficiency measure of field time for the period 1996-1999. The straight lines show the most efficient correlation between number of man-years and field time for 1996 and 1999. The further northwest a line is located, the more efficient are the most efficient districts in that year. The Figure also shows that the line for 1999 is furthest to the southeast, so the most efficient district was less efficient in 1999 than the most efficient district in 1996.

Figure 8. Negative changes of technology for the efficiency measure of field time in 1996 and 1999

35. The points marked in Figure 8 show the current combination between number of man-years and field time for the individual districts in each of the years 1996 and 1999. The points for the individual years are seen to lie largely in a straight line, but the variation around the line was greater in 1996 than in 1999.

Since total efficiency has been largely unchanged, the reduced variation around the lines in Figure 8 indicates that the least efficient districts in 1996 had become more efficient in 1999. At the same time, the most efficient districts in 1996 had become less efficient in 1999. Thus, the most efficient districts in 1996 have been unable to keep up the same high efficiency.

36. The case study points to a way in which analytical methods developed within economics can be used to carry out an in-depth study of the factors underlying changes in efficiency. This allows the auditor to point not only to the existence of a problem, but also to its origin. This in turn provides the prerequisites for its solution.

V. Effectiveness

A. Conceptual system

37. A study of effectiveness focuses on the outcomes of the governmental activities. The concept of effectiveness is applied as an expression of the extent to which the intended goals have been achieved and of the relationship between the activities carried out or the resources spent and the outcomes achieved.

Outcomes express the consequences to society of the individual outputs, and the concept is thus used as an overall designation for impacts and performance.

38. The definition of effectiveness comprises three elements.

· The first element relates to an analysis of the goal achievement, for example, including a comparison of the outcomes set out and achieved.

· The second element is an analysis of the activities carried out and the outcomes achieved. Here, the concept of effectiveness is applied to the relationship between the activities carried out and the outcomes achieved.

· The third element is an analysis of the resources spent and the outcomes achieved. In such an analysis, the concept of cost effectiveness is used about the relationship between the outcomes achieved and the resources applied.

39. The basic principle is that the NAOD does not normally itself perform studies of effectiveness or cost effectiveness, but that it will generally carry out meta-studies of the entities’ own studies and their goal achievement.

Analyses of effectiveness and cost effectiveness presuppose the establishment of a model for the programme-outcome relationship based on theoretical and statistical methods. Their application presupposes the auditees’ acceptance of the methods. If the methods are not accepted, the auditor will have a problem communicating his results in an understandable way when he reports his results.

Meta-study

40. A meta-study is an assessment of the quality of an effectiveness study carried out by the entity in charge of the programme. The purpose of the meta-study is to improve the quality of the effectiveness studies and promote the use of such study results in public administration.

The role of the NAOD will be to check the quality of the entities’ own effectiveness studies and to examine the suitability of the organisational and procedural conditions for a study. The following three criteria can be assessed:

· The professional quality of the studies: That is, whether the studies apply scientifically established theoretical, methodological and technical methods of analysis.

· The applicability of the studies: That is, whether the results of a study can be applied in the administrative process or for the implementation of a policy. This means that a study report should contain information that is of essential importance for addressing a specified political problem in an effective, productive and legitimate way. The report should include a description of

· the need for information intended to be covered by the study,

· the problems in the programmes under review,

· the goals for the programmes to be covered by the study.

· The integration of the studies in the administrative process: That is, an assessment of how a study was planned, who assisted in the study, and how the study results were used.

In a meta-study, the effectiveness studies of the entities should be assessed on the basis of the requirements made by the NAOD of its own studies.

41. If the NAOD itself carries out studies of effectiveness, it is essential that the choice of methodology and the data basis have been agreed with the entity audited.

Audit studies of effectiveness

42. The NAOD carries out studies of goal achievement, effectiveness and cost effectiveness on the basis of the principles described below.

43. The analysis of goal achievement concentrates on goals for outcomes, and the NAOD assesses whether the goals are:

· Specific, that is, concrete or operationalised

· Measurable, that is, allow measurement, preferably quantitatively fixed to the greatest extent possible

· Well-balanced, that is, both realistic or achievable, and attractive, that is, ambitious or desirable

· Limited in time

· Multi-lateral, that is, connected with or not contrary to other programmes, and

· Worded positively.

44. An analysis of effectiveness and cost effectiveness presupposes the establishment of a model for the programme-outcome relationship as well as collection of data for the application of resources, production and outcomes.

45. A study of effectiveness sets out a model for the programme-outcome relationship, and the model is analysed on the basis of the collected data. The model set out for the programme-outcome relationship may contain several different actions and result in several different outcomes, and these outcomes may influence each other or directly counteract each other. At the same time, the outcomes may exhibit a dynamic development, that is, the outcomes may be both short-term and long-term.

The following elements are included in the planning of an effectiveness study and the setting out of the programme-outcome relationship:

· Can the outcomes be divided into several outcome levels, is it necessary, for example, to divide the outcomes into primary and secondary outcomes?

· How does time influence the outcomes, is there a difference, for example, between the short-term and the long-term outcomes of a programme?

· Do the individual outcomes at the individual outcome levels influence each other, and what is the connection?

· Are the outcomes affected by external factors, such as the weather, demography, etc.?

· How is it possible to conclude anything from the analysis if a programme has both positive and negative outcomes?

46. When the model for the programme-outcome relationship has been established, the individual cause-and-effect relationships must be analysed.

The analysis presupposes the collection of data according to which some study subjects have been exposed to an action under a programme while others have not, or that the individual study subjects have been affected by the programme to various degrees, possibly over a long period. The most common quantitative analytical methods are:

· retrospective studies, in which a study group and a control group are compared on the basis of historical data, and in which only the study group has been affected by any given programme,

· cross-sectional analyses, in which two or more comparable study subjects are compared,

· time series analyses, in which a study subject is studied over time,

· panel data analyses, in which two or more study subjects are studied over time.

47. Studies comparing a study group and a control group will normally be retrospective when the NAOD performs them. Retrospective means that the studies look backwards, so that the analyses are made on historical and thus also usually existing data. Whereas the NAOD usually bases its audit on historical data, the entities can set up prospective analyses, in which the studies look forwards, in which the studies are planned as an experiment, and in which data are collected during the actual experiment.

Overall, studies comparing a study group and a control group have the highest validity, and the results are applicable as documentation for various cause-and-effect relationships. But it is a condition that the two groups are equally affected by external factors. If it is impossible to set up a prospective analysis in which the study group and the control group are equally affected by external factors, the studies are called cohort or quasi-experimental studies. In such studies, the most important thing is to isolate the programme-outcome relationship from the other influences.

48. Cross-sectional analysesmake it possible to analyse several groups all affected differently by a programme during one given period. The validity of such an analysis is weakened by the fact that the analysis is made for one period only.

Through a combination of a cross-sectional analysis and a time series analysis, it becomes possible to perform a panel data analysis. This analysis studies how several different groups are affected by a programme over time. The validity of such an analysis becomes higher than the validity of both the cross-sectional analysis and the time series analysis.

49. Time series analyses, including before-after analyses are also valid because the same group is being analysed throughout the study period. It is, however, possible that the group is affected differently by external factors during the study period, which may considerably reduce the validity of the analytical method.

50. The data collection phase comprises the collection of valid, reliable and relevant data on the application of resources, output and outcome. This will allow computation of key figures and ratios for both effectiveness and cost effectiveness. The collection of data should be made with due regard to the study design chosen, that is, the model set up for the programme-outcome relationship and the analytical method chosen.

51. Figure 9 illustrates the connection between the individual analytical methods. The most valid study method is a comparison of a study group and a control group, in which both groups are equally affected by external factors. If the study can be planned before the start of the programme, a prospective analysis can be carried out. If the study is planned after or during a programme, the analysis to be carried out has to be retrospective, that is, based on historical data.

If data can be divided into one or more study groups and possibly a control group with the individual groups being differently affected by external factors, a cross-sectional analysis or a panel data analysis can be performed. If it is impossible to divide data in this way, it is necessary to choose between a time series analysis, a before-after analysis and a descriptive analysis.

Figure 9. Illustration of methods for analysing the programme-outcome relationship

The Figure is to be interpreted to mean that the validity of the study methods becomes poorer, the more times the answer is no.

52. The quantitative analytical methods can be supplemented with statistical tests showing whether the cause and effect relationships found are significant. Two categories of statistical methods can cover most of the analyses that may be required. These two methods are regression analysis and variance analysis.

B. The case study: The outcomes of the activities of the National Working Environment Authority

53. The outcomes of the activities of the National Working Environment Authority should be detectable in the number of received and recorded reports of industrial injuries in the form of accidents at work and occupational diseases.

An employer has a duty to report an accident at work to the National Working Environment Authority if the accident causes absence in addition to the day of the accident. Also doctors and dentists have a duty to report presumed or established occupational diseases to the National Working Environment Authority.

54. The number of industrial injuries from 1990 to 1999 calculated on the basis of the industrial injuries reported to the National Working Environment Authority appears from Figure 4. The number of accidents at work and occupational diseases fell from 59,781 in 1990 to 57,377 in 1991, a drop of approximately 4 per cent. The number of occupational diseases was reasonably constant in the period 1992-1998 (approximately 15,000), whereas the number of reported accidents at work rose from 44,665 in 1990 to 51,323 in 1996, whereupon it has been fairly constant.

The apparent drop in the number of reported industrial injuries in 1999 may be due to the fact that not all injuries are reported within the time limit or are not reported.

Table 5. Development in industrial injuries by the seven vision fields

Vision field	1996	1997	1998	1999
	----- Number -----
Fatal accidents and other serious accidents:
Fatal accidents...............................................................................	75	82	80	67
Other serious accidents at work ................................................	5,534	5,481	5,541	5,355
Carcinogenic substances/occupational brain injuries:
Cases of cancer.............................................................................	200	244	239	208
Brain injuries due to solvents......................................................	297	252	203	159
Brain injuries due to heavy metals.............................................	47	33	26	35
Injuries among children and young people:
Accidents at work among children and young people under the age of 18...................................................................................	680	680	642	632
Accidents at work among the 18 to 24-year olds.....................	8,175	7,747	7,518	6,757
Occupational diseases among children and young people under the age of 18........................................................................	39	42	39	23
Occupational diseases among the 18 to 24-year olds..........	1,223	1,156	977	829
Heavy lifting/monotonous repetitive work:
Accidents at work due to acute strain.........................................	8,375	8,035	7,301	6,958
Occupational diseases, heavy lifting..........................................	5,927	6,811	6,154	4,678
Occupational diseases, monotonous repetitive work............	4,187	4,484	4,191	3,619
Psychosocial risk factors:
Occupational damage to health due to psychosocial factors	1,401	1,300	1,301	1,145
Accidents at work due to violence...............................................	1,922	2,255	2,360	2,234
Poor indoor climate......................................................................	412	287	332	234
Hearing impairment due to noisy work:
Occupational hearing impairment..............................................	2,962	2,684	2,236	1,877
Accidents at work due to acute strain.........................................	48	44	56	106
Total..................................................................................................	41,504	41,617	39,196	34,916
Note: The 1999 figures are based on reports received by 1 February 2000. The National Working Environment Authority expects the final 1999 figures to be higher, since not everybody reports injuries within the deadline.

55. The development in the number of reported industrial injuries specified by vision fields according to Action Programme 2005 is illustrated in Table 5. There is no clear development within the seven vision fields in the period 1996-1998. However, the total number of industrial injuries in 1998 is lower than in 1996. Important areas like fatal accidents, other serious accidents at work, cases of cancer, occupational diseases – heavy lifting, occupational diseases – monotonous repetitive work, accidents at work due to violence and accidents at work due to acute strain all exhibit a rise in 1998 compared with 1996, however.

56. Despite the duty to report accidents at work and occupational diseases to the National Working Environment Authority, the Authority estimates, on the basis of an emergency ward study covering the period 1990-1995, that approximately 55 per cent of accidents at work are not reported, that is, a reporting deficit of 55 per cent. The study showed that the reporting deficit varied depending on the nature of the injury, the victim’s age and the industry in which the victim was employed.

The reporting deficit of accidents at work is confirmed by a study made by Statistics Denmark in 1999, according to which approximately 110,000 accidents at work resulting in absence from work occurred per year.

According to the National Working Environment Authority studies of the reporting deficit of occupational diseases cannot be made, as reporting depends on medical assessment. However, a considerable reporting deficit of diseases is suspected, particularly concerning psychosocial problems, skin diseases and indoor climate.

57. In 1996 the Minister of Labour presented and launched Action Programme 2005 which the Danish Parliament approved. The Ministry of Labour, including the National Working Environment Authority, the labour market organisations and the Working Environment Council had been in charge of the preliminary work.

The point of departure for Action Programme 2005 was recognition of the need for more conscious prioritisation of the working environment efforts. There was a desire for greater active contribution from staff and management of the enterprises towards endeavours to accomplish the vision that all Danish workplaces should become safe, healthy and individual developing to the benefit of staff, the competitiveness of the enterprises and society as a whole.

Action Programme 2005 set out the framework for ten years’ working environment efforts towards 2005 and focused on the achievement of a preventive, effective and high-quality working environment effort. It was binding on and set the course for the stakeholders involved in the working environment system, including the National Working Environment Authority.

58. On the basis of the registration made by the National Working Environment Authority of working environment influences and health factors in various industries (the industry pictures), a wide popular debate and cooperation with the social partners on what was to be emphasised within the working environment work, Action Programme 2005 set out seven vision fields in the form of working environment problems, which all stakeholders have to work to reduce/completely eliminate. The seven vision fields appear from Figure 10.

Figure 10. The seven vision fields according to Action Programme 2005

The seven vision fields

Fatal accidents due to working environment factors

Occupational exposure to carcinogenic chemical substances and occupational brain injuries due to exposure to organic solvents or heavy metals

Injuries among children and young people in connection with work

Injuries due to heavy lifting and occupational diseases due to monotonous repetitive work

Impaired hearing due to noisy work

Health injuries due to psychosocial risk factors at work

Diseases or serious inconveniences due to poor indoor climate at the workplace

59. The first five vision fields aim at known and documented working environment problems, while the two latter vision fields aim at working environment problems whose full consequences are not yet visible.

Action Programme 2005 further establishes that work must be done to enable the enterprises to solve their working environment problems themselves, such as by increased efforts by the enterprises and better prioritisation of the inspection and increased dialogue with the enterprises. The foundation for this is recognition of the fact that the working environment will not improve unless a major part of the working environment initiatives are carried out by the enterprises themselves.

60. It appears from Figure 4 that the vision of Action Programme 2005 did not translate into a marked drop in the total number of industrial injuries.

However, the efforts of the National Working Environment Authority are not the only parameter influencing the development in the number of industrial injuries; a large number of surrounding factors also have influence (the efforts of the other stakeholders, the general employment situation and the relationships between the economic climate within the individual industries also have an impact).

61. In June 1999, as a supplement to Action Programme 2005, the Minister of Labour presented an Action Plan 1999, which is an attempt to strengthen certain elements of Action Programme 2005 and give higher priority to the health-promoting efforts. The Action Plan thus focuses on the ten particularly dangerous types of jobs, improvement of the systematic accident research and increased working environment supervision by the National Working Environment Authority.

62. The NAOD study finds that no drop in the number of industrial injuries was noticeable despite the fact that the Action Programme is supposed to have been in effect since 1996.

The NAOD recommended the Ministry of Labour to set out goals for the progress of the working environment on the basis of Action Programme 2005.

The NAOD was aware that the working environment field has many stakeholders, and that the National Working Environment Authority alone therefore cannot realise the intentions of Action Programme 2005 and Action Plan 1999.

63. The case study shows that a model for performance audit of effectiveness has to include both stakeholders and external factors. Their importance differs. Thus, the stakeholders affect the managerial system, while external factors impact directly on the outcome. A major factor for assessment of outcomes in the working environment field is the ability of the registration systems to collect data on all accidents at work and occupational diseases. A change in the number of accidents at work and occupational diseases is thus the sum of changes in the quality of the recording systems and changes in the actual accidents at work and occupational diseases as a result of improved inspection or external factors.

VI. Conclusion

64. In recent years, the development of performance audit of the NAOD has been characterised by the following elements:

· Establishment of a conceptual framework coupling financial management with the fundamental principles of performance audit.

· The model set up allows for analytical decomposition in accordance with the fundamental concepts of performance audit, to provide a better understanding of how to improve the effectiveness of public administration.

· Coupling of the fundamental concepts of performance audit: efficiency and effectiveness, with a number of methods and techniques that have contributed to a methodological qualitative improvement of the NAOD reports within performance audit.

Both at the national and the international level, the development of the auditing standards tends to provide a clearer connection between the application of the fundamental concepts of performance audit and its methods and techniques. The development work in this performance audit field is carried out naturally within the regional associations of Supreme Audit Institutions and in the INTOSAI Auditing Standards Committee.

Literature

Andersen, Jens Lund: The principles of Sound Financial Management. EUROSAI Magazine no. 6-1999, pp. 30-33.

Andersen, Jens Lund: Governance as part of performance audit examinations. EUROSAI Magazine no. 7-2000, pp. 37-38.

Coelli, Tom, D.S. Prasada Rao, George Battese: An Introduction to Efficiency Analysis. Kluwer Academic Publihers 1998.

European Implementing Guidelines for the INTOSAI Auditing Standards, No. 41,Performance Audit, 1998.

Elm-Larsen, Rolf: Forvaltningsrevision (Performance Audit). Samfundslitteratur2001.

Farrell, M. J.:The Measurement of Productive Efficiency. journal of the Royal Statistical Society. Series A (General). Vol. 120. Part III, 1957, pp. 253-281.

Levysohn, Morten:The Principles of “Good Public Auditing Practice”. EUROSAI Magazine no. 6-1999, pp. 22-25.

Pollit, Chistopher and Hilkha Summa:Performance or Compliance? Performance Audit and Public Management in Five Countries. Oxford University Press 1999.

Rigsrevisionen:Statslig Revision. Håndbog i statslig revision. København.

Rigsrevisionen:God Offentlig Revisionsskik. (Good Public Auditing Practice). København 1998.

Statsrevisorernes beretning nr. 14/99 Beretning om Arbejdstilsynets styring af tilsynsfunktionen. København2000.

1) This paper has been prepared for the author’s presentation at the 6th Tokyo International Audit Forum, September 2001. The author alone bears full responsibility for its contents.

2) See, e.g. the European Implementing Guidelines for the INTOSAI Auditing Standards, No. 41, Performance Audit, 1998.

3) European Implementing Guidelines for the INTOSAI Auditing Standards, No. 41, Performance Audit, 1998, p. 119; Christopher Pollit and Hilkha Summa: Performance or Compliance? Performance Audit and Public Management in Five Countries. Oxford University Press 1999, p. 10.

4) Farrell, M. J.: The Measurement of Productive Efficiency. Journal of the Royal Statistical Society. Series A (General). Vol. 120. Part III, p. 253-281.

5) The 1999 figures are based on reports received by 1 February 2000. The National Working Environment Authority expects the final 1999 figures to be higher, since not everybody reports injuries within the deadline.

6) In June 2001, the NAOD initiated a methodology development project with a view to economy auditing with expected conclusion in mid-2002.