Part B

6.0 Developing a Suitable Victorian Framework

Significant debate persists as to how indicators can meet the demands of quality improvement at institutional or local level whilst concurrently serving as reliable and valid measures for external accountability. Despite considerable research no clear answers yet exist. Nonetheless, it is important to recognise the dual and complementary functions of indicators, which have obvious implications for the way in which indicators are presented and recommended for use. The challenge for Victoria is to try to identify indicators which will withstand public scrutiny and effectively meet the interests of all stakeholders-consumers, providers, purchasers and funders.

In returning to the initial requirements for a Victorian program, it is one which is:

A framework can be established which meets these requirements by initially drawing upon other programs as discussed in Part A.

The framework of the NHMBWG (Appendix 4) provides a good foundation and has been previously adopted in developing a position for Victoria30. The framework has been extended by the Quality & Outcomes Indicator Project (QOIP) in defining eight dimensions of quality of care as a basis for indicator development 15. This framework also draws upon work of Bernstein and Hilborne31 who propose that indicators should measure:

The QOIP project also included, the dimensions of efficiency and safety. By nominating safety as a separate entity, the project highlights the importance given to this by the Taskforce on Quality in Healthcare13, despite its considerable overlap with appropriateness and technical proficiency.

The dimensions of care in existing frameworks are variably reflected in currently available indicators, hence the need to go beyond what already exists. What is now emerging in quality assessment is the need to move from strictly hospital-based outcome measurements to programs which embrace integrated delivery systems. According to Shulkin "quality assessment across the full continuum of care remains the next challenge facing outcomes programs" 32.

Quality does not exist in isolation; within various units, departments or clinical functions. Increasingly, assessment is being approached in an organisation wide context. This is in keeping with the new ACHS Evaluation and Quality Improvement Program 33, which assesses quality across a continuum of care and in key organisational functions of leadership and management, human resource management, safety and environment, information management and performance improvement. A similar approach is advocated by the European Foundation for Quality Management (EFQM) which proposes a framework based on nine service elements and nine key hospital processes. A quality strategy being adopted in Wales is mapping all NHS performance indicators against the elements of the EFQM framework 34.

This therefore lends support to a proposal for a broad-based framework for Victoria which will embrace organisation wide performance evaluation through the use of key (yet to be specified) indicators. There should be a strong focus on (clinical) care provided to patients before, during and after their hospital stay, as well as the management of the organisation in which care is provided.

As stated by O'Leary (JCAHO), "there is a safe assumption that if clinical care and organisational management are "done well", then good outcomes are likely to result"35.

The proposed framework as shown at Table 7 specifies nine areas of evaluation and categories within each area for potential indicator development, plus relevant comments and examples. The categories and examples are provided as a guide and are not all encompassing. Actual indicators have not as yet been defined, as this will depend upon those areas and categories which are agreed upon and targeted for development. In some areas, for example, access to care, indicators are already in place or in development. It is not expected that this work will therefore be duplicated but possibly reviewed and refined in light of the whole strategy. Additional detail is provided for clinical care, in 8.0 and at Table 8.

6.1 Establishing Priorities

It is proposed that the strategy for use in Victoria should be developed and indicators implemented progressively over a number of years. Areas for indicator development and/or enhancement need to be prioritised and ongoing work focused accordingly. With regard to existing resources it will not be feasible to develop and assess indicators in all nine areas concurrently. The following outline is a suggested means to allocate and determine priorities for work in all areas.

Existing areas for cooperative development:

  • Access to Care
    		•  (Ambulatory/Service Development Section-Acute Health).
  • Organisational
    		•  (Financial Analysis and Purchasing Management Section-Acute Health).

    Areas to be adopted for further development:

  • Preventive Care
    		• (Public Health Division)
  • Continuity of Care
    		• (Ambulatory Section-Acute Health).
    Prioritised areas for focused review and development by Quality Section:

    Table 7: Framework for Performance Indicators-Nine Areas of Evaluation

    Areas Potential categories Comments/examples
    1. Preventive Care Measures of population health Involve Public Health Division
    Could address immunisation rates, screening programs, etc.
    Health promotion programs
    Pre-hospital care    		 According to policy targets
    Episode prevention (acute on chronic)
    Care in pre-admission clinics, etc.
    2. Access to Care Elective surgery waiting lists
    Emergency department waiting times
    Access to critical care beds
    Outpatient waiting times    		 Access indicators developed and in use in Victoria
    Review in context of national developments
    3. Clinical Care Ambulatory care
    Surgical care
    Medical care
    Intensive care
    Obstetrical care
    Psychiatric care
    Paediatric care    		 Identification of condition-specific indicators within defined areas. See section 8.0 and Table 8.
    4. Appropriateness of Care Standardised utilisation rates for specific procedures. Utilisation as a proxy for appropriateness
    Select high throughput procedures relating to categories in clinical care
    5. Safety of Care Incident reports
    Risk management (claims)
    Medication errors/adverse reactions
    Hospital-acquired infection    		 Hospital wide and/or drug-specific
    Eg. bacteraemia, multi-resistant organisms
    6. Effectiveness of Care Functional health status (SF36/12)
    Selected mortality rates
    Long term clinical outcomes
    Readmissions and/or unscheduled returns    		 Conditions as per clinical care
    Eg. reoperation rates, prosthesis replacement/infection
    To hospital, emergency dept., operating room, ICU
    Consider condition specific
    7. Continuity of Care Discharge protocols/processes
    Community management / follow up / communication    		 Involve ambulatory section
    Hospital in the Home
    8. Satisfaction with Care Consumer perception/acceptance/feedback re care received
    Patient complaints    		 Determine following results of patient satisfaction survey stage 3
    9. Organisational Management Efficiency measures
    Financial measures
    Internal customers/staffing    		 Efficiency/financial measures in use in Victoria
    Eg. satisfaction, costs, turnover, compensation claims/injuries

    7.0 The Way Ahead-Steps to Achievement

    1. Identify and achieve consensus for a suitable Victorian framework which allows for the comprehensive development of performance indicators for the acute health sector utilising a system-wide and multidisciplinary approach to care delivery.
    2. Prioritise the areas for performance evaluation for focused developmental work with specific time frames for achievement.
    3. Within key areas, identify potentially suitable indicators which reflect upon service delivery and patient care drawing upon a range of available indicators.
    4. Establish and utilise criteria for indicator 'fitness' to allow for initial assessment of indicators by the working group.
    5. Specify indicator definitions and required data elements and determine essential criteria for further indicator assessment.
    6. Undertake detailed feasibility studies of selected indicators to determine accessibility, reliability and validity of data.
    7. Report on feasibility of each indicator for state wide collection, including sample analysis of data with recommendations to accept/reject for further development.
    8. Pilot test defined indicators within a selection of Victorian hospitals of various size and type.
    9. Progressively implement indicators for state-wide collection and reporting within the acute health sector.

    8.0 Clinical Care Indicators

    The identification of categories and indicators in clinical care can be approached in a variety of ways. The proposed approach embraces key dimensions of clinical practice which can be applied in the majority of acute health care settings and are consistent with dimensions of care monitored in other indicator programs, as reviewed.

    The selection of focused indicators will require careful consideration and consultation with expert clinicians. It is suggested that condition-specific indicators be identified initially drawing on the ACHS indicators (provider developed) and others highlighted from programs in Part A of this paper. The conditions will need to be high throughput, commonly treated and/or with the potential for significant (and costly) complications. In reviewing current literature there is a degree of consensus about target conditions and these are brought together for consideration and specified at Table 8.

    9.0 Indicator Attributes

    The performance indicator strategy will aim to identify and design the best possible measures. No indicators are perfect however, but can be improved through their application and use. It is important to remain aware of indicator limitations, as they will not necessarily distinguish between good and poor performance14. According to Boyce et. al., existing, albeit imperfect indicators, do provide useful pointers to the quality of care. If results of indicator data occasionally differ significantly from user expectations, it may be necessary for those using these tools to consider revising their understanding of the role of indicators in monitoring quality, rather than discarding these indicators and persisting with attempts to find 'perfect' indicators15.

    Indicators need to be assessed against a number of criteria and attributes, in order to establish their ability to meet their intended purpose. Various programs have nominated an array of attributes as a means of determining the potential utility of any proposed indicators; these are reproduced at Appendix 5.

    There is considerable agreement in what researchers see as appropriate criteria and, through a process of aggregation, ten key attributes are proposed as a means to evaluate indicators for use in the Victorian context. These are defined in Table 9.

    9.1 Generic vs. Specific Indicators

    An important finding of the Quality & Outcomes Indicator Project was the difficulty in identifying generic indicators which would adequately reflect upon performance and quality across broad levels of care15. This has also been a recognised limitation of the ACHS Hospital Wide Medical Indicators. However, with the introduction of discipline-specific indicator sets, this issue has been overcome by the ACHS to a large extent. Nonetheless, in developing a Victorian strategy, it is important to be aware of the limitations of generic indicators. Performance will be more accurately reflected through the identification of key dimensions of health care quality (such as those proposed in the framework) which target particular conditions, diseases and interventions.

    Table 8: Clinical Care Conditions for Indicator Development (Potential not actual)

    Categories Aspects of Care/Conditions
    Ambulatory Care Day procedures/investigations
    Outpatient care
    Hospital in the home
    Surgical Care Coronary artery bypass grafts
    Bowel resection
    Spinal surgery/laminectomy
    Cholecystectomy
    Hysterectomy
    Dilatation & curettage
    Prostatectomy
    Cataract surgery
    Total hip/knee replacement
    Carotid endarterectomy
    Tonsillectomy
    Perioperative clinical events
    Medical Care Acute myocardial infarction (AMI)

    Angioplasty (PTCA)
    Congestive cardiac failure (CCF)
    Chronic obstructive airways disease (COAD)
    Asthma
    Oncology-Breast, Lung, Colon/rectum
    Diabetes (ketoacidosis)
    Stroke
    Pneumonia
    Renal dialysis
    Haematemesis and melaena
    Intensive Care Acute interhospital transfer
    Mortality
    Length of stay
    Inability to admit
    Procedural complications
    Obstetrical Care Caesarean section
    Vaginal birth after primary caesarean section
    Apgar score
    Low/very low birth weight
    Psychiatric Care Critical incidents
    Prolonged LOS
    Readmissions
    Failure to be given a diagnosis
    Paediatric Care Asthma
    Appendectomy

    Table 9: Key Attributes for Indicator Assessment and Selection

    Attribute Description
    1. Definable Can the indicator be clearly defined?
    2. Clear Intent Is the intent of the indicator easily understood and interpretable by all users?
    3. Relevance Does the indicator measure aspects of care which are relevant and significant?
    4. Accessible Are data easily accessible?
    5. Reliable Is there demonstrated reliability (reproducability) of data? Reliability will largely depend upon standardised definitions and rigour of data collection mechanisms.
    6. Valid Does the indicator measure what is intended and point to issues of quality?
    7. Event Identified Can events be readily identified through diagnoses and/or frequency of occurrence, eg. major complication or commonly treated?
    8. Useful Does the indicator provide useful information to inform quality programs and stakeholders?
    9. Practical Benefit Does the indicator have a strong cost:utility ratio?
    10. Responsive Is the indicator responsive with a potential for action and quality improvement?

    10.0 Administrative Datasets

    The integrity of indicators and hence potential utility are largely dependent upon the quality of data from which indicators are derived. For state-wide monitoring purposes, the use of routine administrative datasets, such as the Victorian Inpatient Minimum Dataset (VIMD) is proposed, to promote data consistency, given the scope and economy of such means of abstraction. Whilst administrative datasets are relatively complete data sources, there are considerable risks of inaccuracy if they are used for purposes other than those for which they were originally intended.

    Datasets such as VIMD are based on the ICD-9-CM coding system which was designed for classification of mortality and morbidity information for statistical purposes and for indexing of hospital records. As information in records is frequently inconsistent and incomplete, this has impact on the accuracy of coding and hence reliability of data extraction. Furthermore, administrative datasets lack clinical detail 'particularly relating to adverse events', affecting the ability to risk adjust at patient level and currently (without unique identifiers) it is not possible to link episodes of care for patients. Recording of external injury codes (E codes), which has been suggested as one method of identifying adverse patient occurrences, is also highly unreliable as there is no monitoring of the data quality27.

    If data are to be abstracted by administrative datasets, consideration will need to be given to:

    10.1 Administrative Datasets-A Conceptual Framework

    In recognising the limits of administrative datasets for performance indicator data extraction, one group of researchers in the United States has proposed a conceptual framework to guide the abstraction and analysis of data to evaluate the quality of medical care. Four interrelated areas are identified and within each, criteria identified to maximise the validity of results. The framework is reproduced at Table 10, as being worthy of consideration in the Victorian context36.

    Table 10: Framework for Use with Administrative Datasets

    Component Criterion Comments/Examples
    A) Selecting disease entities 1. Disease entities/clinical states should be well defined and easily diagnosed. Objective parameters of diagnosis and little influenced by subjective interpretation of clinician/patient. Eg. confirmation of AMI by ECG.
    2. If diagnostic classification systems are used, disease groups should be homogeneous to clinical states contained. Eg. Appendectomy DRG 166-3 ICD 9 codes
    3. Disease entities should be prevalent. Sufficiently prevalent conditions used to represent care given to all patients. Infrequent clinical entities/uncommon outcomes subject to random variation and lack statistical power.
    B) Choosing the outcome of interest 1. Plausible link should exist between the quality (process) of care and frequency of outcome. Good care must be known to reduce frequency of adverse outcomes. Eg. Diabetic ketoacidosis and AMI-evidence exists that current therapies reduce mortality.
    2. Types of care which conform to acceptable practice standards but still lead to variation in outcome of interest should be excluded from analysis. Eg. hospice deaths excluded from mortality studies. Conditions in which admission- planned readmission is acceptable practice are excluded.
    3. The outcomes should be prevalent. Outcomes of interest must occur within study cases with reasonable frequency, eg. at least 5% of cases at risk.
    C) Limitations of administrative databases 1. Constraints of ICD 9 coding should be understood so that only those disease-outcome pairs least affected by limitations are selected. Conditions not selected if limitations pose threat to validity of findings.
    2. Constraints of database structure should be considered when analysis is performed. Eg. lack of differentiation between comorbidities/complications. Lack of accuracy of recording.
    D) Quality Assessment methodology 1. Disease-outcome pairs should be chosen where there is the agreement on processes of care that affect the outcome of interest, favourably or unfavourable. (Expansion of concepts in B). Consensus amongst clinicians re links between process/outcomes of care. Allows implicit/explicit criteria with greatest probability of detecting practice outside accepted norms, ie. true quality of care problems.
    Source: Wray et.al. Soc. Sci. Med. 1995, Vol.40

    The researchers concluded that the analyses of large administrative databases is a valid means for assessing medical practice, if the analysis is guided by a framework which recognises the underlying assumptions of the analysis and important methodological issues. It is suggested that their methods can guide the selection of suitable performance measures and identification of quality of care problems in individual hospitals36.

    11.0 Implementation Issues

    11.1 Stakeholder participation

    The implementation of performance indicators at the State level must be by way of a carefully considered approach. Any strategy will be destined to failure or non-acceptance if it carries the connotation that it has been established only to satisfy external forces or to identify and/or discipline hospitals or individuals. In the words of Deming, "no one can put in his best performance unless he feels secure". In other words, fear is inherent in the 'bad apple' approach to quality and the perception that quality is something imposed by outside forces looking over the shoulders of members of the organisation37.

    It is imperative that all stakeholders have a clear understanding of the Department's reasons for adopting a state-wide approach to performance monitoring. Questions will inevitably be asked, given that many organisations are already monitoring indicators for the ACHS EQuIP and particularly in the context that accreditation is to become mandatory within Victoria. Many providers are also aware of the initiatives being adopted at Commonwealth level to establish national performance indicators. The Department will therefore need clearly defined means of communication with all stakeholders.

    The Departments interest in using performance information is to promote quality improvement at a system wide level. The relative performance of a hospital or clinical unit compared with its peers is not necessarily the result of clinical decision-making by individual clinicians, but more likely a whole range of interrelated system factors28. The effective use of performance indicators permit the better understanding of how errors or adverse outcomes creep into clinical practice-they do not seek to identify errors so as to apportion blame, but assume that 'faulty' systems of care are often responsible. Fixing systemic problems in the provision of care is much more effective in reducing errors and promoting quality than punishing providers38.

    Of paramount importance is the need to engage clinicians at the outset, to allay any fears that performance monitoring may interfere with clinical decision-making and service provision and to ensure that clinicians have a sense of ownership and responsibility for the indicators. The work of the ACHS Care Evaluation Program with Australian medical colleges is fundamental in this regard and the Department should build upon the solid working relationship which has already been established.

    11.2 Data Ownership and Security

    Vast amounts of data are being collected and reported at hospital level for various bodies and authorities and concerns are frequently raised about the adequacy of data protection, particularly from inappropriate usage. Issues of data ownership, privacy and security will need careful consideration in the context of the Departments role as a purchaser of acute health services and custodian of data provided by hospitals.

    A framework for data privacy and security is under development and it is proposed that Victorian legislation will be introduced in the Spring sitting of Parliament, to apply to all Victorian Public Services and operations. A key component is the endorsement of a number of key principles relating to privacy and security, which in turn can be translated into codes and tailored to industry services. Consultants are currently working with the Department to develop a Privacy Policy Framework to support the new legislation. The outcomes of this work will be highly relevant to performance indicator information, and relevant issues for consideration include:

    11.3 Phased Implementation

    The acceptance of indicators will be fostered by adopting a phased approach to implementation. Through processes of validation, testing and evaluation, indicators will be progressively identified as being suitable (or not) for use within the acute sector. Initially this may involve just a few indicators in targeted areas and more will be incorporated as the program evolves. This approach is important as the monitoring of large numbers of indicators can rapidly overwhelm the capacity to use results effectively and to make improvements. If problems are identified it is important that hospitals have the opportunity to change or improve their performance and a process of phased implementation will promote this. On current time lines it is anticipated that some additional indicators could be implemented for State-wide use by the 1998/1999 financial year.

    11.4 Reporting and Results Evaluation

    Appropriate reporting mechanisms and periods of reporting should be determined prior to implementation; so too, mechanisms to verify data reliability and accuracy prior to the public disclosure of any hospital-specific information. The Quality and Outcomes Indicator Project and Taskforce on Quality in Healthcare have both recommended that results verification be conducted by parties independent of government and service providers13,15. This approach is consistent with that to be adopted by the JCAHO ORYX Initiative, whereby indicator results will be reported to JCAHO via nominated information management vendors20. The involvement of a third party for data analysis, verification and reporting remains an option for consideration by the Department.

    The responsible use of performance data by the Department is imperative. The premature release of performance information may have significant consequences for the credibility of the program and in maintaining consumer and provider confidence. The HCFA experience attests to this. Whilst it has been said that public disclosure of performance information may lead to changes in behaviour, this may not always be what is intended28. Studies from the United States report that following the publication of death rates for bypass surgery, there was a reduction in mortality rates but that some unintended consequences of disclosure, such as reduced access for high risk cases were also highlighted28, 39,40. A cautious approach is therefore strongly advocated. It is recommended that the reporting and release of information be an evolving program which takes account of the stage of development of the program and preparedness of stakeholders to receive and accept information. Provision should be made for facilities to address data quality problems and/or respond to issues raised prior to release and/or publication. Nonetheless, as stated by McKee, where care clearly falls below acceptable standards, there is no excuse for failing to take action28.

    Clear guidelines for release of information will need to be established and communicated to all stakeholders. This should include the provision of adequate education and support documentation including methods used for analysis and adjustment of data. This will help to ensure that information can be readily understood and interpreted, allow for meaningful comparisons and minimise potential for distortion of results.

    The linkage of performance indicators to hospital funding and/or incentive schemes will require careful consideration and prior to any such arrangements the reliability and validity of the indicators would need to be assured. A hasty approach to the linking of financing to indicator targets or benchmarks may create a system which provides incentives for non-reporting or manipulation of indicator application. The use of incentives would need to be designed so that they serve to stimulate better quality, including better data collection and reporting, publication of results and quality improvements over time41.

    12.0 Performance Indicator Sub-Committee

    To enable the development and implementation of the proposed strategy to proceed in a timely way, a Performance Indicator Sub-Committee is to be established. It is intended that the committee will oversee, and advise upon, development activities, calling upon relevant external expertise as required and working closely with Divisional staff engaged in development activities. The Committee is to be established as a sub-committee of the Acute Health Quality Committee, reporting to the same, and with a suggested membership structure as follows:

    Back to Index Page