How to use Patient Safety Indicators

Page content: Overview | What do the reports look like? | How you can use the reports | Who should be involved in analysing the reports? | How DHS will use the reports | Where can you find out more? | Acknowledgements

Overview

There is an opportunity to improve measurement performance of Victoria’s hospitals. This measurement focuses on the ability to track improvement and monitor patient safety. It can be done using routinely collected administrative data in conjunction with a set of well developed indicators.

The indicators that will be used rely on two indicator sets that have been specifically developed for use with routine administrative data. In Victoria, the source of routine data is the Victorian Admitted Episodes Dataset (VAED).

The indicator set has been developed to allow changes in performance to be flagged and to identify the trend preceding the flag. As a result, hospitals can target investigation efforts more effectively.

The Patient Safety Indicators are a screening tool and do not provide absolute rates. They show you where an area of concern or exceptional practice might exist to prompt and guide further investigation.

Over the next 12 months, quarterly indicator reports will be provided to health services for 11 indicators to support monitoring of patient safety by health services and Department of Human Services, to allow familiarity with the indicators and the reports and to enable improvement of all parts of the initiative.

This introductory guide will give you an insight into the indicators, how they will be reported and what you can do with the information and what the Department of Human Services will do with that information.

The purpose of this document is to give you an idea of what the indicators are intended for and how you might use them. Detailed technical explanations are referred to in the document and will be provided separately.

What do the reports look like?

The indicators will be reported using an ‘Expected minus Observed’ reporting methodology known as Variable Life Adjusted Display (VLAD). These reports are statistical process control charts that plot a graphical overview of clinical outcomes over the course of a selected period. A sample is presented below.

A key strength of using any expected minus observed chart like VLAD is that each episode of care has an impact on the chart. That is, every patient outcome moves the line up or down according to the outcome of their stay. The amount of this movement is driven by the difference between their expected outcome and their actual outcome.

When used with mortality as the outcome indicator, it displays 'estimated statistical lives gained' by plotting the cumulative difference between expected and actual outcomes. In the non- mortality indicators ‘estimated statistical events avoided’ is used. The expected number is a statistically generated figure based on the risk profile of the patient and organisation. That is, each patient has a likelihood of developing an outcome. It is the difference between this expected outcome and the actual or observed outcome that generates the VLAD graphical line.

A flagging mechanism has been applied that indicates areas of significant change or ‘apparent outliers’. These parameters are calculated to detect if a particular hospital’s outcomes are higher or lower than expected. An apparent outlier may reflect either an area of concern (lower control limit) or of exceptional practice (upper control limit).

The level of flag can be set anywhere and changed to be more or less sensitive. The flag levels will be reviewed by the project team incorporating feedback received. There will initially be one level of flag presented on the charts. The potential exists to include more flagging levels at varied levels of performance change that deserve escalating level of attention.

When the indicators are used with the VLAD reporting methodology they are effective in enabling health services and the Department of Human Services to identify performance changes. It enables rapid identification of trends at hospital level that may not be captured when reporting periodic averages.

The value of this when compared to periodic averages is that trends of poor and good performance are not flattened out or hidden over time. In any period, say a quarter, there may be runs where performance changes. By recognising these it is possible to identify the contributing factors and either correct them or, in the case of better than expected performance, duplicate them.

How you can use the reports

The Patient Safety Indicators are intended for use as a screening tool. Their primary purpose is to allow you to find a change in performance and start your investigation in the right place. They do not provide the absolute rate of an event and should not be used for this purpose.

By looking at the chart you will be able to identify which episodes of care were involved in a change in performance. Behind each chart there is a list of unit record numbers and separation dates that contributed to the chart. Using these and the trend leading to the flag you can extract only those records that are relevant. This allows you to start your investigations in the right place.

Investigation and analysis should include factors that might lead to an event. A simple way to cover multiple factors is to use the ‘pyramid of investigation’ . This lists five factor headings under which an investigation can be conducted. These are:

Data - The best place to start. An example of a contributing factor is a change in coder or coding standard.

Patient case mix - Central risk adjustment can only allow for data fields that are contained within the VAED. Your local information will give you a much clearer picture of how the casemix of your recent group of patients might contribute to an apparent change in performance.

Structure of resource - This category includes such things as bed availability or where patients of a particular group are cared for in different location.

Process of care - For example, new protocols or policies. More specifically, implementation of a new care pathway might result in improved outcomes.

Professional - Finally, have a look at the clinical staff involved in patient care. Sometimes a change to staff or the techniques they use might be a contributing factor to performance change.

Figure: Pyramid model of investigation

Who should be involved in analysing the reports?

As there are many potential contributing factors, such as those outlined above, a variety of expertise should be considered for inclusion in any investigation. A starting list would include individuals with expertise in:

• Clinical (relevant to individual measure)
• Coding (experience in use of VAED [ICD-10-AM] based information)
• Quality/Clinical Governance (system and process)
• Performance (implications of data and information analysis and manipulation)
• Management (understanding of broader organisational issues)

In the report spreadsheet there is a graph and data sheet for each indicator. The graph sheet will show you where ‘flags’ occurred (where the performance line crosses a control limit). The data sheet contains many columns and numbers. These have been used to generate the graph (VLAD chart) and are not all relevant o non-statistical users.

The most important of these numbers are the unique identifier (patient unit record number) and column ‘D’ (titled mortality). Using the unique identifier you can find the trend leading to the flag. The number in column D represents whether the event occurred (1) or not (0) e.g. deep vein thrombosis. There will only be an indicator report where your hospital had sufficient episodes of care to make the indicator workable.

The definitions of the indicators are also contained in the spreadsheet. This will be a useful place to start when considering what contributes to the indicator.

Through investigating the flags, as they show up on the reports, your staff will become more familiar with the reports, be able to offer comment on their value and usability and most importantly they will have a clearer picture of how the hospital is performing.

How DHS will use the reports

The Department of Human Services will provide a selected group of hospitals with the above indicators, as VLAD reports, on a quarterly basis. The intent is to help you to know whether your hospital’s performance is as expected or different to what might be expected.

The selected group of hospitals are Victoria’s largest and have greater technical expertise and capacity to investigate apparent flags/outlier events. The level of expertise and resource will be reviewed as part of implementation.

As well as this review, feedback is also invited about usability, relevance, technical improvements and any other suggestions that you may have. These will be considered and incorporated into improvements as appropriate. Part of this will be setting the flag levels at a level where investigation will bring some value. That is, not too early or too late.

After health services and the Department of Human Services are familiar with the indicators and the reports and improvement have been made based on feedback received and analysis undertaken there is an expectation that response to flags from health services will be a requirement.

Where can you find out more?

The programs have been created for use with SAS and to generate MS Excel based charts. The technical documents such as programs and code sets have been made available so that anyone with appropriate expertise and software can use the indicators. Also in making the indicator definitions publicly available exposes them to review, criticism and feedback enabling them to be improved over time.

Your thoughts on usability, relevance, technical improvements and any other suggestions that you may have are invited at auspsi@dhs.vic.gov.au. This email account will be managed by the Clinical Information & Knowledge Management unit who can be contacted on 9096 7022.

Acknowledgements

Much of the work presented here draws on work previously undertaken by Queensland Health. The willingness of the Clinical Practice Improvement Centre at Queensland Health to share experience and tools has made this work much more.