Data Quality

Introduction

ISD, NHS Boards and the Scottish Government have put in place measures to help ensure the quality of the data held within the Waiting Times Data Warehouse. However, it is inevitable that any new data collection system, and especially one that represents as big a change as New Ways, will require time to settle down before NHS Boards' data is recorded nationally with full consistency.

This fourth publication of waiting times statistics since the introduction of 'New Ways' has provisional National Statistics status. The vast majority of the data have been judged fit for publication. Where the statistics are not yet finalised and may be subject to some change, they are considered to give a reasonable reflection of the current position and therefore to be useable by the public. In line with National Statistics requirements, as much as possible of the planned information is presented in this publication. Some data are not presented at this stage as they require further work to improve their reliability. Data previously published are updated in this publication to reflect on-going work to improve data quality. A description of known data quality issues is available here .

The process to assure the quality of the data began with implementation. The items held within the data warehouse have nationally consistent definitions. Training was provided for the staff in Boards involved. A New Ways website was created, making guidance readily available. Data submitted to the warehouse is subject to central validation. Within NHS Boards, great efforts were made to ensure that new systems and processes were working effectively. Finally, NHS Board Chief Executives were asked to sign off key statistics from the planned publication to confirm their accuracy.

The sections below describe in more detail aspects of the quality of the data held within the Waiting Times Data Warehouse.

Validation

Data completeness

Latest Revisions

Filtration system

Early 'New Ways' data

Data Quality Paper

Tables not included in this publication

The Data Quality Assessment project

Useful New Ways links

Validation

Data provided by NHS Boards must pass central validation before they are accepted into the Warehouse and therefore available for analysis. The validation rules and processes are maintained centrally so that all NHS Boards are working to the same standards. Any records which fail validation are reported back to data providers with an indication of the error. The validation rules are under regular review.

Data completeness

A number of NHS Boards report that, due to issues with local systems, there may be some incompleteness in the records returned to the data warehouse at ISD. This is being addressed by ISD and the NHS Boards. Further work is required to understand and assure these data.

Latest Revisions

Waiting times figures that were published previously have been revised to reflect the latest available update of records held in the national (New Ways) data warehouse (as at 2 February 2009).

The national data warehouse is an 'operational', dynamic database that gathers records very frequently from hospital patient administration systems for those people waiting for care. It was introduced in January 2008 and reflects more rigorous and complex but fairer national rules about measuring patient waits, in particular dealing with patient unavailability without excluding such from published statistics. It is inevitable that such a big change affecting the large number of services and the staff managing them requires a settling in period in terms of the processing of information and before fully comparable Scotland-wide data are available. The revised statistics reflect improvements to the quality of published data.

Any changes in the total volumes of patients seen or waiting equate to a relatively small proportion of total numbers of patients seen/waiting. There is not yet a full understanding of the reasons for these changes, however the revised figures are considered to be largely a result of improved data quality. These improvements include: increased completeness partly arising from improved data validity; and impact of the submission of updated records (for example changing records that were previously classed as 'waiting' at the end of the quarter to being classed as 'being seen' during the quarter). In terms of numbers of patients shown to have waited longer than 18 weeks the revisions are extremely small in relation to total number of patients seen/waiting.

Filtration system

The filter mechanism was introduced early in 2008, when it became apparent that, for some NHS Boards, some known errors in New Ways records could not be corrected in the files submitted to the New Ways database due to technical restrictions of their local IT systems.  The errors are due to a number of reasons, including system issues with data extration and incorrect data entry resulting in failed validation centrally.  This can result in records within the data warehouse not being successfully updated.

As a contingency measure, ISD has centrally filtered these erroneous New Ways records, which are not included in the published statistics.

ISD asked each NHS Board affected to provide a file detailing the records involved and the reasons why a filter is requested. This approach was endorsed on the basis that:

• it is an interim, auditable process and the filter will be removed as soon as possible;
• records are not 'deleted' from the central data warehouse, but filtered from analysis;
• NHS Boards are required to work with their system suppliers to resolve their technical issues before the next publication date.

The table below shows the volume of affected records by NHS Board. These cases represent a very small percentage of the total number of records and the 'filtering' has enabled the provisional publication of related statistics. When considering the filtration system it is important to do so in awareness that this is one of a number of ways in which data quality assurance is managed; there are others, including NHS Board verification of key statistics.

_{Note: IP/DC - Inpatients and Day cases, OP - New Outpatients}

Local 'filtering' of data

Some NHS Boards reported that they had locally 'filtered' (excluded) certain cases from submissions to the national warehouse. This was undertaken by NHS Boards who were not able, due to technical or other reasons, to make changes locally to erroneous data. The following NHS Boards advised that they had locally filtered erroneous records: Dumfries & Galloway, Forth Valley, Orkney, Tayside and Western Isles.

These cases represent a small percentage of the total number of patients and the local 'filtering' has enabled the provisional publication of related statistics.

Early 'New Ways' data

Information derived from New Ways data for early reporting periods are considered unlikely to be fully consistent with data for later periods. This is due to two main reasons, arising from the transition to New Ways:

• New Ways was introduced on 1 January 2008 and it is inevitable that such a big change will require a settling down period before fully comparable Scotland-wide data is available;
• Patients placed on waiting lists prior to 2008, and seen or waiting in 2008, may not have full New Ways information recorded against them (e.g. CNA, DNA, unavailability and offer information). In addition, patients who had Availability Status Codes (ASCs) prior to 2008 were allowed a period of 'retrospective unavailability' to be applied, to cover periods prior to 2008 (it is not known how many sites applied such 'retrospective unavailability').

New Ways records can therefore be categorised into the following:

• 'complete' New Ways data (i.e. for all patients added to lists in 2008 and for some added to lists in 2007);
• incomplete or compensatory New Ways data (i.e. for many patients added to lists prior to 2008.

Data Quality Paper

This document describes further, detailed issues affecting the quality and completeness of the data published on these webpages.

Tables not included in this publication

Prior to the implementation of New Ways, ISD and Scottish Government agreed a list of proposed tables for publication. Some of these analyses have not been included in this publication because the relevant data items need more work to improve their reliability.

• The analyses not presented at this time are: Waiting times for patients who had a procedure at a return outpatient clinic. Quality assurance showed there to be significant under-recording, at this stage, of procedures undertaken at return outpatient appointment. This statistic has not been previously published.
• Time from GP referral to receipt of referral at hospital. This is a new statistic and further work is required to understand and quality assure the statistics.
• Waiting times for patients seen at a Rapid Access Chest Pain Clinic. This table depends on the accurate recording of a field indicating which national standard the patient was covered by. Analyses showed there was significant under-recording at this stage and some inaccurate recording of this field.
• Did not attend / could not attend / cancellation by service rate. Analysis showed that most Boards currently have system and recording issues which affect these statistics.

A full list of the data items collected is published.

Data Quality Assessment Project

In 2008 ISD carried out a Data Quality Assessment of New Ways waiting times data on behalf of the Scottish Government, to assess the quality of New Ways data and to support NHS Boards with their own quality assessment processes. There were four strands to this project: a casenote review, an analysis of data from the national New Ways data warehouse, employee surveys and a patient survey. 
 
The results from the project and recommendations to support the further improvement in the quality of data are reported in the project's Executive Summary. Supplementary reports have also been produced for each of the four project strands and distributed to NHS Boards.

Useful New Ways links

New Ways Guidance  - Applying the Scottish Executive Health Department Guidance is a 24 page document which outlines the key elements of New Ways.

New Ways definitions - The Waiting Times Recording Manual guidance defines each of the data items which form the waiting times 'dataset' from 1 January 2008 onwards.

Calculation of wait - A technical document describing how the data warehouse calculates a patient's waiting time.

Scenarios - A technical document containing ten scenarios which together cover the eight broad categories of New Ways patient journeys (i.e. combinations of DNA/CNA, unavailability, multiple offers). These scenarios were provided for NHS Boards/system users to use on local systems to test these scenarios by comparison with results produced from the national warehouse (i.e. calculation of wait).

Main contact: Waiting Times Programme

 Printer friendly version