Data Quality

Introduction

ISD, NHS Boards and the Scottish Government have put in place measures to help ensure the quality of the data held within the Waiting Times Data Warehouse. However, it is inevitable that any new data collection system, and especially one that represents as big a change as New Ways, will require time to settle down before NHS Boards' data is recorded nationally with full consistency.

The vast majority of the data have been judged fit for publication. Where the statistics are not yet finalised and may be subject to some change, they are considered to give a reasonable reflection of the current position and therefore to be useable by the public. In line with National Statistics requirements, as much as possible of the planned information is presented in this publication. Some data are not presented at this stage as they require further work to improve their reliability. Data previously published are updated in this publication to reflect on-going work to improve data quality. A description of known data quality issues is available here .

The process to assure the quality of the data began with implementation. The items held within the data warehouse have nationally consistent definitions. Training was provided for the staff in NHS Boards involved. A New Ways website was created, making guidance readily available. Data submitted to the warehouse is subject to central validation. Within NHS Boards, great efforts were made to ensure that new systems and processes were working effectively. Finally, NHS Board Chief Executives were asked to sign off key statistics from the planned publication to confirm their accuracy.

The sections below describe in more detail aspects of the quality of the data held within the Waiting Times Data Warehouse.

Validation

Data completeness

Latest Revisions

Filtration system

New Ways Refresh

Data Quality Paper

New Publication Tables

Tables not included in this publication

The Data Quality Assessment project

Early 'New Ways' data

Useful New Ways links

Validation

Data provided by NHS Boards must pass central validation before they are accepted into the Warehouse and therefore available for analysis. The validation rules and processes are maintained centrally so that all NHS Boards are working to the same standards. Any records which fail validation are reported back to data providers with an indication of the error. The validation rules are under regular review.

Data completeness

A number of NHS Boards report that, due to issues with local systems, there may be some incompleteness in the records returned to the data warehouse at ISD. This is being addressed by ISD and the NHS Boards. Further work is required to understand and assure these data.

Latest Revisions

Waiting times figures that were published previously have been revised to reflect the latest available update of records held in the national (New Ways) data warehouse (as at 28 January 2010).

The national data warehouse is an 'operational', dynamic database that gathers records very frequently from hospital patient administration systems for those people waiting for care. It was introduced in January 2008 and reflects more rigorous and complex but fairer national rules about measuring patient waits, in particular dealing with patient unavailability without excluding such from published statistics. It is inevitable that such a big change affecting the large number of services and the staff managing them requires a settling in period in terms of the processing of information and before fully comparable Scotland-wide data are available. The revised statistics reflect improvements to the quality of published data.

Any changes in the total volumes of patients seen or waiting equate to a relatively small proportion of total numbers of patients seen/waiting. There is not yet a full understanding of the reasons for these changes, however the revised figures are considered to be largely a result of improved data quality. These improvements include: increased completeness partly arising from improved data validity; and impact of the submission of updated records (for example changing records that were previously classed as 'waiting' at the end of the quarter to being classed as 'being seen' during the quarter). In terms of numbers of patients shown to have waited longer than the national standard the revisions are extremely small in relation to total number of patients seen/waiting.

Filtration system

The filter mechanism was introduced early in 2008, when it became apparent that, for some NHS Boards, some known errors in New Ways records could not be corrected in the files submitted to the New Ways database due to technical restrictions of their local IT systems.  The errors are due to a number of reasons, including system issues with data extraction and incorrect data entry resulting in failed validation centrally.  This can result in records within the data warehouse not being successfully updated.

As a contingency measure, ISD has centrally filtered these erroneous New Ways records, which are not included in the published statistics.

ISD asked each NHS Board affected to provide a file detailing the records involved and the reasons why a filter is requested. This approach was endorsed on the basis that:

• it is an interim, auditable process and the filter will be removed as soon as possible;
• records are not 'deleted' from the central data warehouse, but filtered from analysis;
• NHS Boards are required to work with their system suppliers to resolve their technical issues.

The table below shows the volume of affected records by NHS Board. These cases represent a very small percentage of the total number of records and the 'filtering' has enabled the provisional publication of related statistics. When considering the filtration system it is important to do so in awareness that this is one of a number of ways in which data quality assurance is managed; there are others, including NHS Board verification of key statistics.

_{Notes: IP/DC - Inpatients and Day cases, OP - New Outpatients
This table shows filtered records for the last year. Data for periods prior to 2009 are given at:   Number of records filtered, by NHS Board (20KB)}

Local 'filtering' of data

Some NHS Boards reported that they had locally 'filtered' (excluded) certain cases from submissions to the national warehouse. This was undertaken by NHS Boards who were not able, due to technical or other reasons, to make changes locally to erroneous data. The following NHS Boards advised that they had locally filtered erroneous records: Dumfries & Galloway, Orkney and Tayside.

These cases represent a small percentage of the total number of patients and the local 'filtering' has enabled the provisional publication of related statistics.

New Ways Refresh

The New Ways Refresh project aimed to help reduce the administrative effort on staff involved in collecting and monitoring waiting times data while still retaining the transparency of waiting times reporting, as intended by the original New Ways policy. It is hoped that the refresh project helps NHS Boards in monitoring the 18 Weeks Referral to Treatment standard.

The changes implemented through New Ways Refresh ensured that the majority of records enter the data warehouse, irrespective of data quality and addressing what has been a long standing issue for Health Boards. The refresh also introduces a new four 'flag' scale for records so users can quickly identify which records are free of errors right through to records that are not valid for waiting time calculations.

This is the first publication to incorporate 'Refresh' data, which has led to minimal differences in the data between this and previous publications. More details are also available here.

Data Quality Paper

This document describes further, detailed issues affecting the quality and completeness of the data published on these webpages.

New Publication Tables

There have been some key changes to the Waiting Times publication for the quarter ending December 2009.

Tables A1c/d - New Outpatient Tables:

The addition of the new outpatient tables (A1c and A1d) reflect ongoing and completed waits for all referral sources. These tables give a breakdown of the number of patients waiting/waited over 12 weeks and are published in anticipation of the ?all referral sources? standard for new outpatients from 31 March 2010.

Within tables A1c and A1d, ISD have only published waits for patients 'added to the waiting list' from 1^st April 2009, one year prior to the introduction of the new standard. This new methodology prevents NHS Boards from having to quality assure historical data within the New Ways warehouse that applies to patients from over 12 months prior to the introduction of the new standard for outpatients; which was felt would be of 'disproportionate burden' for NHS Boards.

Figures for any patients added before 1^st April 2009 have been provided by NHS Boards from their local records and are published in a separate table.

Table I3 - 2nd eye/one-stop cataract clinics:

A new cataract table for second eye and one stop cataract clinics has been added to the publication suite.  

Additional changes:

All cardiac and cataract procedures have been removed from the Summary of Waiting Times tables (A tables) and all related tables and is published separately in the Cardiac and Cataract publication tables (Tables H and I).

Tables not included in this publication

Prior to the implementation of New Ways, ISD and Scottish Government agreed a list of proposed tables for publication. Some of these analyses have not been included in this publication because the relevant data items need more work to improve their reliability.

• The analyses not presented at this time are: Waiting times for patients who had a procedure at a return outpatient clinic. Quality assurance showed there to be significant under-recording, at this stage, of procedures undertaken at return outpatient appointment. This statistic has not been previously published.
• Time from GP referral to receipt of referral at hospital. This is a new statistic and further work is required to understand and quality assure the statistics.
• Waiting times for patients seen at a Rapid Access Chest Pain Clinic. This table depends on the accurate recording of a field indicating which national standard the patient was covered by. Analyses showed there was significant under-recording at this stage and some inaccurate recording of this field.

A full list of the data items collected is published.

Data Quality Assessment Project

In 2008 ISD carried out a Data Quality Assessment of New Ways waiting times data on behalf of the Scottish Government, to assess the quality of New Ways data and to support NHS Boards with their own quality assessment processes. There were four strands to this project: a casenote review, an analysis of data from the national New Ways data warehouse, employee surveys and a patient survey. 
 
The results from the project and recommendations to support the further improvement in the quality of data are reported in the project's Executive Summary. Supplementary reports have also been produced for each of the four project strands and distributed to NHS Boards.

Early 'New Ways' data

Information derived from New Ways data for early reporting periods are considered unlikely to be fully consistent with data for later periods. This is due to two main reasons, arising from the transition to New Ways:

• New Ways was introduced on 1 January 2008 and it is inevitable that such a big change will require a settling down period before fully comparable Scotland-wide data is available;
• Patients placed on waiting lists prior to 2008, and seen or waiting in 2008, may not have full New Ways information recorded against them (e.g. CNA, DNA, unavailability and offer information). In addition, patients who had Availability Status Codes (ASCs) prior to 2008 were allowed a period of 'retrospective unavailability' to be applied, to cover periods prior to 2008 (it is not known how many sites applied such 'retrospective unavailability').

New Ways records can therefore be categorised into the following:

• 'complete' New Ways data (i.e. for all patients added to lists in 2008 and for some added to lists in 2007);
• incomplete or compensatory New Ways data (i.e. for many patients added to lists prior to 2008.

Useful New Ways documents

New Ways guidance is available here including a 24 page document attaining the key elements of New Ways, the Waiting Times Recording Manual and a technical document describing how the data warehouse calculates a patients waiting time.

Main contact: Waiting Times Programme

 Printer friendly version