Data Quality

Introduction

ISD, NHS Boards and the Scottish Government have put in place measures to help ensure the quality of the data held within the Waiting Times Data Warehouse. However, it is inevitable that any new data collection system, and especially one that represents as big a change as New Ways, will require time to settle down before NHS Boards' data is recorded nationally with full consistency.

ISD developed a process for evaluating the fitness for publication of the data in the warehouse. An announcement describing the process was published on 1 May, and an Intention for publication followed on 20 May. The Director of ISD decided to proceed with the publication of provisional waiting times statistics on the pre-announced publication date of 27 May 2008.

This second publication of waiting times statistics since the introduction of 'New Ways' has Provisional National Statistics status. The vast majority of the data have been judged fit for publication. Where the statistics are not yet finalised and may be subject to some change, they are considered to give a reasonable reflection of the current position and therefore to be useable by the public. In line with National Statistics requirements, as much as possible of the planned information is presented in this publication. Some data are not presented at this stage as they require further work to improve their reliability. Data for the quarter end 31st March 2008 were published in May and are updated in this publication to reflect ongoing work by NHS boards to improve data quality. A description of revisions made to quarter end 31st March 2008 data is available here.

The process to assure the quality of the data began with implementation. The items held within the data warehouse have nationally consistent definitions. Training was provided for the staff in Boards involved. A New Ways website was created, making guidance readily available. Data submitted to the warehouse is subject to central validation. Within NHS Boards, great efforts were made to ensure that new systems and processes were working effectively. Finally, NHS Board Chief Executives were asked to sign off key statistics from the planned publication to confirm their accuracy. Longer term, a Data Quality Assessment project is due to report in October.

The sections below describe in more detail aspects of the quality of the data held within the Waiting Times Data Warehouse.

Validation

Data completeness

The filtration system

Early 'New Ways' data

Data Quality Paper

Tables not included in this publication

The Data Quality Assessment project

Useful New Ways links

The validation process

Data provided by NHS Boards must pass central validation before they are accepted into the Warehouse and therefore available for analysis. The validation rules and processes are maintained centrally so that all NHS Boards are working to the same standards. Any records which fail validation are reported back to data providers with an indication of the error. The validation rules are under regular review.

Data completeness

A number of NHS boards report that, due to issues with local systems, there may be some incompleteness in the records returned to the data warehouse at ISD. This is being addressed by ISD and the NHS Boards. Further work is required to understand and assure these data.

Filtration system

Some NHS Boards are currently unable to retrospectively correct known errors in some New Ways data items on their hospital systems.

ISD have centrally filtered these New Ways records, as a contingency measure. The approach that ISD have implemented is to create an 'exclusion' table to filter cases from analysis. In order for this exclusion table to be populated, ISD asked each board affected to provide a file detailing the records involved. This approach was enacted on the basis that:

• it is an interim measure and the filter will be removed as soon as possible;
• records are not be 'deleted' from the central data warehouse, but filtered from analysis;
• boards are required to amend the records in the normal manner, before the next publication date.

The table below shows the volume of affected records by NHS Board. These records were reported by Boards to be erroneously showing patient waits of over 18 weeks before they were filtered from the analysis. These cases are very small percentages of the total number of patients and the 'filtering' has enabled the provisional publication of related statistics.

Local 'filtering' of data

Some Boards reported that they had locally 'filtered' (excluded) certain cases from submissions to the national warehouse. This was undertaken by Boards who were not able, due to technical or other reasons, to make changes locally to erroneous data. The following Boards advised that they had locally filtered some records erroneously showing patients to be waiting, or to have waited, 18 weeks or more: Forth Valley, Tayside, Western Isles.

These cases are understood to be small percentages of the total number of patients and the local 'filtering' has enabled the provisional publication of related statistics.

Early 'New Ways' data

Information derived from New Ways data for early reporting periods are considered unlikely to be fully consistent with data for later periods. This is due to two main reasons, arising from the transition to New Ways:

• New Ways was introduced on 1 January 2008 and it is inevitable that such a big change will require a settling down period before fully comparable Scotland-wide data is available;
• Patients placed on waiting lists prior to 2008, and seen or waiting in 2008, may not have full New Ways information recorded against them (eg CNA, DNA, unavailability and offer information). In addition, patients who had Availability Status Codes (ASCs) prior to 2008 were allowed a period of 'retrospective unavailability' to be applied, to cover periods prior to 2008 (it is not known how many sites applied such 'retrospective unavailability').

New Ways records can therefore be categorised into the following:

• 'complete' New Ways data (ie for all patients added to lists in 2008 and for some added to lists in 2007);
• incomplete or compensatory New Ways data (ie for many patients added to lists prior to 2008.

Data Quality Paper

This document describes further, detailed issues affecting the quality and completeness of the data published on these webpages.

Tables not included in this publication

Prior to the implementation of New Ways, ISD and Scottish Government agreed a list of proposed tables for publication. Some of these analyses have not been included in this publication because the relevant data items need more work to improve their reliability.

• The analyses not presented at this time are: Waiting times for patients who had a procedure at a return outpatient clinic. Quality assurance showed there to be significant under-recording, at this stage, of procedures undertaken at return outpatient appointment. This statistic has not been previously published.
• Time from GP referral to receipt of referral at hospital. This is a new statistic and further work is required to understand and quality assure the statistics;
• Waiting times for patients seen for an outpatient assessment for cataracts and for patients seen at a Rapid Access Chest Pain Clinic. These tables depend on the accurate recording of a field indicating which national standard the patient was covered by. Analyses showed there was significant under-recording at this stage and some inaccurate recording of this field.
• Did not attend / could not attend / cancellation by service rate. Analysis showed that most Boards currently have system and recording issues which affect these statistics.

A full list of the data items collected is published.

Data Quality Assessment Project

ISD is currently carrying out a Data Quality Assessment of New Ways data on behalf of the Scottish Government, to assess the quality of New Ways data and to support Boards with their own quality assurance processes.

The project covers four areas in the assessment of quality as follows:

1. QA Audit

ISD's Data Quality Assurance (DQA) team has taken a random sample of records from selected hospitals to assess the New Ways data items submitted to the national data warehouse via the hospital PAS/HISS. This is being compared with data held locally on case notes and systems.

2. Desktop Analysis

Much QA information can be obtained by a central analysis approach, for example:

• Consistencies between New Ways data and other datasets where there is an obvious relationship e.g. SMR00, SMR01 and ISD(S)1;
• Comparison between hospitals/NHS Boards of individual New Ways data items, e.g. offers, referrals, cancellations, DNAs/CNAs, unavailability, removals, suspensions and referrals back to GPs etc.  Subsequent investigation of exceptions will take place;
• Analyse levels of data queries/errors/amendments.

Desktop analysis is being carried out on all hospitals.

3. Questionnaire

A questionnaire has been issued to hospital staff to determine local practices and protocols for the collection of the New Ways data. The questionnaire also looks at hospital staff training and awareness of New Ways. This will provide 'softer' information which may point to areas where best practice exists and to where improvements could be made.

4. Patient Involvement

A random sample of patients have been issued with a questionnaire asking about their experience of the waiting times process under New Ways.

Useful New Ways links

New Ways Guidance  - Applying the Scottish Executive Health Department Guidance is a 24 page document which outlines the key elements of New Ways.

New Ways definitions - The Waiting Times Recording Manual guidance defines each of the data items which form the waiting times 'dataset' from 1 January 2008 onwards.

Calculation of wait - A technical document describing how the data warehouse calculates a patient's waiting time.

Scenarios - A technical document containing ten scenarios which together cover the eight broad categories of New Ways patient journeys (i.e. combinations of DNA/CNA, unavailability, multiple offers). These scenarios were provided for Boards/system users to use on local systems to test these scenarios by comparison with results produced from the national warehouse (i.e. calculation of wait).

Main contact: Waiting Times Programme

 Printer friendly version