Evaluation of monitoring systems in cardiology

1 - Background

 Clinical diagnoses and investigations are dependent on the ability to record, analyse and interpret physiological signals. Analysing biological systems is still a challenging task, despite technological advances that have allowed for the recording and storage of massive datasets of continuously fluctuating signals.

Methods for signal processing and interpretation are more and more sophisticated and are being developed to study physiological systems with regard to their complexity and involved measurement constraints.

In Cardiology, automated systems are widely applied to the monitoring of patients through the analysis and interpretation of biomedical signals. They range from ECG analysers (resting, exercise, Holter) to multiparameter monitoring systems in critical care settings (CCU, ICU, Operating Room, Cath Lab).

As clinical experience on one hand, has led to more and more prognostic indicators, and biomedical technology on the other hand, has advanced as well, increasingly sophisticated ECG computer-based systems have been developed, such as electrocardiographers automatically measuring signal parameters, long term ECG analysers detecting arrhythmias and transient “ischemic-like” ST change episodes, or advanced systems providing cardiologists with diagnostic information.

2 - Reference databases for system evaluation

Automated analysis of physiologic signals is critically affected by the wide inter- and intra-patient variability of normal and abnormal features or by the presence of noise and/or artefacts, often making the detection of subtle changes of waveform features difficult.

Hence users are concerned with a basic question: “Does the device perform reliably and does it deliver accurate information?”

A meaningful answer can be handed to us by testing the device on collections of “real world” physiologic signals, obtained in a comprehensive variety of healthy and pathological conditions. Using these databases as a reference, manufacturers and medical/research communities are allowed to measure and compare performances of commercial systems.

Many databases, approaching the status of “gold standard”, have been made available, particularly in long term ECG monitoring, each containing well-characterised digital signal recordings representing clinical variability and including rarely observed significant abnormalities. Each signal was annotated by cardiologists, beat by beat (e.g. QRS complex) or episode by episode (e.g. arrhythmia, transient ST change), thus allowing quantitative and reproducible comparison of device analysis.

1. The Massachusetts Institute of Technology – Beth Israel Hospital (MIT-BIH) Arrhythmia database (1,2) was the first generally available set of Holter ECGs (48 half-hour two-channel records, beat-by-beat annotated) for the evaluation of arrhythmia detectors, and has been used for that purpose as well as for basic research into cardiac dynamics in 500 sites worldwide since 1980.
   Together with the American Heart Association (AHA) database (3), it stimulated manufacturers to compete on objectively measurable performances.
2. The European ST-T Database (ESCDB) (4,5) was then released to document significant ischemic-like abnormalities in long term ECG (90 two-hour two-channel records, beat-by-beat annotated), just contributing to the development and evaluation of systems for the analysis of transient episodes of myocardial ischemia. Since 1990 it has been applied in 100 sites worldwide.
3. The ESCDB project originated from the Concerted Action on Ambulatory Monitoring and was finally supported by the European Society of Cardiology. Recently the Long Term ST Database (LTSTDB) was developed (6), by a joint effort of the University of Ljubljana, Beth Israel Deaconess Medical Center and CNR Institute of Clinical Physiology, to complement ESCDB including 24-hour Holter records (86 two- and three-channel records). A common annotation protocol was adopted for developing the different databases.

3 - PhysioNet: Web resource for distribution and exchange of physiologic signals

PhysioNet (http://www.physionet.org/), a project developed by Harvard-MIT (7) under the auspices of the NIH's National Center for Research Resources, was intended to stimulate research by providing an on-line forum for free dissemination and exchange of recorded biomedical signals (PhysioBank) and open-source analysis software (PhysioToolkit).

PhysioBank is a large and growing archive of well-characterised digital recordings of physiologic signals and related data for use by the biomedical research community. PhysioBank currently includes databases of multi-parameter cardiopulmonary, neural, and other biomedical signals from healthy subjects and patients with a variety of conditions. PhysioBank now contains over 40 databases that may be freely downloaded. Over half of ESTDB and LTSTDB are also freely available from PhysioNet.

4 - Standards for performance evaluation

Specific protocols were defined by ANSI-AAMI (EC38, EC57) (actually recognized by FDA  as consensus standards) (8,9) for evaluating performance of automated ECG analysers. A series of evaluation software, in compliance with such protocols, was included in the WFDB Software Package on PhysioNet (10). The ANSI-AAMI standards make a list of the ECG databases suggested for performance evaluation.

The evaluation process is to present the reference signals to the device under test and to then collect the annotations files from the device. A comparison of test and reference annotation files is then performed for each record of the reference database.

Performance statistics (sensitivity, positive predictive accuracy or specificity) are derived for measuring the detection of each event of clinical interest (beats, arrhythmias or ST-T change episodes), thus allowing the assessment of the system’s quality under test and driving developers to revise and improve the devices.

Using standard protocols and reference signal databases, manufacturers are today enabled to qualify reproducibly their instrumentation, finally providing clinicians with information useful for assessment and comparative evaluation.

Automated medical instrumentation - made available by the continuous advances in technology and biomedical research - is largely available in clinical practice today.

Reliability and effectiveness should always be quantitatively assessed however, given the complexity and variability of physiological systems that need analysed. To evaluate the performances of ECG monitoring systems, the use of reference databases of annotated signals is a standard de-facto.

The content of this article reflects the personal opinion of the author/s and is not necessarily the official position of the European Society of Cardiology.