Assessment of the reliability of the Johns Hopkins/Agency for Healthcare Research and Quality hospital disaster drill evaluation tool.
STUDY OBJECTIVE: The Joint Commission requires hospitals to implement 2 drills per year to test the phase of their management plans. Despite this requirement, there is no direct evidence that such drills improve . Furthermore, there is no generally accepted, validated to evaluate hospital performance during disaster drills. We characterize the internal and interrater reliability of a hospital disaster drill performance developed by the Johns Hopkins University Evidence-based Practice Center, under contract from the Agency for Healthcare Research and Quality (AHRQ). METHODS: We evaluated the reliability of the Johns Hopkins/AHRQ drill performance tool by applying it to multiple hospitals in Los Angeles County, CA, participating in the November 2005 California statewide disaster drill. Thirty-two fourth-year medical student observers were deployed to specific zones (incident command, triage, treatment, and decontamination) in participating hospitals. Each completed common tool items, as well as tool items specific to their hospital zone. Two hundred items from the tool were dichotomously coded as indicating better versus poorer . An unweighted 'raw performance' score was calculated by summing these dichotomous indicators. To quantify internal reliability, we calculated the Kuder-Richardson interitem consistency coefficient, and to assess interrater reliability, we computed the kappa coefficient for each of the 11 pairs of observers who were deployed within the same hospital and zone. RESULTS: Of 17 invited hospitals, 6 agreed to participate. The raw performance scores for the 94 common items ranged from 18 (19%) to 63 (67%) across hospitals and zones. The raw performance scores of zone-specific items ranged from 14 of 45 (31%) to 30 of 45 (67%) in the incident command zone, from 2 of 17 (12%) to 15 of 17 (88%) in the triage zone, from 19 of 26 (73%) to 22 of 26 (85%) in the treatment zone, and from 2 of 18 (11%) to 10 of 18 (56%) in the decontamination zone. The Kuder-Richardson internal reliability, by zone, ranged from 0.72 (95% confidence interval [CI] 0.58 to 0.87) in the treatment zone to 0.97 (95% CI 0.95 to 0.99) in the incident command zone. The interrater reliability ranged, across hospital zones, from 0.24 (95% CI 0.09 to 0.38) to 0.72 (95% CI 0.63 to 0.81) for the 11 pairs of observers. CONCLUSION: We found a high degree of internal reliability in the AHRQ instrument's items, suggesting the underlying construct of hospital preparedness is valid. Conversely, we found substantial variability in interrater reliability, suggesting that the instrument needs revision or substantial user , as well as verification of interrater reliability in a particular setting before use.
All data were entered into a Microsoft Access database. Statistical analysis was performed using SAS version 8.1 and Stata SE 8.
Quantitative
Two hundred items from the were dichotomously coded as indicating better versus poorer
Thirty-two fourth-year medical student observers were deployed to specific zones (incident command, triage, treatment, and decontamination) in participating hospitals.Each completed common items, as well as items specific to their hospital zone
Data were obtained from the drill form from each of the observers and recorded on a closed data collection form.
The objective of our study was to assess the internal reliability and the interrater reliability of the AHRQ hospital drill performance by using a cohort of hospitals in Los Angeles County, participating in a statewide drill in November 2005.
This study was limited by the small number of hospitals and the modest number of pairs. Because only 6 of 17 hospitals agreed to participate, there is a possibility the hospitals were not representative.
We found a high degree of internal reliability in the AHRQ instrument’s items, suggesting the underlying construct of hospital is valid.
Six of the 17 invited hospitals agreed to participate.
The reliability of the Johns Hopkins/AHRQ drill performance was evaluated by applying it to multiple hospitals in Los Angeles County, CA, participating in the November 2005 California statewide drill.
 |
Portfolio of Solutions web site has been initially developed in the scope of DRIVER+ project. Today, the service is managed by AIT Austrian Institute of Technology GmbH., for the benefit of the European Management. PoS is endorsed and supported by the Disaster Competence Network Austria (DCNA) as well as by the STAMINA and TeamAware H2020 projects. |