BACKGROUND: Outcome misclassification is a potential threat to the validity of safety studies using administrative claims databases, particularly for outcomes such as anaphylaxis, which have been previously defined by conventional algorithms with poor positive predictive values (PPVs).
OBJECTIVES: Use medical record adjudication and machine learning predictive modeling methods to develop and validate an algorithm to identify anaphylaxis among adults with type 2 diabetes (T2D).
METHODS: A conventional screening algorithm to identify potential anaphylaxis cases that prioritized sensitivity was developed based on previous literature and consisted of diagnosis codes for anaphylaxis (ICD-10 CM: T78.2, T88.6, T80.5) or other signs and symptoms suggestive of the condition. This algorithm was applied to adults (>18) with claims for T2D in the HealthCore Integrated Research Database (HIRD) from 2016-2018. Clinical experts adjudicated anaphylaxis case status from redacted medical records. We used the confirmed case status to develop predictive models that utilized least absolute shrinkage and selection operator (LASSO) regression with 10-fold cross-validation that identified predictors in estimating the probability of confirmed anaphylaxis. Multiple models were evaluated, and in the best performing model we selected a probability threshold based on performance characteristics (i.e. PPV, sensitivity and specificity) and classified individuals above the probability threshold as cases.
RESULTS: Clinical adjudicators reviewed medical records from 272 patients with sufficient information identified by the anaphylaxis screening algorithm, which had an estimated PPV of 65% (95%CI: 60%-71%). The predictive model had a c-statistic=0.95. The model’s probability threshold of 0.60 excluded 89% (84/94) of false positives identified by the screening algorithm, which subsequently resulted in a PPV of 94% (95%CI: 91%-98%). The model excluded very few true positives (15 of 178), yielding a sensitivity of 92% (95%CI: 87%-96%).
CONCLUSION: Medical record review of the potential cases identified by the screening algorithm confirmed previous literature which note a limited performance of conventional algorithms for anaphylaxis. Machine learning techniques yielded an algorithm that achieved a substantially higher PPV for clinically confirmed anaphylaxis while retaining a similar number of cases (true positives) as the conventional screening algorithm. This new algorithm could be considered in future safety studies using claims data to reduce potential outcome misclassification.
