---
title: 'Quick Introduction to _icd_: ICD-9 and ICD-10 comorbidities, validation and manipulation'
author: "Jack O. Wasey"
date: '`r format(Sys.time(), "%Y-%m-%d")`'
output:
  html_document:
    toc: yes
  pdf_document:
    toc: yes
vignette: >
  %\VignetteIndexEntry{Quick Introduction to icd: ICD-9 and ICD-10 comorbidities, validation and manipulation}
  %\VignetteEngine{knitr::rmarkdown}
  \usepackage[utf8]{inputenc}
---
<!--
Copyright (C) 2014 - 2018  Jack O. Wasey

This file is part of icd.

icd is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

icd is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with icd. If not, see <http:#www.gnu.org/licenses/>.
-->
```{r setup, echo = FALSE, cache = FALSE}
suppressWarnings({
  suppressPackageStartupMessages({
    loadNamespace("knitr") # for opts_chunk only
    library(icd)
    library(magrittr)
    library(utils)
    })
  })

knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  fig.path = "README-"
)

patients_icd9 <- data.frame(
  visit_id = c(1000, 1000, 1000, 1000, 1001, 1001, 1002),
  icd9 = as.icd9(c("40201", "2258", "7208", "25001", "34400", "4011", "4011")),
  poa = c("Y", NA, "N", "Y", "X", "Y", "E"),
  stringsAsFactors = FALSE
  )
```

## Introduction

<!-- _TODO: update with first paragraph from DESCRIPTION_ -->
```{r pkgdesc, results='asis', echo = FALSE}
cat(packageDescription("icd")$Description)
```

When calculating which patients have which comorbidities, the data are typically structured in long or wide formats:
```{r show data formats, echo=TRUE}
# long format ICD-9-CM codes, with present-on-arrival flags
patients_icd9

# long format ICD-10 codes, real mortality data
uranium_pathology[1:5, ]

# wide format, real ICD-9 discharge diagnoses
vermont_dx[1:5, c(1, 6:15)]
```

In real life, there are often problems with the data, such is `NA` entries, out-of-order `visit_id`s, non-existent or invalid ICD codes, etc.. Although standard R tools can be used to clean the data, knowing the specific validation rules for ICD-9 and ICD-10 codes, as well as the standardized structure of the data enables faster and more accurate data cleaning.

One use of this package is to identify diagnoses present on admission to hospital (POA). If a POA field is present in the data, it can be used to filter the diagnosis codes. Let's walk through an example for some ICD-9 data:

```{r getcomorbidities}
# use AHRQ revision of Elixhauser comorbidities, show only first eight columns
icd9_comorbid_ahrq(patients_icd9)[, 1:8]
```

Things work beautifully using [magrittr](https://cran.r-project.org/package=magrittr) `%>%` to chain functions together. I use `magrittr` in the package itself, and I recommend trying it to clarify chains of commands, such as the following:
```{r getcomorbidities2}
# find Elixhauser comorbidities present-on-arrival
patients_icd9 %>% icd_filter_poa %>% icd9_comorbid_elix %>% head

# same as above, then summarize first five:
patients_icd9 %>% 
  icd_filter_poa %>% 
  icd9_comorbid_elix %>% 
  extract(, 1:5) %>% 
  apply(2, as.integer) %>% 
  summary

# convert vermont discharge data to wide format, 
# find comorbidities, convert TRUE to 1 and show first few
vermont_dx %>% icd_wide_to_long  %>% icd9_comorbid_quan_deyo  %>% apply(2, as.integer) %>%  head
```

The above can be rewritten in classic R with many parentheses:
```{r lots of brackets, eval = FALSE}
head(apply(icd9_comorbid_quan_deyo(icd_wide_to_long(vermont_dx)), 2, as.integer))
```

## Specifying data types

`icd` will guess the type and form of input data when possible, but there are ambiguities:
```{r type guessing}
icd_is_valid("100") # valid ICD-9 code
icd_is_valid("A1001") # valid ICD-10 code
icd_is_valid(c("100", "A1001")) # they can't both be valid
```

You can let `icd` guess types, or specify the type of your data explicitly:
```{r set type}
# decimal format ICD-10 codes
codes <- c("A10.01", "L40.50", "Z77.098")
# set class to be icd10cm (and implicitly icd10)
as.icd10cm(codes)
# or set class to indicate decimal code and icd10 (not necessarily icd10cm)
codes %>% as.icd_decimal_diag %>% as.icd10
```
Doing this avoids mistakes in guessing type. For example code `V10` is valid in both ICD-9 and ICD-10.

Vectors of codes, data frames and matrices can have a class set. Conflicting classes are not allowed, e.g. if a data frame has columns with both ICD-9 and ICD-10 codes, it doesn't make sense to set an ICD version for the data frame.
```{r mixed ICD-9 and ICD-10 data}
df <- data.frame(i9 = as.icd9(c("100", "001")), 
                 i10 = as.icd10(c("Z771", "Z87820")))

# it is an error to try to convert ICD-9 class codes to ICD-10
# df %>% as.icd9 %>% as.icd10
```

## Converting ICD-9 codes between types
ICD-9 codes are usually presented in *decimal* format (beware, for this is not a number), e.g. `003.21`, whereas most electronic records seem to use the *short* form without a decimal place. These are not interchangeable simply by removing the decimal place, and great care is taken to do this correctly. The functions were also designed to deal with the common problem of incorrectly formatted ICD-9 codes. The assumption is made that short codes of three or fewer characters are  describing only the 'major' part: there is no other reasonable interpretation. For example, `020` must be taken to mean `20`, not `2.0` or even `0.20`. In most cases, when *icd* works on ICD-9 codes, it will convert any codes of fewer than three characters into zero-padded three-digit codes.
```{r simple conversion}
icd_decimal_to_short(c("1", "10.20", "100", "123.45"))
icd_short_to_decimal(c("1", "22", "2244", "1005"))

# similar operations with magrittr, also showing invalid codes
codes <- as.icd9(c("87.65", "9999", "Aesop", -100, "", NA))
icd_decimal_to_short(codes)
```

## Validation of ICD-9 codes
```{r validation}
# guess both ICD version (9, but could be 10?), and decimal vs short form
icd_is_valid("V10.2")

# state we are using short or decimal codes:
icd_is_valid(c("099.17", "-1"), short_code = TRUE)
icd_is_valid(c("099.17", "-1.1"), short_code = FALSE)
icd_is_valid(c("1", "001", "100", "123456", "003.21"), short_code = TRUE)
```

## Ranges of ICD-9 codes

These functions generate syntactically valid ICD-9 codes, without including parent codes when the range limit would subset the parent. E.g. `"100.99" %i9da% "101.01"` does not include `100` or `100.0`, both of which imply larger subsets than requested by the range command (i.e. every code up to `100.99`). The shorter forms `%i9s%` and `%i9d%` return only *real* codes (i.e. listed in the CMS definitions as either three-digit codes or diagnoses), whereas `%i9sa%` and `%i9da%` return all possible syntactically valid ICD-9 codes:
```{r ranges}
# get all possible codes
#"003" %i9sa% "0033" %>% head(9) # show first 9 of 111 values
# just get the ones which correspond to diagnoses (keeping the 3-digit chapters)
#"494" %i9s% "4941"

#"10099" %i9sa% "10101"
#"V10" %i9da% "V10.02"
"E987" %i9da% "E988.1"

# can't range between different types:
# "V10" %i9s% "E800" # throws an error
```

This is used internally to interpret ranges of ICD-9 codes specified in the literature. Sometimes it is not clear exactly what an ICD-9 range presented in a paper means, but at least we can explicitly decide what should be included in our interpretation, and the ranges can be reused even when the underlying codes may be different, as codes are added and removed from time-to-time, and although the original papers would have been based on their ICD-9 ranges resolving to a specific set of codes, they are likely to be valid for new diagnoses in the given subgroups. Ultimately, this needs detailed attention, but the strategy in \code{icd9} is to give a good best guess, given these limitations.

Another way of specifying ranges are to use function calls. These are exactly equivalent to the `%i9s%` and `%i9d%` range operators. This example shows the result when the user specifies a range which would include parents but not all their children:

```{r rangeanomaly}
icd_expand_range("4820", "4823") # default, equivalent to %i9s%
icd_expand_range("4820", "4823", defined = FALSE)
# see the first few differences (which are by definition not 'real' codes):
setdiff(icd_expand_range("4820", "4823", defined = FALSE),
        icd_expand_range("4820", "4823")) %>% head
```

It is easy to find the children of a higher-level ICD-9 code:
```{r "childrenReal"}
icd_children(as.icd9("391"))
# mid-level code
icd:::icd_children.icd9("0032")
# leaf node has no children
# be explicit about the type of code:
test_code <- as.icd9(as.icd_short_diag("00321"))
icd_children(test_code)
# or the same, but guessing the characteristics
icd_children("00321")
# pneumococcal pneumonia is a three-digit ICD-9 code with no descendants
icd_children("481")
```
By adding `defined = TRUE`, all syntactically valid ICD-9 codes are returned, even if not defined by CMS as diagnoses. This is relevant because of minor coding errors, or coding in a different year to the master list. A planned feature is to allow testing of an ICD-9 code against the valid codes for the year it was entered, but at present only the 2014 master list is used. This means that some older valid codes may no longer be on the list. However, there have been very few changes to ICD-9-CM in the last five years with ICD-10-CM in the wings.
```{r all children}
# first ten possible ICD-9 child codes from 391
icd_children("391", defined = FALSE)[1:10]
```

## Decoding ICD-9 codes to descriptions
There are various ways of extracting the description of the condition described by an ICD-9 code. the icd_explain group of functions return a data frame with a column for the ICD-9 code, a column for the full length Diagnosis, and a column for the short Description.
```{r explain simple}
icd_explain("1.0") # 'decimal' format code inferred
icd_explain("0019") # 'short' format code inferred
```

```{r explain complex}
# we can be explicit about short vs decimal
icd_explain("434.00", short_code = FALSE)
icd_explain(c("43410", "43491"), short_code = TRUE)
#explain top level code with children
"391" %>% icd_explain # single three-digit code
"391" %>% icd_children # let's see the child codes
"391" %>% icd_children %>% icd_explain # children condensed to parent code
"391" %>% icd_children %>% icd_explain(condense = FALSE) # prevent condense
```
Arbitrary named list(s) of codes:
```{r explain arbitrary}
icd_explain(list(somecodes = as.icd9(c("001", "391")),
                 morecodes = as.icd9cm(c("001.1", "001.9"))))
```
`001` (Cholera) isn't itself a diagnostic code, i.e. leaf node in the hierarchy, but `390` (Rheumatic fever without heart involvement) is. Both are explained correctly:
```{r cholera}
icd_explain(list(cholera = "001", rheumatic_heart = "390"))
```
Now try to explain on a non-existent (but 'valid') ICD-9 code:
```{r noexplain, eval = FALSE}
s <- icd_explain("001.5") # gives warning
```
As we have just seen, *icd_explain* can convert lists of ICD-9 or ICD-10 codes to a human-readable format. Let's apply the `icd_explain` to a list of comorbidity ICD-9 codes in one of the commonly-used mappings. This makes comprehending a complicated list much easier. Taking the list for dementia:
```{r Example Dementia}
length(icd9_map_quan_deyo[["Dementia"]]) # 133 possible ICD-9 codes
length(icd10_map_quan_deyo[["Dementia"]]) # the ICD-10 map is different
# icd_explain summarizes these to just two groups:
icd9_map_quan_deyo[["Dementia"]] %>% icd_explain(warn = FALSE)
# contrast with:
icd9_map_quan_deyo[["Dementia"]] %>% icd_explain(condense = TRUE, warn = FALSE)
```
Use a range with more than two hundred ICD-9 codes (most of them not real):
```{r Show Range Operator}
length("390" %i9da% "392.1")
"390" %i9da% "392.1" %>% icd_explain(warn = FALSE)
```
The warnings here are irrelevant because we know that `%i9da% produces codes which do not correspond to diagnoses. However, in other usage, the user would typically expect the ICD-9 codes he or she is using to be diagnostic, hence the default to warn.

## Filtering by Present-on-Arrival

This flag is recorded with each ICD-9 code, indicating whether that diagnosis was present on admission. With some caution, codes flagged specifically not *POA* can be treated as new diseases during an admission.

Present-on-arrival (POA) is typically a factor, or vector of values such as "Y", "N", "X", "E", or NA. Intermediate codes, such as "exempt", "unknown" and NA mean that "yes" is not the same as "not no." This requires four functions to cover the possibilities stored in `icd_poa_choices`:
```{r Show POA Choices, echo=FALSE}
icd_poa_choices
```
Filter for present-on-arrival being "Y"
```{r simplepoa}
patients_icd9 %>% icd_filter_poa_yes
```
Show that *yes* is not equal to *not no* (e.g. due to NA in `poa` field)
```{r notnopoa}
patients_icd9 %>% icd_filter_poa_not_no
```

## Comorbidities
The comorbidities from different sources are provided as lists. At present only the most recent mapping of ICD-9 codes to comorbidities is provided. See these [github](https://github.com/jackwasey/icd/issues/6) [issues](https://github.com/jackwasey/icd/issues/7).

This package contains ICD-9-CM to comorbidity mappings from several sources, based on either the Charlson or Elixhauser lists of comorbidities. Updated versions of these lists from [AHRQ](http://www.hcup-us.ahrq.gov/toolssoftware/comorbidity/comorbidity.jsp) and [Quan et al](http://www.ncbi.nlm.nih.gov/pubmed/16224307) are included, along with the original Elixhauser mapping . Since some data is provided in SAS source code format, this package has internal functions to parse this SAS source code and generate R data structures. This processing is limited to what is needed for this purpose, although may be generalizable and useful in other contexts. Other lists are transcribed directly from the published articles, but interpretation of SAS code used for the original publications is preferable.

### AHRQ comorbidity classification

The AHRQ keeps an updated version of the Elixhauser classification of ICD-9-CM codes into comorbidities, useful for research. They provide the data in the form of SAS code. The names of the comorbidities derived from ICD-9 and ICD-10 codes are the same. Maps contain the ICD code to comorbidity mappings; the functions that apply those mappings are called things like `icd10_comorbid_ahrq`.
```{r ahrq}
#icd9_map_ahrq <- icd:::sas_parse_ahrq() # user doesn't need to do this
names(icd9_map_ahrq)
icd9_map_ahrq$CHF[1:5]
icd10_map_ahrq$CHF[1:5]
```

### Elixhauser comorbidities
Elixhauser originally devleoped this set of comorbidities to predict long term mortality based on hospital ICD-9-CM coding records. The AHRQ comorbidities are an updated version of this, however the original Elixhauser have been used in many publications. The ICD-9-CM codes have changed slightly over the years.
```{r elix}
# the names of the comorbidities in each map are available as named lists:
icd_names_elix[1:5]
unlist(unname(icd_names_elix))
# The map contents have ICD codes with the class set
icd9_map_elix$HTNcx
icd10_map_elix$HTNcx
```

### Quan
Quan's paper looked at indices using both ICD-10 and ICD-9-CM. Quan generated updated ICD-9-CM codes for all 30 of Elixhauser and all 17 of Charlson/Deyo's comorbidities. Thus there are two 'Quan' comorbidity mappings.
```{r quan elix}
names(icd10_map_quan_deyo)
names(icd10_map_quan_elix)
```

## Examples

### Filter patients and create comorbidities
Take my patients, find the ones where there definitely or maybe was a diagnosis present on admission, then generate comorbidities based on the AHRQ mapping. N.b. `NotNo` is not the same as `Yes` because of some exempt, unclassifiable conditions, or `NA` values for the present-on-admission flag.
```{r chainpoatocomorbid}
patients_icd9 %>%
  icd_filter_poa_not_no %>%
  icd9_comorbid_ahrq %>%
  extract(1:9)
```

### Compare two comorbidity definitions
We will find the differences between some categories of the original Elixhauser and the updated version by Quan. Just taking the select few comorbidity groups for brevity:
```{r elixvsquanelix}
difference <- icd_diff_comorbid(icd9_map_elix, icd9_map_quan_elix,
                 all_names = c("CHF", "PHTN", "HTN", "Valvular"))
# reuslts also returned as data
str(difference)
```
Which pulmonary hypertension codes are only in Quan's version?
```{r quanonlyphtn}
difference$PHTN$only.y %>% icd_get_defined %>% icd_explain
```
(Passing through `icd_get_defined` stops `icd_explain` complaining that some of the input codes don't exist. This is because the comorbidity mappings have every possible numerical ICD-9 code, not just the official ones. Could also use `warn = FALSE` option in `icd_explain`)

### Find cardiac-related ICD-9 codes:
```{r cardiacgrep}
icd9cm_hierarchy[
  grepl(pattern = "(heart)|(cardiac)",
        x = c(icd9cm_hierarchy$long_desc, icd9cm_hierarchy$short_desc),
        ignore.case = TRUE),
  "code"] %>% unique -> cardiac
```
then explain the list, just showing the first ten:
```{r cardiac Chain Explain Example}
as.icd9(cardiac) %>% icd_explain(warn = FALSE) %>% head(10)
```

### Find comorbidities for a large number of patients.
I understand that comorbiditity assignment using SAS is a lengthy business. Let's generate 100,000 patients with a random selection of comorbidities:
```{r speed, eval = FALSE}
# codes selected from AHRQ mapping
many_patients <- icd:::generate_random_pts(1e7) 
system.time(
  icd999999999_comorbid_ahrq(many_patients)
  )[["elapsed"]] 
```
This is not run to avoid unnecessary load when CRAN builds the package. I get about 5 seconds for 10 million rows of comorbidities on a moderately powerful workstation.

### Arbitrary ICD-9 mapping
The user can provide any ICD-9, ICD-10 or other code mapping to comorbidities they wish. Submissions of other peer-reviewed published mappings could be included in this package, if their license permits. Create an issue in [github](https://github.com/jackwasey/icd/issues) or email me at jack@jackwasey.com) Included in this package is a small data set called `icd9_chapters`, which lists the ICD-9-CM (and indeed ICD-9) Chapters. These can easily be expanded out and used as a mapping, so instead of a comorbidity, we see which patients have codes in each chapter of the ICD-9 defintion.
```{r arbitrary Mapping}
names(icd9_chapters)[c(1:5, 14)]
my_map <- icd:::icd9_chapters_to_map(icd9_chapters[c(2, 5, 14)])
icd9_comorbid(patients_icd9, my_map) # no positive 
```

### Reduce comorbidity mapping from possible values to defined diagnostic codes.
Suppose we want to exact match only real ICD-9 codes when looking up comorbdities for some patients. E.g. if the coder accidentally omitted a trailing zero, e.g. code `003.20` (*Localized salmonella infection, unspecified*) might have been written as `003.2` which has a heading (*Localized salmonella infections*) but is not itself billable. Use of ICD-9 codes for comorbidities generally assumes the codes are either right or wrong. How do we match only real codes, for a strict interpretation of comorbidities? It's one line or R code:

```{r realmapping}
ahrq_strict <- lapply(icd9_map_ahrq, icd_get_defined)
str(icd9_map_ahrq[1:5]) # first five of the original:
str(icd9_map_ahrq[1:5]) # and first five of the result:
```
Note the much smaller numbers of codes in each group, now we have discarded all the ones which are not defined as diagnoses.

### Which three-character ICD-9 codes have no child codes
The ICD-9-CM scheme is structured as follows:
 - Chapter
 - Sub-chapter
 - Major part (three-digit codes)
 - sub-division (first decimal place)
 - sub-sub-division (second decimal place)
 
 For most combinations of zero to nine, nothing is defined. Sometimes, nodes at one level in the hierarchy are descriptive only of their children (branch nodes), whereas some are themselves billable. For this example, let's find those numeric-only codes which have no children, and by implication are themselves directly billable codes. Here are the first ten:
```{r "find three digit billable"}
icd9cm_hierarchy$code %>% icd_get_defined -> all_real
# select the non-V and non-E codes
three_digit_real <- all_real[icd9_is_n(all_real)]
# display
three_digit_df <- data.frame(code = three_digit_real, description = icd_explain(three_digit_real, condense = FALSE))
print(three_digit_df[1:10, ], row.names = FALSE)
```

### Which ICD-9 codes have changed between versions of ICD-9?
```{r "compare ICD-9 versions"}
new_since_27 <- setdiff(icd9cm_billable[["32"]][["code"]],
                         icd9cm_billable[["27"]][["code"]]) %>% head
lost_since_27 <- setdiff(icd9cm_billable[["27"]][["code"]],
                         icd9cm_billable[["32"]][["code"]]) %>% tail
# we know this is an ICD-9-CM code, so declare this using nice magrittr motif:
lost_since_27 %<>% as.icd9cm
lost_since_27 %<>% as.icd9cm

# these are a few which were gained since v27
data.frame(code = new_since_27, desc = new_since_27 %>% icd_explain)
# these are a few which were lost since v27
data.frame(code = lost_since_27, desc = lost_since_27 %>% icd_explain)
```

## Conclusion

This package allows fluid, fast and accurate manipulation of ICD-9 and ICD-10 codes, especially when combined with [magrittr](https://cran.r-project.org/package=magrittr). Suggestions, contributions and comments are welcome via [github](https://github.com/jackwasey/icd/issues).
