Published on in Vol 6 , No 1 (2017) :January

Leveraging Electronic Health Care Record Information to Measure Pressure Ulcer Risk in Veterans With Spinal Cord Injury: A Longitudinal Study Protocol

Leveraging Electronic Health Care Record Information to Measure Pressure Ulcer Risk in Veterans With Spinal Cord Injury: A Longitudinal Study Protocol

Leveraging Electronic Health Care Record Information to Measure Pressure Ulcer Risk in Veterans With Spinal Cord Injury: A Longitudinal Study Protocol


1Center of Innovation on Disability and Rehabilitation Research, Health Services Research and Development, Department of Veterans Affairs, Tampa, FL, United States

2College of Public Health, University of South Florida, Tampa, FL, United States

3Tampa VA Research and Education Foundation, Inc, Tampa, FL, United States

4VA Boston Healthcare System, VA New England Healthcare System, Department of Veterans Affairs, West Roxbury, MA, United States

5Muma College of Business, University of South Florida, Tampa, FL, United States

6Geriatrics Research Education and Clinical Care, Tennessee Valley Healthcare System, Department of Veterans Affairs, Nashville, TN, United States

7Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN, United States

8Department of Biostatistics, Vanderbilt University Medical Center, Nashville, TN, United States

9Division of General Internal Medicine, Vanderbilt University Medical Center, Nashville, TN, United States

10Research and Development Service, Tennessee Valley Healthcare System, Department of Veterans Affairs, Nashville, TN, United States

11College of Nursing, University of South Florida, Tampa, FL, United States

Corresponding Author:

Stephen L Luther, PhD

Center of Innovation on Disability and Rehabilitation Research

Health Services Research and Development

Department of Veterans Affairs

James A Haley Veterans Hospital (151R)

8900 Grand Oak Circle

Tampa, FL, 33636-1022

United States

Phone: 1 813 558 3966

Fax:1 813 558 7616


Background: Pressure ulcers (PrUs) are a frequent, serious, and costly complication for veterans with spinal cord injury (SCI). The health care team should periodically identify PrU risk, although there is no tool in the literature that has been found to be reliable, valid, and sensitive enough to assess risk in this vulnerable population.

Objective: The immediate goal is to develop a risk assessment model that validly estimates the probability of developing a PrU. The long-term goal is to assist veterans with SCI and their providers in preventing PrUs through an automated system of risk assessment integrated into the veteran’s electronic health record (EHR).

Methods: This 5-year longitudinal, retrospective, cohort study targets 12,344 veterans with SCI who were cared for in the Veterans Health Administration (VHA) in fiscal year (FY) 2009 and had no record of a PrU in the prior 12 months. Potential risk factors identified in the literature were reviewed by an expert panel that prioritized factors and determined if these were found in structured data or unstructured form in narrative clinical notes for FY 2009-2013. These data are from the VHA enterprise Corporate Data Warehouse that is derived from the EHR structured (ie, coded in database/table) or narrative (ie, text in clinical notes) data for FY 2009-2013.

Results: This study is ongoing and final results are expected in 2017. Thus far, the expert panel reviewed the initial list of risk factors extracted from the literature; the panel recommended additions and omissions and provided insights about the format in which the documentation of the risk factors might exist in the EHR. This list was then iteratively refined through review and discussed with individual experts in the field. The cohort for the study was then identified, and all structured, unstructured, and semistructured data were extracted. Annotation schemas were developed, samples of documents were extracted, and annotations are ongoing. Operational definitions of structured data elements have been created and steps to create an analytic dataset are underway.

Conclusions: To our knowledge, this is the largest cohort employed to identify PrU risk factors in the United States. It also represents the first time natural language processing and statistical text mining will be used to expand the number of variables available for analysis. A major strength of this quantitative study is that all VHA SCI centers were included in the analysis, reducing potential for selection bias and providing increased power for complex statistical analyses. This longitudinal study will eventually result in a risk prediction tool to assess PrU risk that is reliable and valid, and that is sensitive to this vulnerable population.

JMIR Res Protoc 2017;6(1):e3




The pressure ulcer (PrU) is one of the most significant complications in veterans with spinal cord injury (SCI) in terms of morbidity, mortality, quality of life, and cost of care. The Veterans Health Administration (VHA) has the largest single network of SCI care in the United States providing a full range of care to more than 27,000 veterans. Services by the VHA are delivered through a “hub and spoke” system of care, extending from 24 regional SCI centers, offering primary and specialty care by interdisciplinary teams, to the 135 SCI patient-aligned care teams. These teams are typically comprised of an SCI coordinator (usually a social worker), a nurse, and physician, or support clinics at local VHA medical facilities [1].

The total cost of discharges for veterans with an SCI in the VHA who were discharged from a designated SCI system of care facility (ie, treating-bed section of SCI) with a primary admitting diagnosis of a PrU was over US $300 million for fiscal years (FYs) 2011-2012.  In this time period, veterans with SCI who were discharged from SCI facilities with the primary admitting diagnosis of a PrU—in the International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) 707.00-707.09 series—accounted for 64.1% (US $304M/US $474M) of the total costs for SCI discharges in the VHA (Veterans Health Administration, Managerial Cost Accounting System Discharge Pyramid Report, 2015.06.18). Stroupe and colleagues found that the cost of PrU-related hospitalizations for 150 veterans with SCI in the VHA over a 3-year period was approximately US $9 million and total health care costs were US $87,639 higher for patients with PrUs than those without (US $113,579 vs US $25,940, respectively) [2].

Current clinical practice guidelines for PrU prevention and treatment state that all patients should be assessed for risk of developing PrU using a valid, reliable, and sensitive tool [3,4]. The Braden Risk Assessment Scale [5,6], developed to predict risk in general inpatients, is presently used throughout the VHA for inpatients and outpatients. There is often a ceiling effect with most or all persons with SCI being identified at a high-risk level using the Braden Scale. Surveys conducted between 2008 and 2010 by the VHA External Peer Review Program at VHA Spinal Cord Injury and Disorder (SCI/D) Centers found that 91.3% of those measured in SCI were identified as being at high risk for PrU using the Braden Scale.

Reliability and sensitivity have not been fully established for existing tools in persons with SCI [7] despite the unique characteristics of this population (eg, lack of sensation and muscle wasting) [3]. Salzberg and colleagues devised a risk assessment scale specific to the SCI population; although used in a few SCI/D Centers [8], the psychometrics of this tool are limited [9,10].

This 5-year retrospective cohort study will leverage data available in the VHA’s electronic health record (EHR) to develop SCI-specific predictive models that can be used to better identify risk for PrUs. The new model will enable targeted prevention strategies, thereby reducing the burden of this serious complication on veterans, their caregivers, and the health care system.

Scientific Rationale

Despite widespread implementation of risk assessment tools for PrUs, the incidence of PrUs among veterans with SCI has remained stable. Furthermore, there has been an increase in PrUs in the general population. Data from the Healthcare Cost and Utilization Project (HCUP), and Agency for Healthcare Research and Quality (AHRQ), identified a national increase of almost 80% in PrUs in 2006 compared with 1993; SCI was a frequent comorbidity (29.2%) of those hospitalized with a PrU [11].

In pilot work, we found that the majority of potential PrU risk factors identified in our review were available in the EHR in structured, unstructured, or semistructured data. Structured data, the easiest to extract, are precoded (ie, value or meaning is assigned) and stored in database tables such as the ICD-9-CM codes. Recent studies have shown that it is possible to develop valid risk models based on structured data in the EHR [12-14]. Semistructured data are text formatted into tables, templates, lists, and other documents. Extraction of such data is often difficult because it is commonly embedded within the unstructured clinical narrative and the presentation format is not standardized. Unstructured data are text that is formatted in traditional sentences and paragraphs. It is commonly considered the most difficult data to extract because natural language provides multiple methods of communicating the same essential information.

Recent studies have also shown that natural language processing (NLP) techniques can be used to extract risk and decision-making information from clinical text [15-17]. Natural language processing research focuses on developing computational models for understanding natural language [18]. The specific NLP technique employed varies depending on whether the text is semistructured or unstructured. The NLP techniques are used to extract and encode specifically targeted information in text (ie, information extraction [IE]) and convert it to structured data, which can then be readily used in risk assessment.

Aims and Hypotheses

The goal of this study is to develop an improved SCI PrU risk model to better predict level of risk, guide preventive practices, and ultimately maximize the patient’s clinical outcomes. The resultant risk model, tailored to veterans with SCI, should promote adoption into clinical practice.

We will test two hypotheses to compare a model based on the commonly used Braden Scale versus (1) a risk model developed using structured data alone and (2) a model using both structured and NLP (ie, semistructured and unstructured) data. A third hypothesis will test the added value of NLP-based data over structured data. The hypotheses are as follows:

1. Hypothesis 1: The risk model using structured data alone will predict the development of PrUs better than the model based on the Braden Scale.

2. Hypothesis 2: The risk model based on combined structured and NLP data will predict the development of PrUs better than the model based on the Braden Scale.

3. Hypothesis 3: The risk model based on combined structured and NLP data will predict the development of PrUs better than the model based on structured data alone.

Conceptual Framework

In developing this protocol, we employ the Aggregation, Organization, Reduction, Transformation, Interpretation, and Synthesis (AORTIS) model of understanding clinical summarization in both computer-independent and computer-supported clinical tasks [19]. Using this framework, “clinical summarization” is any act that can be carried out by a health care team member who utilizes patient and clinical information to create a structured data summary, which in turn supports clinical tasks.

In our conceptual model, any or all of the steps to produce a concise and accurate summary could be performed by either a clinician or an automated system. The model was designed to be sequential, with the output from one step providing input into the next, and task dependent, with the content of each step varying based on the clinical task that the summary was designed to support (see Figure 1). In this study, we include many but not all of the steps of the model to employ AORTIS.

Figure 1. The model used to understand clinical summarization in both computer-independent and computer-supported clinical tasks. Info: information.
View this figure

In AORTIS, data are organized around the specific clinical situation, development of a PrU. This expedites IE and clinical decision making but requires the application of a significant clinical knowledge base. The process of identifying and extracting the necessary data from structured data or text represents activities related to aggregation and organization. The application of NLP programs represents the reduction and transformation of complex text data, which was not readily computable, into structured data elements that could be included in prediction models. The development of prediction models represents the interpretation and synthesis of these data, which could be presented to clinicians for interpretation.


This is a 5-year longitudinal, retrospective, inception, cohort study targeting veterans with SCI. This research was approved by the Department of Research and Development, James A Haley Veterans’ Hospital, and the VHA Central Institutional Review Board. As described below, the study is ongoing. Each section describes the current status of the study activities as complete, ongoing, or planned.

Expert Panel—Complete

An expert panel meeting was held in March 2014 to discuss PrU risk factors. The goal of the panel was to identify factors that were likely present in the EHR and could be used to develop improved risk models for PrU development. Participants included two physicians specializing in care for veterans with SCI, three nurse researchers, a wound care specialist, an SCI outcomes registry coordinator, a physical therapist, a dietitian, and a social worker. All members of the expert panel had multiple years of experience in managing persons with PrUs and SCI. Members of the study team also attended the expert panel meeting. Results of the expert panel were iteratively reviewed and refined by two additional physicians and a nurse specializing in research and treatment of PrUs in veterans with SCI.

Variables Selection and Definition—Ongoing

The dependent variable for this study will be documentation of the occurrence of a PrU in the EHR within 1 year (ie, 1-year incidence risk of PrU). The definition of the presence of a PrU used for case identification is based on ICD-9-CM codes (707.0-707.9) and the application of the classification algorithm based on text analysis or a combination of these two sources, depending on what is found to be more accurate. The potential independent variables for this study include the risk factors that were identified through a literature review and validated by expert panel discussion. The final list of potential risk factors, with input from the expert panel, will be extracted from multiple tables in the EHR, or from semistructured and unstructured text notes, for use in analysis. Three sets of candidate predictors based on data sources will be generated to develop each risk model: (1) structured data extracted directly from the EHR, (2) Braden scores, and (3) a combination of existing structured data and data extracted from text through the NLP processes.

The investigators conducted a comprehensive literature review generating a list of approximately 50 potential independent variables or risk factors (eg, demographics, diseases status, comorbidities, health behaviors, psychosocial factors, and home care). This list was reviewed by the expert panel for logical consistency, completeness, clinical relevance, and whether they were modifiable or not. The panel identified high-priority risk factors and discussed practical considerations including where the risk factor might be located in the EHR, how reliable these data might be, and alternative ways to document risk factors. Subsequently, persistent moisture on the skin, the type of living situation of the patient, evidence of malnutrition, and documentation that the patient took steps to redistribute their weight during the day were targeted as risk factors available in unstructured data. Structured data contained elements of the Braden Scale, lab values, and other variables (see Table 1). Several coding systems are used to classify structured data variables, including the ICD-9-CM, Logical Identifiers Names and Codes (LOINC), and the Healthcare Common Procedural Coding System (HCPCS). International Classification of Diseases, Tenth Revision, Clinical Modification (ICD-10-CM) codes were not implemented in the VHA until after the end of this study and are not used.

Sample and Sampling—Complete

Using VHA data in its Corporate Data Warehouse obtained through its Veterans Affairs Informatics and Computing Infrastructure (VINCI), we identified an inception cohort (N=12,344) of veterans with SCI who received care in the VHA in FY 2009, but had no PrU in the 12 months prior to 2009 based on recorded ICD-9-CM codes. This group was considered free of PrUs for the purpose of the study. All structured hospital discharges (N=4090), outpatient encounters (N=5,202,804), and text data (N=9,888,245) for the cohort for FY 2009-2013 (October 1, 2008-September 30, 2013) were obtained through the VINCI. Veterans from this cohort were classified as having a PrU (ie, PrU+) or not having a PrU (ie, PrU-) during the study period based on both structured and text data. A total of 3860 PrUs were identified based on structured data and 829 based on text data alone, resulting in a total incidence of 4689 (4689/12,344, 37.99%) across the study period.

Table 1. Risk factors categorized by structured, semistructured, and unstructured data sources.
Data sourceRisk factorDescription or definition
StructuredAgeAge in years



Education levelHighest level of education

VHAa eligibilityVHA category, service connection

Body weightBody mass index, most recent weight, or change in weight

Patient statusInpatient/outpatient when pressure ulcer develops

Pattern of preventive careConsistency of annual well-patient exam

Cognitive dysfunctionICD-9-CMb codes (ie, traumatic brain injury, dementia, and Alzheimer’s disease)

Comorbidities—psychologicalICD-9-CM codes (ie, depression, anxiety, and posttraumatic stress disorder)

Comorbidities—physicalICD-9-CM codes (ie, diabetes mellitus, pulmonary disease, cardiac disease, renal disease, peripheral vascular disease, anemia, deep vein thrombosis, cancer, infection, lower extremity fractures, spasticity, autonomic dysreflexia, hypotension, and heterotopic ossification)

Laboratory analysisLOINCc codes for lab values (ie, C-reactive protein, white blood cells, erythrocyte sedimentation rate, renal function, hemoglobin, hematocrit, albumin, and prealbumin)

MedicationsVHA formulary to identify chemotherapeutics, steroids, and benzodiazepines

SurgeryICD-9-CM codes of surgical procedures within previous 6 months of pressure ulcer development (administrative data)

VHA-issued equipmentHCPCSd codes for wheelchair cushion, bed support surface, and other equipment

TelehealthDistant observation of skin integrity
SemistructuredMotor/sensory assessmentAmerican Spinal Cord Injury Association (ASIA) classification of SCIe (eg, ASIA A= Complete injury of spinal cord)

Level of injuryCervical (C1-C8), thoracic (T1-T12), lumbar (L1-L5), and sacral (S1-S5)

Alcohol useAlcohol Use Disorders Identification Test—Consumption (AUDIT-C)

Tobacco useCurrent/former/nonsmoker

Caregiver resourcesReceiving aid and attendant, VHA-supported/reimbursed bowel/bladder care

Duration of SCIYears since injury/onset

SensationfAbility to feel and relieve discomfort

MoisturefExposure to moisture on trunk (ie, bowel/bladder leakage/accidents due to management or lack of containment, diarrhea, and diaphoresis)

MobilityfAbility to relieve pressure on high-risk areas (eg, trunk and heels)

ActivityfLevel of activity (ie, ability to get out of bed and ambulation)

NutritionfFood intake malnutrition, undernutrition, and moderate/severe compromise

Friction and shearfAbility to minimize resistance between two parallel surfaces (ie, bed linens and skin)

Living situationLiving arrangements (eg, alone, immediate family, extended family, roommate, and group setting)
UnstructuredNoncompliance with pressure redistributionPatient refusal/decline to turn in bed

PainNumeric pain rating scale (1-10) when pressure ulcer develops

Pressure ulcer historyPrevious pressure ulcer

aVHA: Veterans Health Administration.

bICD-9-CM: International Classification of Diseases, Ninth Revision, Clinical Modification.

cLOINC: Logical Identifiers Names and Codes.

dHCPCS: Healthcare Common Procedural Coding System.

eSCI: spinal cord injury.

fElement of the Braden Scale.

Data Extraction and Synthesis (Structured Data)—Ongoing

To determine whether the targeted structured variables were present in a patient’s record, we identified lists of codes (ie, ICD-9-CM codes) to operationally define the variables. When available, definitions employed by the VHA Support Service Center (VSSC), a centralized reporting portal, were used. The VSSC provides data summaries to VHA administrators and policy makers regarding business operations, clinical care, quality performance, and resource management. Otherwise, operational definitions were obtained from the AHRQ’s HCUP Clinical Classifications Software [20] or the Centers for Medicare and Medicaid Services definitions. Extraction routines were used to identify each occurrence of the variable in the patient’s record. These data were then transformed into single elements in the analytic data table based on clinically relevant temporal considerations. For example, some comorbidities such as diabetes were considered chronic conditions, while others such as fractures were framed in a 1-year period.

Data Extraction and Synthesis (Unstructured and Semistructured Data)

We will employ NLP techniques to extract targeted information from the documents in the EHR. This will be accomplished using the following steps.

Step 1: Create Reference Standard—Complete

Large samples of documents (approximately 2000) will be extracted from the larger corpus of documents. A human-annotated (ie, chart review) corpus will be developed, which will be used as a reference standard to train and evaluate NLP algorithms. An annotation schema was developed, which defines the classes of concepts based on the variables reviewed by the expert panel. The concepts will be annotated in each document in the corpus by two clinicians and adjudicated by a third clinical expert. Interrater reliability will be evaluated by calculating interannotator agreement. Agreement between the two annotators will be evaluated by calculating the F measure, proposed by Hripcsak and Rothschild [21], against the final reference that will be adjudicated by our clinical expert.

Step 2: Extract Information From Unstructured and Semistructured Text Data—Ongoing

We will use the annotated reference set of documents to develop and evaluate distinct methods to extract information from unstructured and semistructured text.

Step 2a

The General Architecture for Text Engineering (GATE) software will be used to create an NLP pipeline to extract the information from the unstructured text data [22]. The results of the NLP pipeline will be iteratively compared to the annotated terms from the reference standard on 70% of the documents. The remaining 30% of the documents will be used to validate the best model. This will enable the pipeline to be iteratively refined to more accurately identify the potential predicators located in the narrative documents.

Step 2b

We will develop an NLP system for semistructured text data. The composition of semistructured text presents unique challenges to NLP, making information extraction especially challenging [23,24]. Semistructured text consists of short, simple phrases that are not grammatically correct.

A traditional grammatical NLP system may fail to negate the presence of moisture for the patient. While this “structure” aids readability, it hinders traditional grammatical NLP systems. However, this “structure” makes rule-based systems using regular expressions a viable alternative for extracting targeted information. Thus, we will employ a custom-developed, rule-based system using named entities- and structural-based regular expressions, which will also be developed and evaluated using an annotated reference set of documents.

Data Analysis—Planned

Three separate predictive models of PrU occurrence based on each set of predictors in each hypothesis will be developed using R statistical program version 3.3.2 (The R Foundation). To prepare, the distribution of each continuous predictor will be examined and variable transformation will be carried out as appropriate to approximate normality (eg, log transformation of skewed data). However, to avoid information loss, no continuous variable will be categorized in an initial prognostic model. Using the generalized additive model (GAM) approach, a bivariate logistic regression will be performed to test potential nonlinearity between each continuous candidate predictor and the binary PrU outcome [25]. This will provide further guidance for appropriate transformation needed for the predictor.

The analytic data sample (N=12,344) will be randomly divided into two groups in the ratio of 70% (n=8641) for derivation and 30% (n=3703) for validation. The randomization will be stratified by site, age group, and gender to increase similarity in the distribution of both validation and derivation random samples.

Model Development

An automated model selection procedure will be performed with the R package glmulti [26] to select variables and interaction terms in a logistic regression model of PrU status in the training sample. The automated model selection procedure will return a set of n best models, ranked by their information criteria values as well as estimating importance scores for the predictors. Thus, it offers us the flexibility to select from the set a more parsimonious or more intuitively reasonable final model that is context based. Another consideration is for the final prognostic model to include variables that could be easily assessed in practice and with minimal burden on the respondent SCI patients. The fully specified prognostic model will eventually be part of an automated decision support system.

The clinical utility of the risk scores will be evaluated for different risk thresholds and the sensitivity, specificity, positive predictive value, and negative predictive value will be included as performance measures in the automated output. In addition, a separate listing of risk factors will be provided for each risk model and ranked by their rounded Z-scores (or standardized regression coefficients). A higher Z-score implies a more “important” contribution of that variable in the model. These lists will assist the clinicians in prioritizing what variables to assess. Each product will be accompanied with appropriate complete operational definitions and scoring instructions.

Data analysis will be focused on testing two hypotheses that compare the ability of the current standard risk assessment measure—Braden Scale—with models based on structure data alone and in combination with data extracted from text. Comparisons will be made between a pair of risk models to test each hypothesis using exactly the same veterans from the validating sample. Our priority will be to develop models with low false positive rate (ie, the rate at which negative instances are incorrectly classified as having PrU). This consideration will guide how any two competing risk models (ie, classifiers) will be compared using the receiver operating characteristic (ROC) curve analysis. The logistic model will output proper probability scores from which a full ROC curve will be generated for each classifier with overlay in one plot. Generally, a higher calculated total area under the curve (AUC) implies a better prognostic performance. However, when comparing two classifiers, it is possible for the classifier with a higher total AUC to show poorer local performance in our priority region. Therefore, we will assess classifier performance in the overall AUC as well as partial AUC (pAUC) in our priority local region of low false positive rate. The classifier that has a higher sensitivity (ie, true positive rate) in the priority region has superior local performance. That is, an appropriate pAUC will also be computed in addition to the standard full AUC.

Both discrimination and calibration will be investigated as validation measures for each prognostic model using the validation sample. The better discriminating models have higher AUC and pAUC values based on their CIs (ie, better able to distinguish between patients with SCI who have and do not have PrU). To assess calibration, the predictions of each model will be compared with the veterans’ actual outcomes in the validation sample by performing the Hosmer-Lemeshow test using the hoslem.test function in the Resource Selection library of R. To ensure that the overall type I error rate of α is maintained, first, we will repeatedly sample from each model and calculate the Hosmer-Lemeshow P value 1000 times, and calculate the proportion of P values less than .05 (ie, type I error rate should be no greater than 5%).

A graphical approach will be used to compliment the Hosmer-Lemeshow test since it has been known to be sensitive to sample size. For the graphical method, first, the sample will be divided into 10 risk groups of equal size with each group having similar model predicted probabilities. Next, the observed proportions of PrUs will be plotted against predicted probabilities for these groups with one smooth curve fit (eg, lowess) as well as a linear fit to the data points. We anticipate that the observed and predicted occurrence of PrUs will be similar across the risk groups. A perfect calibration (ie, perfect agreement between the predicted and observed outcomes) will produce a calibration line with an intercept and slope of 0 and 1, respectively [27-29]. The smooth curve will be added to reveal differential calibration among risk deciles, if present.

We expect the validation study to span across different domains (eg, inpatient vs outpatient, site, age group, and gender) as this provided the strongest evidence of generalizability of the prediction rule to new patients [30]. Since stratified randomization ensured a similar case mix within each of our samples—derivation and validation—and both samples were representative of our population of interest, we believe that the results of our validation tests will serve as strong evidence of generalizability.

The study is ongoing with results expected in 2017. The expert panel met and reviewed the initial list of risk factors based on the literature review (see Table 1). They made recommendations for additions and deletions and provided insight into where, and in what format, the documentation of the risk factors might exist in the EHR. This list was then iteratively refined through review and discussion with individual experts in the field. The cohort for the study has been identified and all structured, unstructured, and semistructured data have been exported into a relational database for analysis. A description of the cohort is provided in Table 2.

The cohort is almost exclusively male (11,796/12,344, 95.56%), with 68.96% (8513/12,344) being white and 19.64% (2424/12,344) being black or African American. The mean age of the cohort was 58 years (SD 14) with more than half (6782/12,344, 54.94%) being between 50 and 69 years of age. Approximately half (6873/12,344, 55.68%) reported being married.

Annotation schemas have been developed, samples of documents have been extracted, and annotation and adjudication are ongoing. Table 3 outlines the variable targeted through the annotation process.

Table 2. Description of study cohort (N=12,344).
Characteristicn (%)

Female548 (4.44)

Male11,796 (95.56)

Asian74 (0.60)

Black or African American2424 (19.64)

Native Hawaiian or other Pacific Islander1 (0.01)

White8513 (68.96)

Other28 (0.23)

Unknown by patient/missing1214 (9.83)

Hispanic or Latino939 (7.61)

Non-Hispanic or non-Latino10,723 (86.87)

Unknown by patient/missing82 (0.66)
Age group, years

<2124 (0.19)

21-30491 (3.98)

31-40824 (6.68)

41-501854 (15.02)

51-603753 (30.40)

61-703120 (25.28)

71-801540 (12.48)

81-90701 (5.68)

>9036 (0.29)

Missing1 (0.01)
Marital status

Single/never married2354 (19.07)

Married6125 (49.62)

Divorced3097 (25.09)

Widow/widowed560 (4.54)

Unknown/missing187 (1.51)
Table 3. Pressure ulcer (PrU) annotation schemas.
TaskVariableAttributesDescriptionExample text spans
Annotation task 1Pressure ulcerPressure ulcerText indicating presence of a PrUPressure ulcer, skin sore, decubitus ulcer

StageStage of the PrUStage 1, stage 2, unable to stage, stage 3-4

LateralityRight, left, bilateral, midline, not applicable, unspecifiedRight, left, lower extremity

LocationAnatomic location of PrUCoccyx, heel, trochanter, sacrum, ankle, ischial

OrientationMedial, lateral, proximal, distal, dorsum, plantar, anterior, superior, posterior, inferior, unspecifiedBilateral

TemporalityDate: date of examination
Duration: span of time since PrU discovered
Sept 10, 2009, approximately one year

AssertionModifiers: historical, recurrent, negated, hypothetical, not PrU, unspecifiedHealed pressure ulcer, history of pressure ulcers, no pressure ulcers, if pressure ulcers develop

NoncomplianceNoneDocumentation of noncompliance with pressure releasePatient refuses to turn because of pain

In labelNonePrU mentions within section labels in documents (ie, are not indications of a PrU)Pressure ulcer protocol
Pressure ulcer education
Annotation task 2Living situation
LS cueKey phrase indicating presence of LSLiving at, living with, address, living arrangements

LS mainActual value assigned to LS (ie, alone, nuclear family, extended family, roommate, group, homeless)Mother’s residence, girlfriend, half way house, caregiver, spouse

MN cueKey phrase indicating presence of MNNutrition status:

MN mainActual value assigned to MNModerately compromised, severely compromised

MO cueKey phrase indicating statement about MOMoisture:
Constantly moist:

MO mainText indicating presence of MO (ie, fecal, urinary, sweat)Multiple loose stools, copious amounts foul purulent drainage, night sweats, perspiration, urine

AssertionModifiers: asserted, historical, hypothetical, negation, not patient, uncertainWas, prior medical history, nutrition risk, manage, []a, history of
Annotation task 3Template componentsHeaderTitle line naming the templateBraden Scale, ASIAb score

Item labelLabel for individual assessment itemsMoisture, turgor, color, temp

Item scoreValue (text or number) assigned to itemMoist, red, 4, 0

Total labelText identifying the summed scoreTotal score:

Total scoreFinal value for the assessment30, high risk

Whole templateOffsets from block of text identifying template boundaries132-165

aThese empty brackets would be generated by the system and placed in a progress note when a check box was left blank in a template in the electronic medical record system.

bASIA: American Spinal Cord Injury Association.

The annotation process was split into three tasks to allow for sampling of different types of documents, depending on the targeted variables. This also reduces the cognitive burden for the annotators in each of the tasks. In the first task, the emphasis is on identifying attributes of the pressure ulcers and on any documentation of patient noncompliance with steps to reduce pressure ulcers, such as pressure release. In the second task, the primary targets were the patients’ living situation, whether there was any evidence of malnutrition, and whether there was evidence of ongoing moisture on the skin. Annotation for each of the targeted variables included both a cue (phrase indicating presence) for the variable and the actual value assigned to the variable. The third annotation task was slightly different in that it emphasized labeling of the structural component of semistructured text (stored in templates). This information will allow for targeted extraction of the information contained in the template. Operational definitions of ICD-9-CM codes are being created. Once all of the various data are defined and extracted, they will be combined into an analytic dataset for development of risk models.

When completed, the results of this study will provide clinicians in the VHA system with a PrU risk assessment model specific to veterans with SCI. As part of the study, the predictive ability of the new model will be compared directly with that of the Braden Scale, the current risk assessment. Also, individual items on the Braden Scale will be tested for inclusion in the model. This will provide clinicians with information about the clinical utility of the new model. Initially, the new model will be distributed in the form of a simple stand-alone desktop computer program; however, our long-term goal is to deploy the risk assessment as an automated, integrated part of the VHA EHR. Clinicians could use the improved risk model to (1) maximize the impact of expensive resources for prevention of PrU (eg, specialty mattress and paid caregiver) by identifying those veterans at highest risk, (2) justify allocation of staffing resources (eg, home health care or telehealth), and (3) institute policies (eg, frequency of turning the veteran in bed). Further, we believe this study represents a model of how to leverage information from the EHR for risk assessment that could be applied to other clinical problems.


This work was supported by the Health Services Research and Development Service of the Veterans Health Administration. The contents do not represent the views of the Department of Veterans Affairs or the United States Government. This research is the result of work supported with resources and the use of facilities at the James A Haley Veterans Hospital and our Health Services Research and Development Center of Innovation on Disability and Rehabilitation Research. We would like to thank our project staff: Bridget Hahn, William Lapcevic, Kevin White, Keith White, and Lisa Goff.

Authors' Contributions

SL made substantial contributions to the conception and design of the work and interpretation of data.  He is accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved and he will have final approval of the version to be published. SST made substantial contributions to the conception and design of the work and drafted and revised the paper critically for important intellectual content. SS and GPC made substantial contributions to the conception and design of the work and revised it critically for important intellectual content. DKF, JM, PT, LM, and GTG made substantial contributions to the conception and design of the work and the acquisition, analysis, and interpretation of data for the work. MEM made substantial contributions to the conception and design of the work and interpretation of data.

Conflicts of Interest

None declared.

  1. Spinal Cord Injury: Veterans Affairs Quality Enhancement Research Initiative (QUERI) Fact Sheet. Washington, DC: US Department of Veterans Affairs; 2014 Jul.   URL: [accessed 2016-12-13] [WebCite Cache]
  2. Stroupe KT, Manheim L, Evans CT, Guihan M, Ho C, Li K, et al. Cost of treating pressure ulcers for veterans with spinal cord injury. Top Spinal Cord Inj Rehabil 2011;16(4):62-73. [CrossRef]
  3. Consortium for Spinal Cord Medicine. Pressure Ulcer Prevention and Treatment Following Spinal Cord Injury: A Clinical Practice Guideline for Health-Care Professionals. 2nd edition. Washington, DC: Paralyzed Veterans of America; 2014.
  4. Pressure Ulcer Advisory Panel, European Pressure Ulcer Advisory Panel and Pan Pacific Pressure Injury Alliance. In: Haesler E, editor. Prevention and Treatment of Pressure Ulcers: Clinical Practice Guideline. Perth, Australia: Cambridge Media; 2014.
  5. Wilchesky M, Lungu O. Predictive and concurrent validity of the Braden scale in long-term care: A meta-analysis. Wound Repair Regen 2015;23(1):44-56. [CrossRef] [Medline]
  6. Gadd MM, Morris SM. Use of the Braden Scale for pressure ulcer risk assessment in a community hospital setting: The role of total score and individual subscale scores in triggering preventive interventions. J Wound Ostomy Continence Nurs 2014;41(6):535-538. [CrossRef] [Medline]
  7. Mortenson WB, Miller WC. A review of scales for assessing the risk of developing a pressure ulcer in individuals with SCI. Spinal Cord 2008 Mar;46(3):168-175 [FREE Full text] [CrossRef] [Medline]
  8. Scovil CY, Flett HM, McMillan LT, Delparte JJ, Leber DJ, Brown J, Spinal Cord Injury Knowledge Mobilization Network (SCI KMN). The application of implementation science for pressure ulcer prevention best practices in an inpatient spinal cord injury rehabilitation program. J Spinal Cord Med 2014 Sep;37(5):589-597 [FREE Full text] [CrossRef] [Medline]
  9. Salzberg CA, Byrne DW, Cayten CG, van Niewerburgh P, Murphy JG, Viehbeck M. A new pressure ulcer risk assessment scale for individuals with spinal cord injury. Am J Phys Med Rehabil 1996;75(2):96-104. [Medline]
  10. Salzberg CA, Byrne DW, Cayten CG, Kabir R, van Niewerburgh P, Viehbeck M, et al. Predicting and preventing pressure ulcers in adults with paralysis. Adv Wound Care 1998 Sep;11(5):237-246. [Medline]
  11. Russo CA, Steiner C, Spector W. Hospitalizations related to pressure ulcers among adults 18 years and older, 2006. Healthcare Cost and Utilization Project (HCUP). Rockville, MD: Agency for Healthcare Research and Quality; 2008 Dec.   URL: [accessed 2016-12-15] [WebCite Cache]
  12. Roth CP, Lim Y, Pevnick JM, Asch SM, McGlynn EA. The challenge of measuring quality of care from the electronic health record. Am J Med Qual 2009;24(5):385-394. [CrossRef] [Medline]
  13. Amarasingham R, Moore BJ, Tabak YP, Drazner MH, Clark CA, Zhang S, et al. An automated model to identify heart failure patients at risk for 30-day readmission or death using electronic medical record data. Med Care 2010 Nov;48(11):981-988. [CrossRef] [Medline]
  14. Watson AJ, O'Rourke J, Jethwani K, Cami A, Stern TA, Kvedar JC, et al. Linking electronic health record-extracted psychosocial data in real-time to risk of readmission for heart failure. Psychosomatics 2011;52(4):319-327 [FREE Full text] [CrossRef] [Medline]
  15. Pakhomov SV, Hanson PL, Bjornsen SS, Smith SA. Automatic classification of foot examination findings using clinical notes and machine learning. J Am Med Inform Assoc 2008;15(2):198-202 [FREE Full text] [CrossRef] [Medline]
  16. Brown SH, Elkin PL, Rosenbloom ST, Fielstein E, Speroff T. eQuality for all: Extending automated quality measurement of free text clinical narratives. AMIA Annu Symp Proc 2008 Nov 06:71-75 [FREE Full text] [Medline]
  17. Friedlin J, Grannis S, Overhage JM. Using natural language processing to improve accuracy of automated notifiable disease reporting. AMIA Annu Symp Proc 2008 Nov 06:207-211 [FREE Full text] [Medline]
  18. Carbonell JG, Hayes PJ. Natural language understanding. In: Shapiro SC, editor. Encyclopedia of Artificial Intelligence. New York, NY: Wiley & Sons; 1987:660-677.
  19. Feblowitz JC, Wright A, Singh H, Samal L, Sittig DF. Summarization of clinical information: A conceptual model. J Biomed Inform 2011 Aug;44(4):688-699 [FREE Full text] [CrossRef] [Medline]
  20. Healthcare Cost and Utilization Project (H-CUP). Tools & software.   URL: [accessed 2016-05-02] [WebCite Cache]
  21. Hripcsak G, Rothschild AS. Agreement, the f-measure, and reliability in information retrieval. J Am Med Inform Assoc 2005;12(3):296-298 [FREE Full text] [CrossRef] [Medline]
  22. Cunningham H. GATE, a general architecture for text engineering. Comput Hum 2002 May;36(2):223-254. [CrossRef]
  23. Meystre SM, Savova GK, Kipper-Schuler KC, Hurdle JF. Extracting information from textual documents in the electronic health record: A review of recent research. Yearb Med Inform 2008:128-144. [Medline]
  24. Meystre SM, Haug PJ. Comparing natural language processing tools to extract medical problems from narrative text. AMIA Annu Symp Proc 2005:525-529 [FREE Full text] [Medline]
  25. Royston P, Moons KG, Altman DG, Vergouwe Y. Prognosis and prognostic research: Developing a prognostic model. BMJ 2009 Mar 31;338:b604. [Medline]
  26. Calcagno V, de Mazancourt C. glmulti: An R package for easy automated model selection with (generalized) linear models. J Stat Softw 2010 May;34(12):1-29 [FREE Full text] [CrossRef]
  27. Altman DG, Vergouwe Y, Royston P, Moons KG. Prognosis and prognostic research: Validating a prognostic model. BMJ 2009 May 28;338:b605. [Medline]
  28. van den Bosch JE, Kalkman CJ, Vergouwe Y, Van Klei WA, Bonsel GJ, Grobbee DE, et al. Assessing the applicability of scoring systems for predicting postoperative nausea and vomiting. Anaesthesia 2005 Apr;60(4):323-331 [FREE Full text] [CrossRef] [Medline]
  29. Arkes HR, Dawson NV, Speroff T, Harrell FE, Alzola C, Phillips R, et al. The covariance decomposition of the probability score and its use in evaluating prognostic estimates. SUPPORT Investigators. Med Decis Making 1995;15(2):120-131. [Medline]
  30. Toll DB, Janssen KJ, Vergouwe Y, Moons KG. Validation, updating and impact of clinical prediction rules: A review. J Clin Epidemiol 2008 Nov;61(11):1085-1094. [CrossRef] [Medline]

AHRQ: Agency for Healthcare Research and Quality
AORTIS: Aggregation, Organization, Reduction, Transformation, Interpretation, and Synthesis
ASIA: American Spinal Cord Injury Association
AUC: area under the curve
AUDIT-C: Alcohol Use Disorders Identification Test—Consumption
EHR: electronic health record
FY: fiscal year
GAM: generalized additive model
GATE: General Architecture for Text Engineering
HCPCS: Healthcare Common Procedural Coding System
HCUP: Healthcare Cost and Utilization Project
hx: history
ICD-9-CM: International Classification of Diseases, Ninth Revision, Clinical Modification
ICD-10-CM: International Classification of Diseases, Tenth Revision, Clinical Modification
IE: information extraction
info: information
LOINC: Logical Identifiers Names and Codes
LS: living situation
MN: malnutrition
MO: moisture
NLP: natural language processing
pAUC: partial area under the curve
PrU: pressure ulcer
PrU+: presence of a pressure ulcer
PrU-: absence of a pressure ulcer
ROC: receiver operating characteristic
SCI: spinal cord injury
SCI/D: Spinal Cord Injury and Disorder
VHA: Veterans Health Administration
VINCI: Veterans Affairs Informatics and Computing Infrastructure
VSSC: VHA Support Service Center

Edited by CL Parra-Calderón; submitted 03.06.16; peer-reviewed by G Seçkin, P Elkin; comments to author 13.07.16; revised version received 09.09.16; accepted 30.10.16; published 19.01.17


©Stephen L Luther, Susan S Thomason, Sunil Sabharwal, Dezon K Finch, James McCart, Peter Toyinbo, Lina Bouayad, Michael E Matheny, Glenn T Gobbel, Gail Powell-Cope. Originally published in JMIR Research Protocols (, 19.01.2017.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.