The Comparative Validity of Interactive Multimedia Questionnaires to Paper-Administered Questionnaires for Beverage Intake and Physical Activity: Pilot Study

Introduction

Assessment Methods

Multiple unannounced 24-hour recalls, food records, and physical activity recalls have historically been recognized as gold standard approaches to assessing dietary and physical activity behaviors [1-3]. However, these assessment methods often require trained staff to administer and analyze, and are labor-intensive for both researchers and participants [1,2]. For these reasons, other valid methods of diet and physical activity assessments have been developed, such as food-frequency questionnaires as well as brief assessments of diet and physical activity. Recently, computerized diet and physical activity assessments have emerged as a way to decrease literacy barriers for participants, as well as decrease the research burden of processing paper-based surveys in large studies [4-8]. However, attention to the reliability, validity, and usability of computerized approaches to assessing diet and physical activity behaviors are imperative.

Due in part to the increased use and accessibility of computers in multiple settings (eg, homes, libraries, churches, recreational community centers, grocery stores, and schools) [7,9], the use of Web and computer-based assessments in large research trials have increased over the past 10 years [4,6,10]. The National Institutes of Health has recognized the need for novel/innovative assessment methods using technological advances in physical activity and dietary assessment (eg, PAR-12-198). There is no consensus to whether a paper-based assessment is superior to a computerized one [11]; however, computer-based tools can provide an alternative means to collect and analyze data [12] and may be appealing to practitioners and researchers because of their proposed benefits. Computer-administered assessments may overcome difficulties sometimes associated with paper-based surveys as they allow for interactivity-two-way communication between computer and participant through photographs, videos, and displayed text with or without audio [7]. Other advantages of computerized questionnaire administration include-complete responses (ie, prompting individuals to answer all questions), written and narrated text, visual cues of portion sizes, immediate and rapid data entry and scoring, decreased scoring errors, increased attentiveness from participants, instantaneous feedback, and a greater ability to access understudied populations [6,8,10,13,14]. Additionally, multi-part questions of computerized assessments can be programmed to reduce administration time by providing only relevant data and information for the participant [7]. In low health literacy populations, computerized questionnaires may be advantageous since text can be narrated and visual aids can be used, which may reduce response errors and the necessity of advanced reading skills [7]. Another potential advantage of computer-based assessments is that response-bias and intimidation may be reduced with computer-administered surveys, although additional research addressing this possibility is needed [5,7,15]. However, when using identical computerized versions of paper assessments comparability cannot be assumed because interface characteristics like font size, line length, scrolling ability, and amount of information visible on the screen can all influence user performance [16,17].

Two Paper-Based Questionnaires

Prior research has demonstrated the reliability and validity of two self-administered paper-based questionnaires. One assesses habitual beverage intake (BEVQ-15) [18], and the other measures usual physical activity level, Stanford Leisure-Time Activity Categorical Item 2.2 (L-Cat) [19]. There are several computerized nutrition education delivery [7,20,21] and dietary assessment tools [8,13,22-28] and a few Web-based physical activity questionnaires [12,29,30] currently available; however, to the best of our knowledge, no computer-based beverage intake questionnaire exists. The recently developed Automated Self-Administered 24-hour Recall [31] is computer-based and does contain questions about beverage intake; however, results on its validity and usability have yet to be published [32]. The purpose of this pilot investigation was to evaluate the comparative validity and reliability of newly developed interactive multimedia (IMM) versions compared to validated paper-administered (PP) versions of the BEVQ-15 [18] and L-Cat [19]. Individuals with lower educational attainment and/or health literacy levels may be at increased risk for health complications associated with poor dietary and health behaviors such as obesity, diabetes, hypertension, and coronary heart disease [33]. Therefore, a secondary purpose of this investigation was to evaluate the validity and reliability of the major BEVQ-15 categories, for example, total water, sugar-sweetened beverage (SSB), and total beverage grams and kcal, and L-Cat category across two education levels, in order to determine the suitability of the IMM versions for individuals from varying educational backgrounds.

Methods

Recruitment

Adults 21 years or older (n=60) were recruited from several community settings (a local university community, free medical clinic, area Community Services building, and church congregation) between January and August 2012 in southwest Virginia. The Virginia Tech Institutional Review Board approved the study protocol and participants provided written informed consent prior to enrollment.

Protocol

Participation entailed three laboratory sessions with three to seven days between each session. Sessions were completed in one of two randomly assigned visit sequences that differed in questionnaire administration format (ie, taking the paper or computerized instruments initially). Randomization was done to avoid a potential effect of session order on study outcomes. In addition to providing demographic information, each participant completed a total of two identical self-administered IMM BEVQ-15 [18] and L-Cat [19] questionnaires (denoted IMM-1 for the first administration and IMM-2 for the second administration), one PP BEVQ-15 and L-Cat (ie, one set being BEVQ-15 and L-Cat at each of the three lab sessions), the Newest Vital Sign tool to assess health literacy [34], and an open-ended feedback survey on the IMM questionnaire to address usability; a total of five questionnaires were completed by each participant. Responses from the feedback survey were either "yes" or "no", or rated on an ordered-response scale (1=easy, 5=hard) with open-ended areas for comments following each question. Investigators supervised the assessments and provided limited instructions, but were available to answer questions during the survey if needed. Participants were compensated in the form of a $25 gift card upon completion of all three study visits.

Measurements

Participants provided information on demographic characteristics (ie, age, race/ethnicity, income level, and highest education level attained), and this was used to categorize participants into one of two education categories: (1) less than high school/high school, and (2) some college/college degree. Prior research suggests that the level of education reached can be a strong socioeconomic determinant of beverage intake [33,35]. Descriptive measurements were conducted by a graduate research assistant who is a registered dietitian (SKR) and a trained research assistant (ACP) and included the following-height measured in meters without shoes using a wall mounted stadiometer (Seca 216, Hamburg, Germany); body weight measured in light clothing without shoes to the nearest 0.2 kg using a digital scale (Scale-Tronix, Wheaton, IL); and body mass index (BMI), calculated as kg/m². The Newest Vital Sign (NVS) is a valid and sensitive tool that was used to assess health literacy and includes six questions based upon information contained in a nutrition facts label for a pint of ice cream. The scores range from 0-6 (0=limited health literacy, 6=adequate health literacy) [34].

The BEVQ-15 and L-Cat

The BEVQ-15 [18] is a brief, valid, and reliable quantitative food frequency questionnaire providing an estimate of habitual beverage intake across 15 beverage categories, which evaluates total beverage and SSB intake (ie, grams and kcal) over the past 30 days. Details regarding the development and evaluation of the BEVQ-15 have been previously published [18,36]. The PP BEVQ took 2 minutes-15 seconds to complete during its initial testing [18]. Self-reported physical activity was assessed using the brief L-Cat [19]. This tool was developed from the Stanford Brief Activity Survey [37-39] and consists of six descriptive categories (eg, 3="About three times a week, I did moderate activities such as brisk walking, swimming, or riding a bike for about 15-20 minutes each time. Or about once a week, I did moderately difficult chores such as raking or mowing the lawn for about 45-60 minutes. Or about once a week, I played sports such as softball, basketball, or soccer for about 45-60 minutes.") ranging from inactive (1=I did not do much physical activity) to very active (6=Almost daily, that is five or more times a week, I did vigorous activities). In a randomized trial involving 267 obese women, the L-Cat was found to be valid and reliable [19]. The paper versions of these tools were read and completed by the study participants independently with an investigator available for questions throughout the administration.

Using the validated PP versions of these two tools, computerized versions were developed. The IMM BEVQ-15 began with narrated text and graphic on-screen directions taking approximately three to four minutes. For each drink category, which replicated the paper-based BEVQ-15 in content and sequence, a photo of the beverage was presented on-screen (Figure 1 shows an example of the water intake category). Silhouettes of portion sizes with the quantities they represented in fluid ounces and cups were presented for each beverage category. For example, a soft drink can silhouette was pictured with other beverage containers with the text stating, "a typical beverage can represents 12 fluid ounces or 1 ½ cups." Once the IMM BEVQ-15 was completed, the participant was directed to an instructional page with narrated text (approximately 25 seconds) describing the L-Cat. As with the IMM BEVQ-15, the IMM L-Cat provided audio for each category wherever the mouse's cursor was placed. When the participant chose the physical activity/leisure time activity statement that best reflected their usual physical activity level another completion page was displayed which informed the participant that they were through with the computerized assessment. Completion time was covertly monitored for the IMM BEVQ and L-Cat. After finishing the first IMM administration, participants were invited to complete a user feedback survey that contained seven questions with ordered-response (eg, 1=easy and 5= hard), "Yes" or "No," and open-end response formats.

‎View this figure

Statistical Analysis

Statistical analyses were performed using SPSS statistical analysis software (version 20.0 for Macintosh, 2011, IBM Corporation, Chicago, IL). Demographic characteristics and mean daily beverage consumption (grams, kcal) for the two IMM and one PP BEVQ are reported as mean and standard error of the mean (SEM) or as frequencies (categorical variables). Paired sample t tests and bivariate correlations (Pearson's r) were used to assess validity (first IMM administration vs PP) and reliability (first vs second IMM administration, or IMM-1 vs IMM-2). Due to multiple t tests being conducted, data were reanalyzed to evaluate major outcomes using repeated measures analysis of variance with covariates where appropriate (eg, education), and results were consistent across analytical approaches. Chi square analyses and independent sample t tests were used to assess differences between education groups on categorical variables (eg, gender, race/ethnicity) and continuous variables (eg, BMI, beverage consumption), respectively. The alpha level was set a priori at P≤.05. Responses from the open-ended ease of use questions on the IMM feedback survey were grouped into themes and quantified. Major themes were considered similar responses from ≥50% of participants (≥30 of 60 participants), while minor themes were considered similar responses from 25%-49% of participants (15 to 29, of 60 participants) [40]. A second investigator independently verified themes.

Results

Demographic Characteristics

Demographic characteristics of the study sample are presented in Table 1. Participants were predominantly white (88%, 53/60 participants), but balanced with respect to gender (55% female, 33/60 participants). Age ranged 21 to 70 years, with a mean age of 37 (SEM 2) years. There were no significant differences between education categories for age (P=.38), gender (P=.17), or race/ethnicity (P=.33); however, there were differences in BMI (P=.01), income (P<.001), and health literacy (P<.001) with those in the lower educational category having a higher BMI, and lower income level and health literacy score compared to those in the higher educational category. Differences were found by testing sequence, which was attributed to an unintentional greater random allocation of more participants in the "less than high school/high school" education group being assigned to one of the two sequences.

Completion times for the PP, IMM-1 BEVQ-15 and L-Cat, and IMM-2 BEVQ-15 and L-Cat were approximately three, five, and four minutes, respectively. The paper-based questionnaire time to completion was significantly shorter than both IMM-1 and IMM-2 (both P<.001). Time to completion on IMM-2 versus IMM-1 was also significantly different (P<.001) with the IMM-1 administration taking longer. There were no differences between education categories for time to completion between the PP (P=.69) and the computer-administered versions (IMM-1: P=.44; IMM-2: P=.73).

Comparative Validity of the IMM BEVQ-15 and L-Cat

Results from the comparative validity (ie, comparison of the responses from the IMM BEVQ-15 with the validated PP BEVQ-15) assessment of IMM and PP tools for beverage categories are presented in Table 2. Responses from all beverage categories from the IMM-1 and PP versions were correlated (r=.34-.98, all P<.001), and SSB and total beverage gram and kcal responses were correlated on IMM-1 and PP versions (r=.92-.95, all P<.001). Between IMM-1 and the PP version, no significant differences in beverage category responses were noted. The mean scores for the PP and IMM-1 L-Cat were 3.5 (SEM 0.2) and 2.4 (SEM 0.2), respectively. The paper-based and IMM-1 L-Cat responses were correlated (r=.85, P<.001), but mean values were different (P<.001).

Test-Retest Reliability of the IMM BEVQ-15 and L-Cat

All beverage category responses on IMM-1 and IMM-2 administrations were correlated (Table 3; r=.61-.94, all P<.001). SSB and total beverage gram and kcal responses were correlated on both IMM versions (r=.73-.96, all P<.001). No significant differences in beverage category responses were noted between IMM-1 and IMM-2 with the exception of SSB grams and kcal (P=.02 and P=.01, respectively) and total beverage kcal (P=.01). However, when comparing the responses from the paper and IMM-2 questionnaire administrations on these categories there were no significant differences (mean differences: SSB 63 grams, SEM 87; P=.52; SSB 21 kcal, SEM 33; P=.48; and total beverage 65 kcal, SEM 49; P=.19). The mean L-Cat score on IMM-2 was 2.5 (SEM 0.2), and the IMM L-Cat responses were correlated (r=.86, P<.001). No significant differences were observed between L-Cat responses on the IMM-1 and IMM-2 questionnaires (P=.72); however, differences were noted in IMM-2 and PP L-Cat responses (P<.001).

Comparative Validity and Reliability Within Educational Categories

Figures 2-4 show the results of the major beverage outcomes (water, SSB, total beverage intake) and L-Cat according to educational level. The IMM version of the beverage questionnaire demonstrated comparative validity across the major beverage outcomes with the exception of water intake in the "some college/college degree" participants (mean difference between PP and IMM-1 119, SEM 59; P=.048) with correlation coefficients ranging from .76-.95 (all P<.001). However, responses to the L-Cat were significantly higher with the PP version in both educational groups (mean differences between PP and IMM-1 in less than high school/high school 0.9, SEM 0.2; and in some college/college degree 1.1, SEM 0.1; both P<.001).

No differences were noted in repeated IMM responses for beverage intake or physical activity. Differences were noted in the repeated IMM responses from the "less than high school/high school" participants for SSB grams and kcal (both P=.02) and total beverage energy (P=.03), with lower intake reported on the second administration. However, pair samples t tests results revealed no significant differences between the PP and IMM-2 tools (mean differences–SSB 99 grams, SEM 245; P=.69; SSB 24 kcal, SEM 92; P=.79; total beverage 152 kcal, SEM 134; P=.27). There were no significant differences on L-Cat responses in the "less than high school/high school" participants (P=.67).

‎View this figure

‎View this figure

Comparison of Lower and Higher Educational Groups

Differences in water consumption between education categories on the first IMM and PP questionnaire administrations were significant (both P=.02), but not on the second IMM (P=.14), with the "less than high school/high school" participants reporting significantly lower water consumption than the "some college/college degree" participants (Figure 2). Daily SSB (grams and kcal) and total beverage consumption (grams and kcal) were different between education categories at each questionnaire administration (IMM-1, IMM-2, PP) (Figures 2 and 3). Differences between education categories were noted at each L-Cat administration (IMM-1: P=.008; IMM-2: P=.001; PP: P=.002) (Figure 4).

‎View this figure

User Feedback

Of the 60 participants, 59 completed the feedback survey. Major themes were as follows. Most believed that the IMM BEVQ-15 was "easy"(mean ordered-response rating 1.2, SEM 0.1; n=54), that it was "clear" or "straightforward"(mean 1.4, SEM 0.1; n=30), and that it covered beverages consistent with their usual intake habits (mean 1.1, SEM 0.0; n=56). Most also reported that the computerized L-Cat was "easy" (mean 1.2, SEM 0.1; n=49), and that they were able to identify a physical activity statement relating to their lifestyle with the L-Cat (mean 1.1, SEM 0.1; n=49). Minor themes were that the graphics, images, and voiceover made completing the questionnaires "easy" (mean 1.4, SEM 0.1; n=15), and that nothing needed to be changed in the IMM BEVQ-15 (mean 1.0, SEM 0.1; n=22). Many participants reported that the IMM L-Cat assessment was "clear" or "straightforward" (mean 1.6, SEM 0.1; n=23), and recommended changing nothing about the IMM L-Cat (mean 1.0, SEM 0.1; n=23). Eighteen participants of 60 (30%) suggested that the speed of the narrated text be increased on both the IMM BEVQ-15 and L-Cat. Only two participants (one in each education category) reported the computerized BEVQ-15 as being "hard."

Discussion

Comparative Validity and Test-Reliability of the IMM BEVQ-15 and L-Cat

With the exception of reduced-fat milk, all correlations were greater than .74 (Table 2) and this can be considered superior to other validation studies where typical r values range from .4 to .7 [3], and consistent with initial BEVQ testing [36]. The lower correlation coefficient for reduced-fat milk may be due to participants not being familiar with the form of milk they consume. Some participants may not be the primary food shoppers in their home and not read product packaging prior to consumption; they may not know if the milk they choose to consume is skim, 1%, 2%, or whole milk, and thus choose an option arbitrarily on the questionnaire. The differences between PP and IMM-1 for total beverage intake were 25 kcal, and between PP and IMM-2, while higher at approximately 90 kcal, were not statistically significant. However, we recognize that a 90 kcal difference can be clinically significant at the individual level. Compared to the original BEVQ validation studies, the present differences are higher than what was observed for SSB and total beverage kcal intake. The lower mean age, smaller sample size, and the beverage intake patterns of the "less than high school/high school" education group may have contributed to differences across studies. Future investigations including a larger sample size could provide greater insight into this possibility.

While no significant differences in beverage category responses were noted between IMM-1 and the PP version, mean values were different on the PP and both IMM physical activity items (both P<.001). Computer-based tools can be perceived as more private and less intimidating [7]. If this were so in the present study, the IMM L-Cat responses may be more reflective of actual physical activity levels. Participants may have felt more comfortable reporting a less socially desirable level of physical activity on the computerized assessment versus on the paper-based tool since, when taking the paper version, responses were immediately observable to study personnel, while computerized responses would be accessed following the participant's departure. Overall, the findings of this pilot investigation indicate that the IMM BEVQ-15 is a valid measure of beverage intake when compared to the PP version. Further research is warranted to assess the comparative validity of the IMM L-Cat.

Differences were observed between SSB grams and kcal and total beverage intake (P=.02 and P=.01, respectively) and total beverage kcal (P=.01) between IMM-1 and IMM-2; however, no differences were apparent when comparing these responses on the paper and IMM-2 responses. Thus, the lower responses for the reported usual intake on the second IMM administration were closer to that reported in the PP tool. This may be attributed to a familiarity effect as observed in other computerized assessments [42-44]. In a trial investigating how test mode may impact assessment outcomes, content familiarity and not computer familiarity, gender, or competitiveness positively influenced test performance [16].

Food frequency questionnaires are considered reliable with correlations ranging from .5 to .7 [3,45,46], and many of the coefficients observed for reliability testing of the IMM questionnaires are within or exceed this rage. Thus, the IMM BEVQ-15 and L-Cat can be considered reliable measures of habitual beverage intake and physical activity patterns.

Comparative Validity and Reliability Within Educational Categories

With the exception of water intake in the "some college/college degree" group, the IMM version of the BEVQ-15 demonstrated comparative validity across the major beverage categories. As depicted in Figure 4, both educational groups responded significantly higher on the paper version versus the computerized version of the L-Cat. Since the layout and appearance of computerized surveys can impact participant responses [16,17], differences between the IMM and PP modes of assessment may have occurred in the present investigation. Although lower intakes of SSB grams, kcal, and total beverage kcal were reported on IMM-2 compared to IMM-1 from the "less than high school/high school" participants, no differences were observed between the paper and IMM-2 tools. These results may be attributed to participants being more familiar with the IMM version at the second administration, as stated earlier. Participants potentially had a greater awareness of the upcoming beverage categories within the IMM tool, and thus were able to answer each question more accurately, better reflecting their usual consumption habits.

Comparison of Lower and Higher Educational Groups

In the present investigation, the "less than high school/high school" participants reported significantly lower water consumption than the "some college/college degree" participants (Figure 2). Similarly, the National Health and Nutrition Examination Surveys from 1999-2006 revealed that adults with higher education attainment had a higher plain water intake [47]. In addition, daily grams and kcal were different between educational categories at each questionnaire administration (Figures 2 and 3). Consistent with prior research addressing the influence of educational level and health literacy on beverage consumption patterns [48-50], participants in the lower educational category consumed significantly more total beverage and SSB (grams, kcal). This is noteworthy since excessive SSB consumption has been related to the development of some chronic diseases [51-54]. Similar to the discrepancies observed between physical activity engagement and education attainment in a recent report from the American Heart Association Statistic Committee and Stroke Statistics Subcommittee [55], the "less than high school/high school" category reported lower physical activity engagement than those with "some college/college degree" (Figure 4). However, neither group reported a level of physical activity that met current guidelines [56], exemplifying the need for continued efforts to promote the benefits of regular participation in physical activity.

Usability and User Feedback

Although no differences in completion time were observed between educational categories, the IMM-1 administration took significantly more time to complete versus IMM-2. This is possibly due to unfamiliarity with page-to-page navigation and questionnaire content at the first IMM administration [16].

Participants found the IMM questionnaires easy to use, and that they "fit" their usual beverage intake and leisure-time activity habits. Our results are comparable to others who have reported positive feedback with IMM delivery of nutrition education and dietary and physical activity assessments [7,8,12,14], suggesting acceptability and promise for the use of computer-administered surveys in future research. One area for potential improvement in the IMM tools is the speed of narrated text. Analogous to prior research [7], approximately one-third (n=18) of participants suggested that the speed of the voiceover be increased. The present study received positive feedback overall; however, improvements can be made with the IMM itself (eg, touch screens) [7], which may further increase ease of use and administration.

Limitations

Strengths of this pilot investigation include the random assignment of participants to study session sequences, the novel method to assess dietary intake and physical activity engagement, and the inclusion of participants with lower and higher educational attainment levels. However, several limitations are recognized. The short duration between participant sessions could have caused acclimatization to the questionnaires or participants to be more aware of their beverage intake patterns, thus biasing their responses. Subsequent trials should consider both familiarity with computers and questionnaire content [16,57,58] during participant screening and longer intervals between study sessions. Second, we used validated paper versions of the BEVQ-15 and L-Cat as our comparative criterion; however, self-reported diet and physical activity assessments may not reflect an individual's true intake [1,3,59] or physical activity engagement [60]. In addition, some consider paper versions of computerized surveys to be anything but a "gold standard" when assessing computerized versions of similar assessments [6]. Future studies should not only use validated paper forms of computerized questionnaires, but multiple modes of dietary and physical activity assessment (eg, 24 hour recalls and doubly labeled water) for the greatest degree of accuracy. Although different circumstances may impact beverage intake from day to day (eg, illness, activity level, social events), we do not believe this influenced our overall findings, since the BEVQ-15 has been found valid in estimating group habitual beverage intake [18]. Another potential limitation of the present investigation is the limited racial representation and small sample size. Subsequent larger-scale investigations should include a more diverse sample in terms of race/ethnicity and educational attainment.

Conclusions

There is a need for reliable and valid dietary and physical activity assessment tools that are brief and easily administered [61]. As many as 20% of American adults read at a fifth-grade level or less [62,63], and health literacy is thought to better predict a person's health than ethnicity, employment status, age, income, and education level [64]. Using computer-based assessments can overcome some common barriers preventing the collection of complete dietary data [13], particularly in populations with lower educational achievement [8]. Interactive multimedia versions of dietary and physical activity questionnaires have the potential to decrease participant and study personnel burden, allowing for high quality data to be collected and analyzed [6-8,10,13,14]. Further, computerized assessments could be advantageous for large epidemiological studies [6] as they may reduce costs [8] by streamlining data collection and analysis [4,6-8,13,14]. Overall, the results of the present investigation show that the IMM BEVQ-15 may be used to evaluate habitual beverage intake; although, familiarizing participants with the software prior to data collection may assist in obtaining more accurate data. Respondents may have answered differently on the IMM L-Cat due to computerized tools being considered more confidential and less intimidating [7]. Further research is necessary to fully evaluate the validity of the IMM L-Cat due to its consistency between IMM measures, but differences from the PP version. Future investigations are warranted to include more participants from racially diverse and hard-to-reach audiences (ie, low educational and health literacy levels), develop assessment tools that may be administered to both younger and older individuals (eg, children, adolescents, seniors), and utilize contemporary technological features to further reduce participant burden.