This paper is in the following e-collection/theme issue:

Non-randomized Protocols and Methods (ehealth) (568) Machine Learning (1471) Autism Spectrum Disorder (ASD) (109) Artificial Intelligence (1377)

Published on in Vol 11, No 7 (2022): July

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/37576, first published .
Feasibility and Impact of Integrating an Artificial Intelligence–Based Diagnosis Aid for Autism Into the Extension for Community Health Outcomes Autism Primary Care Model: Protocol for a Prospective Observational Study

Feasibility and Impact of Integrating an Artificial Intelligence–Based Diagnosis Aid for Autism Into the Extension for Community Health Outcomes Autism Primary Care Model: Protocol for a Prospective Observational Study

Feasibility and Impact of Integrating an Artificial Intelligence–Based Diagnosis Aid for Autism Into the Extension for Community Health Outcomes Autism Primary Care Model: Protocol for a Prospective Observational Study

Authors of this article:

Kristin Sohl1 Author Orcid Image ;   Rachel Kilian2 Author Orcid Image ;   Alicia Brewer Curran1 Author Orcid Image ;   Melissa Mahurin1 Author Orcid Image ;   Valeria Nanclares-Nogués1 Author Orcid Image ;   Stuart Liu-Mayo3 Author Orcid Image ;   Carmela Salomon3 Author Orcid Image ;   Jennifer Shannon3 Author Orcid Image ;   Sharief Taraman3, 4, 5, 6 Author Orcid Image
Article Authors Cited by (4) Tweetations (5) Metrics

    Protocol

    1ECHO Autism Communities, University of Missouri School of Medicine, Columbia, MO, United States

    2SSI Strategy, Parsippany, NJ, United States

    3Cognoa, Inc, Palo Alto, CA, United States

    4Children’s Health of Orange County, Orange, CA, United States

    5Department of Pediatrics, School of Medicine, University of California, Irvine, Irvine, CA, United States

    6Dale E and Sarah Ann Fowler School of Engineering, Chapman University, Orange, CA, United States

    Corresponding Author:

    Sharief Taraman, MD

    Cognoa, Inc

    2185 Park Blvd

    Palo Alto, CA, 94306

    United States

    Phone: 1 6507852624

    Email: sharief@cognoa.com


    Background: The Extension for Community Health Outcomes (ECHO) Autism Program trains clinicians to screen, diagnose, and care for children with autism spectrum disorder (ASD) in primary care settings. This study will assess the feasibility and impact of integrating an artificial intelligence (AI)–based ASD diagnosis aid (the device) into the existing ECHO Autism Screening Tool for Autism in Toddlers and Young Children (STAT) diagnosis model. The prescription-only Software as a Medical Device, designed for use in children aged 18 to 72 months at risk for developmental delay, produces ASD diagnostic recommendations after analyzing behavioral features from 3 distinct inputs: a caregiver questionnaire, 2 short home videos analyzed by trained video analysts, and a health care provider questionnaire. The device is not a stand-alone diagnostic and should be used in conjunction with clinical judgment.

    Objective: This study aims to assess the feasibility and impact of integrating an AI-based ASD diagnosis aid into the ECHO Autism STAT diagnosis model. The time from initial ECHO Autism clinician concern to ASD diagnosis is the primary end point. Secondary end points include the time from initial caregiver concern to ASD diagnosis, time from diagnosis to treatment initiation, and clinician and caregiver experience of device use as part of the ASD diagnostic journey.

    Methods: Research participants for this prospective observational study will be patients suspected of having ASD (aged 18-72 months) and their caregivers and up to 15 trained ECHO Autism clinicians recruited by the ECHO Autism Communities research team from across rural and suburban areas of the United States. Clinicians will provide routine clinical care and conduct best practice ECHO Autism diagnostic evaluations in addition to prescribing the device. Outcome data will be collected via a combination of electronic questionnaires, reviews of standard clinical care records, and analysis of device outputs. The expected study duration is no more than 12 months. The study was approved by the institutional review board of the University of Missouri-Columbia (institutional review board–assigned project number 2075722).

    Results: Participant recruitment began in April 2022. As of June 2022, a total of 41 participants have been enrolled.

    Conclusions: This prospective observational study will be the first to evaluate the use of a novel AI-based ASD diagnosis aid as part of a real-world primary care diagnostic pathway. If device integration into primary care proves feasible and efficacious, prolonged delays between the first ASD concern and eventual diagnosis may be reduced. Streamlining primary care ASD diagnosis could potentially reduce the strain on specialty services and allow a greater proportion of children to commence early intervention during a critical neurodevelopmental window.

    Trial Registration: ClinicalTrials.gov NCT05223374; https://clinicaltrials.gov/ct2/show/NCT05223374

    International Registered Report Identifier (IRRID): PRR1-10.2196/37576

    JMIR Res Protoc 2022;11(7):e37576

    doi:10.2196/37576

    Keywords

    autism spectrum disorder; diagnosis; artificial intelligence; primary care; machine learning; Software as a Medical Device; mobile phone 


    Background

    Autism spectrum disorder (ASD), a neurodevelopmental disorder impacting both communication and behavior, affects approximately 1 in every 44 children in the United States [1]. Targeted early intervention during a critical neurodevelopmental window is recommended to maximize long-term cognitive gains [2,3], adaptive behaviors [4-6], social and emotional functioning [7,8], and verbal fluency [8,9]. ASD evaluations have traditionally been managed in specialist settings; however, a rapid rise in prevalence rates over the past half century [10] has outpaced specialist capacity and led to prolonged wait times for diagnostic evaluations [10-13]. Currently, families in the United States may wait as long as 18 months between initial screening by their primary care clinician and diagnosis by a specialist [11].

    Rising ASD prevalence rates, combined with specialist shortages, have contributed to significant diagnostic delays in the United States. Although ASD can be reliably diagnosed at as early as 18 months, the mean age of ASD diagnosis has remained >4 years since the Centers for Disease Control and Prevention began tracking prevalence rates [1,14-16]. The average 3 years delay from first parental concern to eventual diagnosis recorded in the literature is even longer for demographic groups that include girls, children of color, and children who are rural residing or of lower socioeconomic status [17-20]. An estimated 27% of children with ASD in the United States remain undiagnosed at the age of 8 years [20].

    Given the importance of early diagnosis and intervention, the American Academy of Pediatrics has recommended that all children be screened in primary care at 18 and 24 months, in addition to whenever caregivers express concern [12]. When they feel confident, upon a failed screen, primary care clinicians are encouraged to make the ASD diagnosis themselves using the Diagnostic and Statistical Manual of Mental Disorders–fifth edition (DSM-5) criteria [12]. Alternatively, they can refer to specialists for further assessment. However, studies suggest that clinicians do not consistently screen or refer to these recommendations [21]. Currently, <1% of all ASD diagnoses are made by primary care clinicians in the United States [21,22].

    The low rates of ASD diagnosis in primary care settings may be explained by several factors. Clinician–identified barriers include a lack of ASD-specific training and low self-efficacy in making the diagnosis, as well as time constraints that limit their ability to comprehensively evaluate, review results, and discuss treatment plans with families in primary care [23-26]. Complex presentations in children with ASD, including multiple overlapping comorbidities and a diverse spectrum of symptoms, can also confound the diagnosis [27]. Although diagnostic tools have been developed to support accurate evaluations [28], gold standard ASD diagnostic instruments with the highest validity [28-30] are typically poorly suited to use in primary care. This is because they are time intensive and typically require in-person delivery in a clinical facility and specialist training for reliable administration. Even when clinicians receive specialist training, primary care reimbursement rates may be insufficient to justify the effort [31].

    Device Development

    Diagnosis aids tailored specifically for use in primary care are urgently needed to streamline ASD evaluations and enhance primary care diagnostic capacity. In response to this need, an artificial intelligence (AI)–based Software as a Medical Device [32] was developed to support health care providers in diagnosing or ruling out ASD in children aged 18 to 72 months with concerns for developmental delay. The device is indicated for use by health care providers who have prescriptive authority in their state of practice. For the purposes of this study the device will be used exclusively by primary care Extension for Community Health Outcomes (ECHO) Autism clinicians. The device was developed using patient record data from thousands of children with diverse conditions, presentations, and comorbidities who were either diagnosed with ASD or confirmed not to have ASD based on standardized diagnostic tools [29,30] and representing both sexes across the supported age range. The underlying device algorithm was iteratively improved, supplemented with ASD expert input, and prospectively validated for over 7 years [33-38]. Key steps involved in device use are illustrated in Figure 1.

    ‎
    Figure 1. Overview of device use. ASD: autism spectrum disorder; HCP: health care provider.
    View this figure

    The device integrates 3 independent inputs within its gradient-boosted decision tree algorithm: a brief caregiver questionnaire (completed via a mobile app), a video analyst questionnaire based on the analysis of 2 short home videos uploaded by the caregiver (completed in the video analyst portal), and a health care provider questionnaire (completed in the health care provider portal). The device evaluates inputs based on the predictive features that are most indicative of ASD. If the information provided is sufficiently granular, the device will provide an ASD Positive or ASD Negative output. When a highly predictive determination cannot be made, the device abstains from providing a result (No Result output). Following strong pivotal trial results [39], the device received Food and Drug Administration (FDA) marketing authorization on June 2, 2021 [40]. In a prospective, double-blinded multisite pivotal trial, the device was found to facilitate timely and accurate ASD diagnostic evaluation in nearly a third of the study cohort while minimizing false negatives to maintain clinical safety [39]. However, real-world evidence is needed to learn more about the utility and impact of the device in everyday US primary care settings.

    The ECHO Autism Primary Care Model

    ECHO Autism primary care [41,42] is a virtual learning program in which primary care clinicians receive guidance from ASD experts regarding best practices for ASD screening, identification, and management of medical and psychiatric comorbidities. The diagnostic component of the ECHO Autism primary care model involves training clinicians to conduct diagnostic assessments for ASD using the Screening Tool for Autism in Toddlers and Young Children (STAT), a level 2 ASD screener used as a diagnostic observational measure. The ECHO Autism STAT model provides training and mentorship to equip clinicians to diagnose ASD in cases that are clear and unambiguous and appropriately identify and refer complex cases for further assessment [43]. A 12-month pilot study with 18 participating clinicians was conducted using the ECHO Autism STAT model [43]. In this study, participating clinicians presented the results of their assessment to the ECHO Autism expert hub team for specialist input before guiding the diagnostic pathway to include either a diagnosis of ASD or a referral for further specialist evaluation. This pilot study showed that 73% of the participants reported acceptance of referrals for ASD diagnostic evaluations following program completion, and 80% reported an increase in the number of children with ASD in their caseloads. In addition, the pilot demonstrated a 2- to 6-month decrease in time to access services when compared with standard-of-care diagnostic pathways [41].

    Study Objectives

    The objective of this study is to assess the feasibility and impact of integrating the device into the existing primary care diagnostic pathway, the ECHO Autism STAT model.


    Ethics Approval

    The study was approved by the institutional review board of the University of Missouri-Columbia (institutional review board–assigned project number 2075722) and registered with ClinicalTrials.gov (NCT05223374).

    Study Design

    This prospective observational study was designed to obtain real-world evidence pertaining to the feasibility and impact of integrating the device into a primary care ASD diagnostic pathway. Demographic, clinical, and qualitative user experience data will be obtained from eligible participants. All decisions regarding patient care will be determined by the ECHO Autism clinicians, as the study is noninterventional. All clinical outcomes will be assessed by the ECHO Autism clinician, as in routine clinical practice.

    Study Population

    ECHO Autism Clinicians
    Inclusion Criteria

    Clinicians eligible for study participation will be practicing primary care physicians and nurse practitioners who are able to evaluate patients for ASD as part of their scope of practice. All clinicians must have received training in the ECHO Autism Communities diagnostic model from the University of Missouri ECHO Autism team before study enrollment.

    Exclusion Criteria

    Clinicians who are unable to evaluate patients with suspected ASD or who have not completed the ECHO Autism diagnostic training will be excluded from study enrollment.

    ECHO Autism Program Recruitment

    Starting in 2016, practicing primary care clinicians located in geographically diverse areas across Missouri were invited to participate in ECHO Autism STAT. Participants were identified through word of mouth, professional association listservs, and strategic recruitment from underserved locations. This program was a continuation of the existing ECHO Autism primary care program that started in 2015. A 2-day training was added focused on use of a direct behavior observation tool (STAT) and comprehensive diagnostic interviewing. The ECHO Autism STAT participants completed in-person training, regularly participated in ECHO Autism telementoring sessions, and achieved reliability with the certified trainer on the STAT observation. After completion of the training elements, these participants were eligible to bring diagnostic cases for discussion during the tele–ECHO Autism sessions. Only clinicians completing all elements of the ECHO Autism diagnostic training are eligible for participation in this study. Eligible clinician participants will be informed of the planned study, offered 2 sessions to ask questions about study participation, and consented through usual research procedures.

    Patients and Caregivers
    Inclusion and Exclusion Criteria

    At the time of screening, patients must meet all of the inclusion criteria (Textbox 1) to be eligible for study enrollment. Patients who meet any of the exclusion criteria (Textbox 1) will not be eligible for enrollment in the study.

    Inclusion and exclusion criteria.

    Inclusion criteria

    • Must be aged 18 to 72 months
    • Must have developmental delay concerns expressed by caregivers, clinicians, or other community-based professionals
    • The participating caregiver must speak English proficiently in the opinion of the clinician.
    • Must be able to read, understand, sign, and date the informed consent form
    • Must have a smartphone using either an iOS or Android operating system (as of June 2022, iOS 14 or 15 and Android 11 or 12, excluding Android Go, are supported; future updates are planned to ensure ongoing compatibility with new operating system versions as they are released)
    • Must be willing to have the child videotaped as part of the device input
    • Must meet device labeling requirements

    Exclusion criteria

    • Participants with a prior diagnosis of autism spectrum disorder rendered by a clinician
    • Participants having any other medical, behavioral, or developmental condition that in the opinion of the clinician, may confound study data or assessments
    • Participants whose age on the date of enrollment is outside the target age range
    • Participants who have, to the best of their knowledge, been previously enrolled in any clinical study or survey involving the device or who have a caregiver who has been enrolled in any such study
    • Participants whose medical records have been included in any internal device machine learning training or validation sets
    Textbox 1. Inclusion and exclusion criteria.
    Patient Recruitment

    Participants will be recruited from the population of patients that the ECHO Autism clinician would see as part of their usual enhanced primary care practice. As part of routine practice, ECHO Autism clinicians receive referrals from external primary care clinicians, and when the referrals are received, families will be informed of the opportunity to participate in the study if deemed eligible. Participating clinicians or their designees will inform caregivers of children presenting with a concern for developmental delay about the study. Additional recruitment strategies such as social media, flyers, and email communication with community clinicians may be used as needed to reach recruitment goals.

    Study Flow

    Overview

    This study is being conducted by the University of Missouri ECHO Autism Communities Research Team. Eligible caregivers of patients with a concern for developmental delay and ECHO Autism clinicians will complete the informed consent process before entering the study. Once enrolled, ECHO Autism clinicians will provide routine clinical care and conduct best practice ECHO Autism diagnostic evaluations with the addition of device prescriptions for patients suspected of having ASD. Patients will be followed up with routine clinical care on the schedule shown in Figure 2. Data will be collected at the end of the 3-month follow-up period. The clinician will complete a visit form for any interim visits that occur between visit 1 and the 3-month follow-up appointment.

    The study flow is summarized in Figure 2 and described in detail in the following sections.

    ‎
    Figure 2. Study flow. ASD: autism spectrum disorder; ECHO: Extension for Community Health Outcomes; HCP: health care provider.
    View this figure
    Visit 1: Eligibility Determination

    An appointment is scheduled for a participant (patient) with concern for developmental delay and their caregiver. During the first clinic visit, the ECHO Autism clinician will observe the participant and conduct a DSM-5 interview to determine whether the participant requires further ASD evaluation.

    Baseline Data Collection

    The ECHO Autism clinician will connect the caregiver of any potential participant requiring further evaluation for ASD to the University of Missouri ECHO Autism Communities research team. Before visit 1, the research team will obtain consent from the family. The research team will provide instructions on how to complete the electronic study baseline caregiver questionnaires and download the device’s caregiver app. The research team will also send electronic baseline ECHO Autism clinician questionnaires to the clinician to complete. The clinician will prescribe the device.

    Device Completion

    Once the device is prescribed by the clinician, the study sponsor will release the access code to the study team, who will then email it to the caregiver. The caregiver will download the mobile app to their smartphone and access the app using the code. The caregiver will then complete an 18- or 21-item age-dependent caregiver questionnaire (device input 1) and record and upload two 1.5-minute to 5-minute videos of the child on the mobile app. The uploaded videos will be remotely reviewed and scored on a 28- or 33-item video analyst questionnaire (device input 2) by a trained video analyst via the video analyst portal. The ECHO Autism clinician then has the option to either complete the health care provider questionnaire, which is a 13- or 15-item age-dependent questionnaire (device input 3), themselves or schedule the participant for a telehealth provider appointment, during which the questionnaire will be completed by the telehealth provider. The device algorithm then analyzes all 3 inputs and produces either a positive ASD, negative ASD, or indeterminate/no result output for the clinician to review.

    Visit 2: Diagnostic Results Meeting

    After the device output is received, the clinician will synthesize their diagnostic impressions (ie, in-person observation, DSM-5 interview, and questionnaires) and the device output to determine whether the participant meets the criteria for a diagnosis of ASD. The clinician may also request access to view the participant’s videos when making a diagnosis. In cases where the clinician’s judgment conflicts with the device output, the diagnostic determination of the clinician (who remains the decision-maker in the diagnostic process) takes precedence. This is consistent with the device’s FDA market authorization status as a diagnosis aid intended to support health care providers in diagnosing ASD rather than as a stand-alone diagnostic device.

    For participants whom the clinician determines as meeting the criteria of ASD, the clinician will deliver the diagnosis to the caregiver during visit 2. In addition, the clinician will schedule a 3-month follow-up visit. The research team will then send electronic postdiagnostic questionnaires to the caregiver and the clinician to be completed 1 to 3 (up to 10) days after visit 2.

    Visit 3: Treatment Progress Meeting

    The clinician will follow-up with the caregiver for 3 months (+21 or –21 days) after the participant receives an ASD diagnosis to discuss treatment progress and screen for any other medical and psychiatric comorbidities. The research team will send electronic postdiagnostic questionnaires to the caregiver and clinician to complete 1 to 3 (up to 10) days after this follow-up visit.

    Interim Visit Forms

    Clinicians will complete interim visit forms for any additional visits that may occur between study enrollment and the scheduled final 3-month follow-up visit.

    Long-term Follow-up

    Participants and their caregivers may be considered for enrollment if they choose to consent to an extension study to continue data collection related to long-term outcomes.

    Number of Sites Enrolling Participants

    Up to 15 University of Missouri ECHO Autism clinicians in both rural and suburban areas across the United States will be recruited.

    Study Duration

    Study enrollment is expected to take approximately 12 months. Patient participation will occur from the time the ECHO Autism clinician identifies a concern for ASD through +21 or −21 days of the 3-month follow-up visit.

    Potential Risks and Benefits of Participation

    There are no physical risks associated with participation in the study. Caregivers may feel some psychological discomfort when answering questions about their children’s behaviors. Children might also feel some psychological discomfort when being filmed for the video analysis. Videos will be used to evaluate participants’ behaviors as part of the diagnostic process. The videos will be captured within the mobile app, stored on Amazon’s secure cloud service (Amazon S3), and analyzed by the device manufacturer through a secure web portal. By agreeing to participate in the study, participants will grant and release all rights, titles, and interests in the videos to the device manufacturer. Another potential risk is the possible release of sensitive medical, behavioral, or educational information. To mitigate the risk of releasing sensitive personal information, the device manufacturer will adhere to confidentiality protocols aligned with the US Health Insurance Portability and Accountability Act and FDA 21 Code of Federal Regulations Part 11, as well as other federal, national, and local laws and regulations, to protect the privacy of personal information. All content within the mobile app will be stored on Amazon S3 and analyzed by the device manufacturer through a secure web portal. When data are published, they will be anonymized in a manner that prevents the identification of specific caregivers or participants. To the best of the manufacturer’s knowledge, there were no data or security breaches in any prior studies involving the device.

    A potential benefit of the study is that device use could support ECHO Autism clinicians in identifying participants with ASD earlier than they would otherwise have been identified, thus enabling earlier therapy, which is known to positively affect long-term outcomes.

    Participant Withdrawal or Termination

    Participation is completely voluntary. Caregivers may withdraw consent for study participation at any time by informing the principal investigator or the study staff. The principal investigator may also withdraw a participant from the study if determined that further participation would not be in the participant’s best interest. Participants may be discontinued by the study staff if determined unable to fully meet the requirements of participation, including the completion of study visits and necessary assessments. The reasons for discontinuation will be recorded for all participants. If lost to follow-up, the research team will attempt to contact the caregiver by telephone, followed by registered mail, to establish the reason for withdrawal as completely as possible.

    Study Objectives

    Primary Objective

    The primary objective is to assess the time from the initial concern by the ECHO Autism clinician to diagnosis when using the device as part of the diagnostic pathway.

    Secondary Objectives

    Secondary objectives include the collection and analysis of data from both the ECHO Autism clinician and caregiver participants to learn about the information described in Textbox 2.

    Data collected for secondary objectives.

    Data collected for secondary objectives

    • Patient outcomes correlating with time to initiating therapy following a positive autism spectrum disorder (ASD) diagnosis
    • Distance from patient home to nearest ASD specialist center
    • Level of clinician self-efficacy in using the device as part of the evaluation and diagnosis of children with suspected ASD
    • Qualitative data on the following: caregiver satisfaction with the diagnostic process; time from caregiver concern to diagnosis; clinician certainty rating for diagnostic determination; clinician satisfaction with using the device as part of the Extension for Community Health Outcomes (ECHO) Autism diagnostic process; caregiver perception of the device with validated questionnaire; determination of whether the use of the device enhances the efficiency and feasibility of the ECHO Autism Communities diagnostic process; likelihood to recommend (clinician and caregiver); efficiency and feasibility of ECHO Autism Community–based diagnostic process, as measured by self-efficacy, certainty ratings, perceived barriers, and practice patterns questionnaires; percentage of positive outcomes correlating with shorter durations between diagnosis and treatment; percentage of caregivers who feel they received a high-quality evaluation; percentage of clinicians with increased self-efficacy in using the device as part of the diagnostic process; percentage of clinicians finding the device helpful in the diagnostic process; percentage of positive device perceptions from caregiver; barriers to app use for caregivers; clinician-perceived barriers to practice implementation of device
    • Assessment of insurance coverage or reimbursement (although insurance will not be billed for this study)
    • Amount of time required for clinicians to make a diagnosis
    • Percentage of clinician participants reporting a positive change in practice flow because of using the device as part of their diagnostic process
    • Percentage of patients for whom the device can provide a determinate result (ASD positive or negative)
    • Summary of interventional treatments recommended by the clinician following diagnosis
    • Changes in clinician clinical practice flow
    • Change in quality of life measures
    • Time associated with device use
    Textbox 2. Data collected for secondary objectives.

    Data Collection Instruments and Schedule of Assessments

    Clinical History

    Details of participants’ clinical and medical histories will be reviewed and captured, including the description and timing of the initial concern for ASD.

    Diagnostic Procedures

    Diagnostic procedures will be completed according to the ECHO Autism Communities standard of care, as determined by the treating clinician in routine clinical practice. Details of any such procedure will be captured throughout the study, including the follow-up period after a participant has received diagnostic results.

    Therapeutic Interventions

    Any new therapeutic interventions recommended by the ECHO Autism clinician will be captured at time points 1, 2, and 3 and for any interim visit. Once a recommendation has been made, data will be collected during subsequent visits to indicate patient compliance with the recommendations.

    All data collection instruments are listed and described in Textbox 3.

    Data collection instruments.

    Caregiver questionnaire (device input 1)

    • This is not a stand-alone assessment. The caregiver questionnaire will be completed within the mobile caregiver app upon receipt of the access code following completion of the informed consent form. This is an 18- or 21-item questionnaire that collects details related to the caregiver’s observations of the participant’s developmental behaviors.

    Video analyst questionnaire (device input 2)

    • This is not a stand-alone assessment. Video analysts will answer a 28- or 33-item age-dependent questionnaire in the video analyst portal after viewing the 2 short video submissions uploaded by the caregiver. Video analysts will evaluate the observed behaviors by answering a series of multiple-choice questions evaluating phenotypic features of autism spectrum disorder (ASD) such as nonverbal and verbal communication, social interaction, unusual sensory interests or reactions, stereotypic or repetitive motor movements, use of objects, or speech on the combinative videos.

    Health care provider questionnaire (device input 3)

    • This is not a stand-alone assessment. The clinician or telehealth provider will schedule a visit with the caregiver and participant (which can be remote). During this visit, the participant must be available for observation. The clinician or telehealth vendor will complete the 13- or 15-item questionnaire within the health care provider portal. The questionnaire addresses aspects of the child’s behavior and development.

    Device output

    • Device output is produced by the device summarizing findings after algorithmic analysis of the 3 device inputs and contains one of 3 possible results: positive ASD, negative ASD, or no result. It is to be used by the clinician as part of the Extension for Community Health Outcomes (ECHO) Autism Community–based diagnostic process.

    Measure of processes of care

    • This measure assesses the family’s perceptions of the care they and their child receive within a particular setting. It is a means to assess the family-centered behaviors of clinicians. This is a validated measure that has been adapted for this study.

    ECHO Autism caregiver presurvey

    • This survey collects initial demographic information regarding each family and assesses the family’s initial perception of the ECHO Autism clinician as they begin the diagnostic process.

    ECHO Autism caregiver postdiagnostic survey

    • This survey assesses the caregiver’s satisfaction with the services they received.

    ECHO Autism clinician preproject survey

    • This survey measures the ECHO Autism clinician’s self-efficacy, perceived barriers, and satisfaction with the current ECHO Autism standard-of-care procedure.

    ECHO Autism clinician postproject survey

    • This survey measures the self-efficacy, perceived barriers, and satisfaction with the use of the device as part of the ECHO Autism standard-of-care procedure.

    ECHO Autism 3-month follow-up questionnaires

    • The purpose of these questionnaires is to address the data points required for the study’s secondary and exploratory objectives.

    ECHO Autism interim visit form

    • The purpose of the form is to summarize any visit from a research participant occurring between the time of concern and the 3-month follow-up visit.
    Textbox 3. Data collection instruments.

    Statistical Analyses

    Statistical and Analytical Plans

    Descriptive statistics will be used to summarize the data. Continuous variables will be summarized as n, mean, SD, median, minimum, and maximum values. Categorical variables will be summarized as participant counts and related percentages.

    General Approach

    All available data will be included in data listings and tabulations. Data will be summarized by participant counts and related percentages. In general, categorical data will be presented using counts and percentages, whereas continuous variables will be presented using mean, SD, median, minimum, and maximum values. Counts of observed and missing data will also be reported.

    Safety Analysis

    Device- and procedure-related adverse effects will be tabulated and analyzed.

    Sample Size

    This is a prospective study with up to 100 patient participants determined eligible for an evaluation for ASD and up to 15 ECHO Autism clinicians who conduct assessments.


    Participant recruitment began in April 2022. As of June 2022, a total of 41 participants have been enrolled in the study. Data collection is anticipated to continue through 2022, with the data analysis and publication of findings being planned for 2023.


    Principal Findings

    To our knowledge, this collaborative observational study will be the first to evaluate the use of a novel AI-based ASD diagnosis aid as part of a real-world primary care diagnostic pathway. It is anticipated that this study will provide valuable feasibility data regarding device integration into a primary care workflow. The study results may help clarify the length of time between initial clinician concern and diagnosis when an AI-based device is used as part of the evaluation process. The results may also provide insights into device performance outside of a research setting, including the percentage of primary care participants for whom it is able to provide a determinate ASD positive or negative result, and concordance between device output and clinical diagnostic determination. Qualitative user experience measures captured as part of the study are anticipated to shed light on the acceptability of the device to both families and clinicians and may help drive future product improvements.

    Comparison With Previous Literature

    Although to date, no previous AI-based ASD diagnosis aid primary care–focused studies have been conducted, there is literature supporting the feasibility of diagnosing and managing ASD in primary care settings. For example, a recent systematic review focused on training clinicians to diagnose and manage ASD in primary care [44] found that with appropriate training and support, primary care clinicians can render diagnostic decisions that closely align with specialty team assessments. The reviewed studies also reported reduced wait times for diagnosis when a proportion of evaluations were moved to the primary care setting.

    Strengths and Limitations

    Despite well-documented bottlenecks in ASD diagnosis and treatment initiation in the United States [11,45], to date, very few diagnostic tools have been designed to enhance primary care diagnostic capacity. The integration of the device into the ECHO Autism primary care diagnostic pathway in this study offers several potential benefits over existing tools [28-30]. As it was designed with primary care use in mind, the device is more time-efficient than existing instruments and does not require specialist training to administer. For example, in the pivotal trial, the total time burden associated with device use was <30 minutes (median time for caregiver questionnaire completion was 4 minutes and 56 seconds; median time for video analyst scoring from the time video review began to submission of scores was 10 minutes and 54 seconds; self-reported time for health care providers to complete their questionnaire was approximately 10 minutes) [39].

    The device is also amenable to remote administration. Caregivers will complete their questionnaires and upload videos remotely using an app on their mobile phones. Video analysts will receive inputs and upload questionnaire outputs remotely through an analyst portal. Pivotal trial data showed no evidence of performance degradation in cases where health care providers completed their questionnaires remotely versus in person. Remote administration may support diagnostic access for families residing in rural and remote locations, as well as families without access to transportation. Remote administration is also useful during public health emergencies such as the COVID-19 pandemic.

    Compared with other ASD assessment instruments where the child is observed in a clinical setting, the device leverages home videos that provide naturalistic data about the child’s behavior outside of the clinical setting. These data may help to overcome some of the challenges clinicians and caregivers have described capturing typical child behavior within time-limited clinical encounters [46]. For example, in the clinical setting, the child may become overly reactive to changes in the environment or may not display the behaviors of concern they exhibit at home. Without a picture of the child’s typical behavior, it can be challenging for the clinician to make a confident diagnostic assessment.

    The device was also trained on a large and diverse data set in terms of race and ethnicity, socioeconomic status, and gender. Although the pivotal trial was not powered for statistical inference on these covariates, our initial analysis indicated no evidence of inconsistent device performance across these variables [39]. This finding is encouraging and points to the potential of the device to address some of the well-documented disparities in current diagnostic approaches whereby girls and African American and Hispanic children are misdiagnosed more often, as well as diagnosed at a later age, on average, than other groups [47,48].

    The device itself and the overall study design have several constraints and limitations. Device use requires access to a smartphone and wireless internet, which not all low-income American families have [49]. In addition, the device is only available in English. Thus, many culturally and linguistically diverse potential participants will be excluded from the study. Selection bias may also be a concern in this study. For example, parents with a higher number of concerns about their child’s development may be more motivated to enroll in the study and complete all device inputs than parents with milder concerns. Attrition is another concern in this study as participation involves multiple steps and engagement over a 3-month period. Given the multiple steps involved, it is possible that study completers may end up representing a more motivated cohort than study leavers. The clinicians who will be recruited into the trial may also represent a biased sample, as by enrolling in ECHO training, they have already demonstrated an interest in enhancing their knowledge of developmental disabilities. In addition, owing to the ECHO Autism curriculum training they receive, clinicians may have more experience and background in ASD diagnosis than other primary care clinicians. In other settings, clinicians with less interest in ASD may be less motivated or confident in using the device and may be less willing to diagnose in the primary care setting.

    Future Directions

    The study follows participants for a total of 3 months. Additional follow-up studies are recommended to assess the stability of device performance over a longer period. For the purposes of this study, the sponsors will cover the device cost and associated clinician fees. However, if the device were to become a standard tool in primary care settings, it is unknown to what extent insurance companies would accept and reimburse these costs. Questions also remain regarding whether and how families with no health insurance can access the device. We recommend that future device studies be conducted in populations who have not completed the ECHO Autism diagnostic training program. Such research could help answer important questions about the feasibility of device use in primary care clinician populations with less ASD-specific knowledge and training.

    Conclusions

    If device integration into primary care practice proves feasible and efficacious, it may support clinicians in diagnosing more children in the primary care setting and reducing current delays between the first ASD concern and eventual diagnosis. Streamlining primary care ASD diagnosis could potentially reduce strain on specialist services by allowing for the diagnosis of less ambiguous cases in primary care and quicker referral of complex cases to relevant specialist teams. Ultimately, we hope that decreasing the time to diagnosis and age of diagnosis through enhanced primary care capacity will allow a greater proportion of children to commence early intervention during a critical neurodevelopmental window.

    Acknowledgments

    This study is sponsored by Cognoa Inc.

    Conflicts of Interest

    KS has received consulting fees from Cognoa, is on the Medical Advisory Board for Quadrant Biosciences, and provides research support for Autism Speaks. SL-M, JS, CS, and ST are employees of Cognoa and have Cognoa stock options. ST additionally receives consulting fees for Cognito Therapeutics, volunteers as a board member of the American Academy of Pediatrics-Orange County Chapter and American Academy of Pediatrics California, is a paid adviser for MI10 LLC, and owns stock for NTX, Inc, and HandzIn.

    1. Maenner MJ, Shaw KA, Bakian AV, Bilder DA, Durkin MS, Esler A, et al. Prevalence and Characteristics of Autism Spectrum Disorder Among Children Aged 8 Years - Autism and Developmental Disabilities Monitoring Network, 11 Sites, United States, 2018. MMWR Surveill Summ 2021 Dec 03;70(11):1-16 [FREE Full text] [CrossRef] [Medline]
    2. Ben Itzchak E, Zachor DA. Who benefits from early intervention in autism spectrum disorders? Res Autism Spectrum Disorders 2011 Jan;5(1):345-350. [CrossRef]
    3. Flanagan H, Perry A, Freeman N. Effectiveness of large-scale community-based Intensive Behavioral Intervention: a waitlist comparison study exploring outcomes and predictors. Res Autism Spectrum Disorders 2012 Apr;6(2):673-682. [CrossRef]
    4. Smith T, Klorman R, Mruzek DW. Predicting outcome of community-based early intensive behavioral intervention for children with autism. J Abnorm Child Psychol 2015 Oct 17;43(7):1271-1282. [CrossRef] [Medline]
    5. Dawson G, Rogers S, Munson J, Smith M, Winter J, Greenson J, et al. Randomized, controlled trial of an intervention for toddlers with autism: the Early Start Denver Model. Pediatrics 2010 Jan;125(1):e17-e23 [FREE Full text] [CrossRef] [Medline]
    6. Baker-Ericzén M, Stahmer A, Burns A. Child demographics associated with outcomes in a community-based pivotal response training program. J Positive Behav Interv 2016 Aug 13;9(1):52-60. [CrossRef]
    7. Mazurek MO, Kanne SM, Miles JH. Predicting improvement in social–communication symptoms of autism spectrum disorders using retrospective treatment data. Res Autism Spectrum Disorder 2012 Jan;6(1):535-545. [CrossRef]
    8. MacDonald R, Parry-Cruwys D, Dupere S, Ahearn W. Assessing progress and outcome of early intensive behavioral intervention for toddlers with autism. Res Dev Disabil 2014 Dec;35(12):3632-3644. [CrossRef] [Medline]
    9. Vivanti G, Dissanayake C, Victorian ASELCC Team. Outcome for children receiving the early start denver model before and after 48 months. J Autism Dev Disord 2016 Jul 28;46(7):2441-2449. [CrossRef] [Medline]
    10. Elsabbagh M, Divan G, Koh Y, Kim YS, Kauchali S, Marcín C, et al. Global prevalence of autism and other pervasive developmental disorders. Autism Res 2012 Jun 11;5(3):160-179 [FREE Full text] [CrossRef] [Medline]
    11. Gordon-Lipkin E, Foster J, Peacock G. Whittling down the wait time: exploring models to minimize the delay from initial concern to diagnosis and treatment of autism spectrum disorder. Pediatr Clin North Am 2016 Oct;63(5):851-859 [FREE Full text] [CrossRef] [Medline]
    12. Hyman S, Levy S, Myers S, Council on Children With Disabilities‚ Section on Developmental and Behavioral Pediatrics. Identification, evaluation, and management of children with autism spectrum disorder. Pediatrics 2020 Jan;145(1):e20193447. [CrossRef] [Medline]
    13. Bridgemohan C, Bauer N, Nielsen B, DeBattista A, Ruch-Ross H, Paul L, et al. A workforce survey on developmental-behavioral pediatrics. Pediatrics 2018 Mar;141(3):e20172164. [CrossRef] [Medline]
    14. Baio J, Wiggins L, Christensen DL, Maenner MJ, Daniels J, Warren Z, et al. Prevalence of autism spectrum disorder among children aged 8 years - autism and developmental disabilities monitoring network, 11 sites, United States, 2014. MMWR Surveill Summ 2018 Apr 27;67(6):1-23 [FREE Full text] [CrossRef] [Medline]
    15. van 't Hof M, Tisseur C, van Berckelear-Onnes I, van Nieuwenhuyzen A, Daniels A, Deen M, et al. Age at autism spectrum disorder diagnosis: a systematic review and meta-analysis from 2012 to 2019. Autism 2021 May;25(4):862-873 [FREE Full text] [CrossRef] [Medline]
    16. Pierce K, Gazestani VH, Bacon E, Barnes CC, Cha D, Nalabolu S, et al. Evaluation of the diagnostic stability of the early autism spectrum disorder phenotype in the general population starting at 12 months. JAMA Pediatr 2019 Jun 01;173(6):578-587 [FREE Full text] [CrossRef] [Medline]
    17. Delobel-Ayoub M, Ehlinger V, Klapouszczak D, Maffre T, Raynaud J, Delpierre C, et al. Socioeconomic disparities and prevalence of autism spectrum disorders and intellectual disability. PLoS One 2015 Nov 5;10(11):e0141964 [FREE Full text] [CrossRef] [Medline]
    18. Oswald D, Haworth S, Mackenzie B, Willis J. Parental report of the diagnostic process and outcome: ASD compared with other developmental disabilities. Focus Autism Other Dev Disabl 2015 Jun 09;32(2):152-160. [CrossRef]
    19. Wiggins LD, Durkin M, Esler A, Lee L, Zahorodny W, Rice C, et al. Disparities in documented diagnoses of autism spectrum disorder based on demographic, individual, and service factors. Autism Res 2020 Mar 23;13(3):464-473 [FREE Full text] [CrossRef] [Medline]
    20. Shattuck PT, Durkin M, Maenner M, Newschaffer C, Mandell DS, Wiggins L, et al. Timing of identification among children with an autism spectrum disorder: findings from a population-based surveillance study. J Am Academy Child Adolescent Psychiatry 2009 May;48(5):474-483. [CrossRef]
    21. Monteiro SA, Dempsey J, Berry LN, Voigt RG, Goin-Kochel RP. Screening and referral practices for autism spectrum disorder in primary pediatric care. Pediatrics 2019 Oct 12;144(4):e20183326. [CrossRef] [Medline]
    22. Rhoades RA, Scarpa A, Salley B. The importance of physician knowledge of autism spectrum disorder: results of a parent survey. BMC Pediatr 2007 Nov 20;7(1):37 [FREE Full text] [CrossRef] [Medline]
    23. Carbone P, Norlin C, Young P. Improving early identification and ongoing care of children with autism spectrum disorder. Pediatrics 2016 Jun;137(6):e20151850. [CrossRef] [Medline]
    24. Fenikilé T, Ellerbeck K, Filippi M, Daley C. Barriers to autism screening in family medicine practice: a qualitative study. Prim Health Care Res Dev 2014 Nov 4;16(04):356-366. [CrossRef]
    25. Self T, Parham D, Rajagopalan J. Autism spectrum disorder early screening practices. Commun Disorder Q 2014 Dec 30;36(4):195-207. [CrossRef]
    26. Mazurek MO, Harkins C, Menezes M, Chan J, Parker RA, Kuhlthau K, et al. Primary care providers' perceived barriers and needs for support in caring for children with autism. J Pediatr 2020 Jun;221:240-5.e1. [CrossRef] [Medline]
    27. Lai M, Kassee C, Besney R, Bonato S, Hull L, Mandy W, et al. Prevalence of co-occurring mental health diagnoses in the autism population: a systematic review and meta-analysis. Lancet Psychiatry 2019 Oct;6(10):819-829. [CrossRef]
    28. Falkmer T, Anderson K, Falkmer M, Horlin C. Diagnostic procedures in autism spectrum disorders: a systematic literature review. Eur Child Adolesc Psychiatry 2013 Jun 16;22(6):329-340. [CrossRef] [Medline]
    29. Lord C, Rutter M, Le Couteur A. Autism diagnostic interview-revised: a revised version of a diagnostic interview for caregivers of individuals with possible pervasive developmental disorders. J Autism Dev Disord 1994 Oct;24(5):659-685. [CrossRef]
    30. (ADOS®-2) Autism Diagnostic Observation Schedule™, Second Edition. WPS Publish.   URL: https://www.wpspublish.com/ados-2-autism-diagnostic-observation-schedule-second-edition [accessed 2021-12-14]
    31. Kaufman NK. Rethinking "gold standards" and "best practices" in the assessment of autism. Appl Neuropsychol Child 2022 Aug 27;11(3):529-540. [CrossRef] [Medline]
    32. Software as a Medical Device (SaMD). U.S. Food & Drug Administration.   URL: https://www.fda.gov/medical-devices/digital-health-center-excellence/software-medical-device-samd [accessed 2021-12-12]
    33. Abbas H, Garberson F, Liu-Mayo S, Glover E, Wall DP. Multi-modular AI approach to streamline autism diagnosis in young children. Sci Rep 2020 Mar 19;10(1):5014 [FREE Full text] [CrossRef] [Medline]
    34. Abbas H, Garberson F, Glover E, Wall D. Machine learning approach for early detection of autism by combining questionnaire and home video screening. J Am Med Inform Assoc 2018 Aug 01;25(8):1000-1007 [FREE Full text] [CrossRef] [Medline]
    35. Levy S, Duda M, Haber N, Wall DP. Sparsifying machine learning models identify stable subsets of predictive features for behavioral detection of autism. Mol Autism 2017 Dec 19;8(1):65 [FREE Full text] [CrossRef] [Medline]
    36. Tariq Q, Daniels J, Schwartz J, Washington P, Kalantarian H, Wall D. Mobile detection of autism through machine learning on home video: a development and prospective validation study. PLoS Med 2018 Nov;15(11):e1002705 [FREE Full text] [CrossRef] [Medline]
    37. Wall DP, Dally R, Luyster R, Jung J, Deluca TF. Use of artificial intelligence to shorten the behavioral diagnosis of autism. PLoS One 2012 Aug 27;7(8):e43855 [FREE Full text] [CrossRef] [Medline]
    38. Kosmicki JA, Sochat V, Duda M, Wall DP. Searching for a minimal set of behaviors for autism detection through feature selection-based machine learning. Transl Psychiatry 2015 Mar 24;5(2):e514 [FREE Full text] [CrossRef] [Medline]
    39. Megerian JT, Dey S, Melmed RD, Coury DL, Lerner M, Nicholls CJ, et al. Evaluation of an artificial intelligence-based medical device for diagnosis of autism spectrum disorder. NPJ Digit Med 2022 May 05;5(1):57 [FREE Full text] [CrossRef] [Medline]
    40. Cognoa receives FDA marketing authorization for first-of-its-kind Autism Diagnosis Aid. Cognoa.   URL: https:/​/cognoa.​com/​cognoa-receives-fda-marketing-authorization-for-first-of-its-kind-autism-diagnosis-aid/​ [accessed 2021-10-10]
    41. Mazurek M, Brown R, Curran A, Sohl K. ECHO Autism. Clin Pediatr (Phila) 2017 Mar;56(3):247-256. [CrossRef] [Medline]
    42. Sohl K, Mazurek M, Brown R. ECHO Autism: using technology and mentorship to bridge gaps, increase access to care, and bring best practice autism care to primary care. Clin Pediatr (Phila) 2017 Jun;56(6):509-511. [CrossRef] [Medline]
    43. Mazurek M, Curran A, Burnette C, Sohl K. ECHO Autism STAT: accelerating early access to autism diagnosis. J Autism Dev Disord 2019 Jan;49(1):127-137. [CrossRef] [Medline]
    44. Guan X, Zwaigenbaum L, Sonnenberg L. Building capacity for community pediatric autism diagnosis: a systemic review of physician training programs. J Dev Behav Pediatr 2021 Dec 15;43(1):44-54. [CrossRef]
    45. Bishop-Fitzpatrick L, Kind AJ. A scoping review of health disparities in autism spectrum disorder. J Autism Dev Disord 2017 Nov 29;47(11):3380-3391 [FREE Full text] [CrossRef] [Medline]
    46. Nazneen N, Rozga A, Romero M, Findley AJ, Call NA, Abowd GD, et al. Supporting parents for in-home capture of problem behaviors of children with developmental disabilities. Pers Ubiquit Comput 2011 May 1;16(2):193-207. [CrossRef]
    47. Constantino JN, Abbacchi AM, Saulnier C, Klaiman C, Mandell DS, Zhang Y, et al. Timing of the diagnosis of autism in African American children. Pediatrics 2020 Sep 24;146(3):e20193629 [FREE Full text] [CrossRef] [Medline]
    48. Yingling ME, Hock RM, Bell BA. Time-lag between diagnosis of autism spectrum disorder and onset of publicly-funded early intensive behavioral intervention: do race-ethnicity and neighborhood matter? J Autism Dev Disord 2018 Feb 28;48(2):561-571. [CrossRef] [Medline]
    49. Mobile Fact Sheet. Pew Research Center.   URL: https://www.pewresearch.org/internet/fact-sheet/mobile/ [accessed 2021-12-11]

    ‎
    AI: artificial intelligence
    ASD: autism spectrum disorder
    DSM-5: Diagnostic and Statistical Manual of Mental Disorders–fifth edition
    ECHO: Extension for Community Health Outcomes
    FDA: Food and Drug Administration
    STAT: Screening Tool for Autism in Toddlers and Young Children

    Edited by T Leung; submitted 25.02.22; peer-reviewed by B Nievas Soriano, K Shamlian; comments to author 24.05.22; revised version received 20.06.22; accepted 24.06.22; published 19.07.22

    Copyright

    ©Kristin Sohl, Rachel Kilian, Alicia Brewer Curran, Melissa Mahurin, Valeria Nanclares-Nogués, Stuart Liu-Mayo, Carmela Salomon, Jennifer Shannon, Sharief Taraman. Originally published in JMIR Research Protocols (https://www.researchprotocols.org), 19.07.2022.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on https://www.researchprotocols.org, as well as this copyright and license information must be included.