Autism or autistic spectrum disorder (ASD) is a neurodevelopmental condition (affecting nearly 1 per cent of the population) characterised by core differences in social communication, interaction and repetitive behaviours across a variety of contexts (American Psychological Association, 2013). Virtual Reality (VR) includes some interactive video gaming, virtual environments, and, commonly, a multisensory experience. Virtual reality (VR) technologies can provide authentic ‘real world’ contexts that target social and life skills training in safe, controllable and repeatable virtual environments. This article presents some recent work in the development of VR tools for the education of autistic children in Shanghai, China.
“Autism spectrum disorder (ASD), hereafter referred to as Autism (which includes Asperger’s Disorder and Pervasive Developmental Disorder — not Otherwise Specified [PDD-NOS]), is a complex, lifelong developmental condition that typically appears during early childhood and can impact a person’s social skills, communication, relationships, and self-regulation. The autism experience is different for different ASD affected people. It is defined by a certain set of behaviours and is often referred to as a “spectrum condition” that affects people differently and to varying degrees.”1
1 Autism Society, see https://autismsociety.org/the-autism-experience/, accessed Jan 2022
Virtual reality (VR) is defined as “an artificial environment which is experienced through sensory stimuli (such as sights and sounds) provided by a computer and in which one’s actions partially determine what happens in the environment”1. VR uses many different technologies: monoscopic or stereoscopic displays, user tracking technologies, augmented reality (AR) to merge real and virtual worlds, etc.
VR has emerged as an effective new treatment approach in different areas of the health field, such as rehabilitation, promotion of emotional wellbeing in inpatients, diagnosis, surgery training and mental health treatment. With regard to mental health treatment, VR is used in the treatment of a wide range of disorders: Phobias, post-traumatic stress disorders, obsessive-compulsive disorders, and, of course, ASD. Specifically, in this intervention area, VR has shown some advantages, allowing ASD patients to be trained in a realistic environment that could be manipulated and adapted to the characteristics and capabilities of the subject .2
AR is a technology where digital information (images, audio, text) is superimposed on the real world, either with mobile devices or headsets and glasses. AR is emerging as a promising technology to help those with ASD understand the world more fully by bridging the physical and digital worlds. AR can encourage play and improve language, communication, emotion identification, and vocabulary. Results also indicate benefits like increased motivation, attention, and the learning of new tasks.
This article presents the work led by Prof Guangtao Zhai at Shanghai Jiao Tong University, China on the development and assessment of VR tools (apps) for the education of autistic children. The article starts with the characterisation of VR in three dimensions, namely interaction, immersion and imagination to address the ASD triad of communication, interaction and imagination. This is followed by a historical roadmap of VR (superimposed with that of ASD treatment) and a simple conceptual model based on four stages (acceptance, cognition, social, merge) to help guide the development of VR tools. Finally, the article provides a simple example of how to teach an autistic child the idea of safe road crossing, using VR means involving cartoon and robotic characters to attract the attention of autistic children. This is followed by a summary of the test results.
The interactive three-dimensional interface of the virtual world allows users to directly participate in and explore the role and changes of simulated objects in their environment, creating a sense of immersion, and then gaining freedom of thinking and imagination. Regarding the basic characteristics of virtual reality technology, American scientists Grigore Burdea and Philippe Coiffet proposed the three most prominent characteristics of virtual reality at the 1993 World Electronic Conference: interactivity, immersion, and imagination3, which is the 3I characteristics of virtual reality. This kind of virtual world is generated by a computer and can be a reproduction of the real world or a conceived world. The user can use various sensing channels such as vision, hearing and touch to communicate with the virtual world. (Figure 1)
ASD triangle of lack of communication, interaction and imagination (in the middle) just matches Virtual Reality's Three I dimensions: interactivity, immersion and imagination4 (Figure 1). This strong correlation above led to the idea to apply VR tech on the applications of ASD early training and the R & D on this direction in recent years by Prof. Guangtao Zhai.
The world interacts naturally.
INTERACTIVITY refers to the interaction between participants and the virtual environment in a natural way. Users can interact with dedicated interactive devices (such as HMDs, handle controllers, data gloves, position trackers and other sensing devices). The HMD is a display device that provides users with a sense of visual immersion. The handle controller and data glove are the main input devices for virtual reality applications. It is used as a virtual hand or control to simulate interaction in a VR scene.
IMMERSION refers to the degree of reality that users feel as a subject in a virtual environment, and is a direct measure of the performance of a VR system. The main reasons for the user's immersion are as follows: multiperception, the ideal virtual reality should have a variety of visual-based perception functions that humans have; autonomy, the objects in the virtual environment are made according to the laws of physics. Feedback that corresponds to the subject's actions. In addition, the factors that affect immersion include image quality, depth information, field of view, and time delay.
IMAGINATION means that in a virtual environment, users can use various information acquired and their own behaviour in the system, through thought processes such as association, reasoning, and logical judgment, as the operating state of the system changes to affect the future of system movement. Imagine progress in order to acquire more knowledge and understand the deep-level movement mechanisms and regularities of complex systems. With the improvement of related technologies in the virtual reality industry, the capital-driven industry's explosive point is approaching. Some supporting technologies required by virtual reality itself have gradually matured, and the user experience created has begun to gradually reach a level accepted by the public and consumers.
As a large proportion of children with ASD are afraid of hair-washing, haircut, braiding and other different levels of head contact sensitivity, guiding children to accept VR helmets is the key to the following social training. The problem of contact sensitivity in a high proportion of children with ASD has become the biggest obstacle restricting the application of VR technology to ASD research, diagnosis and treatment. After experiments and discussions over 3 months in early 2017, the process of reducing children's insecurity and attracting children to participate in VR training voluntarily has come into being. The overall procedure of the VRASD training includes 4 steps (Figure 2), resolving ASD child’s acceptance of the training interface and system, cognition basis for social contact, social experience and skill training, and merge training for further training in next stage5.
ACCEPTANCE TRAINING: The most important assumption in this stage is the understanding and supporting from the parents, who manage to understand that the key of VRASD startup is ASD children’s free will during VRASD training. We have to wait for ASD children’s interest in VRASD app from inside other than forced by outside. This gains them the sense of security, willing to the steps following. And the performance assessment in this step could well percept the style and speed of the 3 steps of training later6.
COGNITION TRAINING: As the basis of social training, cognition training provides the ASD children with the elements of objects, relationships between objects, the procedure how we operate them, which as the raw materials for information assimilation and integration later.
SOCIAL TRAINING: After the preparation in the previous 2 steps, social training could be much smoother than ever, otherwise this key stage would be full of masses of blackbox without resolution and internal details to the ASD children. All the elements during cognition training are integrated in different scenes in a virtual large shopping-mall (Figure 3) for case study and training. VR could provide practices with quality wellcontrolled in low cost7.
MERGE TRAINING. Merge means the ability of independence, and reliable relationship with others. Therefore, as the last step for this VR training system, VR would show its great advantages on feedback processing and progress assessing, and as the final target to encourage the ASD children with the will and courage to leave VR for more active interactions with real people in the real world. (Figure 3 & 4)
Figure 3: Scenes integrated in a virtual large shopping-mall
During the software development test with ASD children, we created this operation procedure to help about 89 per cent of ASD children accept VR HMD device within 3-time trainings. In the practical operation, if the child hesitates or withdraws before accepting the VR headset, it is helpful to give the child the "right to refuse".
In the large virtual shopping-mall, VRASD integrates 10 grades of scenes with space large enough for indoor navigation training and performance assessment. And the 5-7 levels in each grade provide totally 55 levels to locate ASD child's status and dynamic progress by time. Different scenes emphasis on different aspects, while the scene of crossing road in the last scene of Hello Home is used for exit mechanics after the core mechanics of all the other trainings, which is unique from most of ASD training app around the world. It mainly covers the abilities of reaction in middle level, locating in the urban street, outsides the shopping mall with the movement mechanism of portal (Figure 5). As the last scene, an exit mechanism is applied to reduce the challenge and complexity in this scene greatly from the top challenges in the previous Grade of Find Friend, in order to provide better subjective experience of sense of achievement for ASD children, and encourage them to interact with real people in the real world. (Figure 5)
Here is an interesting example during the experiments: a 5-year-old volunteer ASD child who participated in the experiment had certain language barriers. At the beginning, he could only speak two words. He did not suffer from color blindness, but he could not distinguish colours. He could not use a smart phone or iPad, or complete tasks such as reading and writing on paper. After 3 times VR-ASD experiments, he accepted it. After 3-6 months of systematic VR training once a week, the number of words spoken gradually increased to around 10. He learned more words, and learned the corresponding expressions of various colors and shapes through scene experiments, and began to actively request the time of electronic devices such as smart phones and iPad. This project suffered substantially from the COVID-19 pandemic since 2020, as it required continuous training lasting for years for process evaluation.
Considering that most of the children participating in the ASD test are at the age of intellectual development, and the parents and the school also carry out a lot of training and hard work simultaneously, it cannot be taken for granted that VR adjuvant therapy is the only or the main reason leading to similar obvious effects, however, this type of continuous observation of the phenomenon deserve further experiments in the future and analysis will help to discover more rules for ASD training, thereby providing more clues for the research on the pathogenicity and treatment mechanism of ASD, and then guiding the in-depth research and development of related VR.
This article has presented some initial work in the use of VR for children with ASD in China, which is among the few countries with highest number of children with ASD. For example, VR-ASD helps in improving the attention of ASD children in a stepby- step method. So these children can gradually improve their learning like climbing stairs in a ladder.
The encouraging test results may motivate other countries in the world to deploy VR for the education of children with ASD. However, this VR method requires more research with large number of test subjects and more focused experimentation to help specialists in recommending VR based learning to the appropriate group of children with ASD.
References:
1 Merriam-Webster. Available online: https://www.merriam-webster.com/dictionary/virtual%20reality (accessed on 20 February 2018).
2 P. Mesa-Gresa et al.,Effectiveness of Virtual Reality for Children and Adolescents with Autism Spectrum Disorder: An Evidence-Based Systematic Review, Sensors 2018, 18(8), 2486; https://doi.org/10.3390/s18082486
3 P. Richard, G. Burdea, and P. Coiffet, “Performances Humaines dans des Taches Impliquant des Objets Virtuells avec Retour d'Effort,” Interface to Real and Virtual Worlds Conference, pp. 229-238, Montpellier, France, March, 1993.
4 A. P. Association, “Diagnostic and statistical manual of mental disorders (DSM-IV®),” American Psychiatric Pub, 1994.
5 P. T. Shattuck et al., "Timing of Identification Among Children With an Autism Spectrum Disorder: Findings From a Population-Based Surveillance Study," Journal of the American Academy of Child Adolescent Psychiatry, vol. 48, no. 5, pp. 474-483, 2009.
6 H. Duan, G. Zhai, X. Min, Y. Fang, Z. Che, X. Yang, C. Zhi, H. Yang, and N. Liu, “Learning to Predict where the Children with ASD Look”. IEEE International Conference on Image Processing (ICIP). 704–708, Athens, Greece, 2018.
7 A. P. Association, “Diagnostic and statistical manual of mental disorders (DSM-5®),” American Psychiatric Pub, 2013.
