Systematic literature review and meta-analysis of emotion recognition in children with autism

The aim of the intellectual output was to investigate state-of-the-art in emotion recognition in children with autism.

The automatic emotion recognition domain brings new methods and technologies that might be used to enhance the therapy of children with autism. The study aimed at the exploration of methods and tools used to recognize emotions in children. It was performed as a literature review study using a systematic approach and PRISMA methodology for reporting quantitative and qualitative results. Diverse observation channels and modalities are used in the analyzed studies, including facial expressions, the prosody of speech, and physiological signals. Regarding representation models, the basic emotions are the most frequently recognized, especially happiness, fear, and sadness. Both single-channel and multichannel approaches are applied, with a preference for the first one. For multimodal recognition, early fusion was the most frequently applied. SVM and neural networks were the most popular for building classifiers. Qualitative analysis revealed important clues on participant group construction and the most common combinations of modalities and methods. All channels are reported to be prone to some disturbance, and as a result, information on specific symptoms of emotions might be temporarily or permanently unavailable. The challenges of proper stimuli, labeling methods, and the creation of open datasets were also identified.

Results

1. Presentation of the collection of papers "Emotion recognition in children with autism. A collection of papers extracted from systematic literature review under Erasmus+ EMBOA project", [download];
2. Technical report: Aleksadra Karpus, Agnieszka Landowska, Jakub Miler, Małgorzata Pykała: "SYSTEMATIC LITERATURE REVIEW – METHODS AND HINTS", Technical report of ETI Faculty 1/2020, Politechnika Gdanska, 1/2020, [download] ;
3. Journal publication: Landowska, A.; Karpus, A.; Zawadzka, T.; Robins, B.; Erol Barkana, D.; Kose, H.; Zorcec, T.; Cummins, N. Automatic Emotion Recognition in Children with Autism: A Systematic Literature Review. Sensors 2022, 22, 1649, [download], https://doi.org/10.3390/s22041649.

Systematic literature review and meta-analysis of interventions regarding Robots and emotional skills in children with autism

The aim of the intellectual output was to explore the latest developments in social robots in intervention for children with autism.

Children with autism spectrum disorder (ASD) have deficits in the socio-communicative domain and frequently face severe difficulties in the recognition and expression of emotions. Existing literature suggested that children with ASD benefit from robot-based interventions. However, studies varied considerably in participant characteristics, applied robots, and trained skills. Nao and ZECA were the most frequently used robots; recognition of basic emotions and getting into interaction were the most frequently trained skills, while happiness, sadness, fear, and anger were the most frequently trained emotions. The studies reported a wide range of challenges with respect to robot-based intervention, ranging from limitations for certain ASD subgroups and security aspects of the robots to efforts regarding the automatic recognition of the children’s emotional state by the robotic systems. Finally, we summarised and discussed recommendations regarding the application of robot-based interventions for children with ASD.

Results

1. Presentation of the collection of papers "Emotions in robot-based interventions in children with autism. A collection of papers extracted from systematic literature review under Erasmus+ EMBOA project", [download];
2. Technical report: Katrin D. Bartl-Pokorny, Małgorzata Pykała, Duygun Erol Barkana, Alice Baird, Hatice Köse, Tatjana Zorcec, Ben Robins, Björn W. Schuller, Agnieszka Landowska: Systematic Literature Review - Robot-Based Intervention for Children with Autism Spectrum Disorder, Technical report of ETI Faculty 1/2021, Politechnika Gdanska, 1/2021, [download];
3. Journal publication: Bartl-Pokorny, K.D., Pykała, M., Uluer, P., Barkana, D.E., Baird, A., Kose, H., Zorcec, T., Robins, B., Schuller, B.W. and Landowska, A., 2021. Robot-based intervention for children with autism spectrum disorder: a systematic literature review. IEEE Access., 2021, [download], doi: 10.1109/ACCESS.2021.3132785.

Observational study of robot-assisted intervention supported with emotion recognition technologies with data analysis

The intellectual output aimed to conduct research involving children with autism and social robots.

The observational studies revealed a number of insights into conducting therapy for children with ASD using social robots in terms of their ability to recognize emotions. The gathered information formed the basis for the development of the guidelines.

Results

1. Dataset containing annotated recordings from observations of children's interactions with the robot [download];
2. Buket Coşkun, Elif Toprak, Pınar Uluer , Duygun Erol Barkana, Hatice Köse: Analysis of the usability of the physiological data and detection of stress, Technical Report of Istanbul Technical University and Yeditepe University., 1/2022, [download];
3. Manuel Milling, Katrin D. Bartl-Pokorny, Björn W. Schuller: Investigating automatic speech emotion recognition for children with autism spectrum disorder in interactive intervention session, Technical Report of University of Augsburg, [download];
4. Agata Kołakowska, Jan Kowalina,, Agnieszka Landowska, Michal R. Wrobel, Ihar Uzun: Analyzing the emotion recognition capabilities based on video recordings of the faces of children on the autism spectrum while interacting with a social robot, Technical report of ETI Faculty, Politechnika Gdanska, 1/2022, [download];
5. Technical report: Aleksadra Karpus, Michal R. Wrobel: An analysis of the eyetracker usabibilty based on recordings of children with autism spectrum disorder while interacting with a social robot, Technical report of ETI Faculty, Politechnika Gdanska, 2/2022, [download];
6. Milling, M., A. Baird, K. D. Bartl-Pokorny, S. Liu, A. M. Alcorn, J. Shen, T. Tavassoli et al. "Evaluating the Impact of Voice Activity Detection on Speech Emotion Recognition for Autistic Children." Frontiers in Computer Science 4 (2022), [download];
7. Landowska, A. and Robins, B., 2020, April. Robot eye perspective in perceiving facial expressions in interaction with children with autism. In Workshops of the International Conference on Advanced Information Networking and Applications (pp. 1287-1297). Springer, Cham, [download];
8. Biçer, E., Takır, Ş., Gürpınar, C., Uluer, P. and Köse, H., 2022, May. Masking and Compression Techniques for Efficient Action Unit Detection of Children for Social Robots. In 2022 30th Signal Processing and Communications Applications Conference (SIU) (pp. 1-4). IEEE, [download];
9. Coşkun B., Uluer P., Toprak E., Erol Barkana D., Köse H., Zorcec, T., Robins, B., Landowska, A.,“Stress Detection of Children with Autism using Physiological Signals in Kaspar Robot-Based Intervention Studies”, 9th IEEE RAS/EMBS International Conference on Biomedical Robotics & Biomechatronics (BioRob 2022), 21-24 August, Seoul, Korea, [download];
10. Aktaş, S.N.B., Uluer, P., Coşkun, B., Toprak, E., Barkana, D.E., Köse, H., Zorcec, T., Robins, B. and Landowska, A., "Stress Detection of Children With ASD Using Physiological Signals". In 2022 30th Signal Processing and Communications Applications Conference (SIU) (pp. 1-4)., IEEE, 2022 [download].

Guidelines for application of emotion recognition algorithms in robot-supported interventions in autism

The aim of the intellectual output was to develop the guidelines on affective loop in robot-assisted intervention in children with autism.

The EMBOA project goal was to confirm the possibility of the application (feasibility study), and in particular, we aim at the identification of the best practices and obstacles in using the combination of the technologies. The lessons learned from observational studies supported by systematic literature reviews, summarized in the form of guidelines, might be used in higher education in all involved countries in robotics, computer science, and special pedagogy fields of study.

Results

1. Technical report: Duygun Erol Barkana,, Katrin D. Bartl-Pokorny, Hatice Kose, Agnieszka Landowska, Michal R. Wrobel, Ben Robins, Tatjana Zorcec: Guidelines for emotion recognition in robot-supported interventions in autism, Technical report of ETI Faculty, Politechnika Gdanska, [download: English, Turkish, Macedonian, Polish, German];
2. Technical report: Agnieszka Landowska, Michal R. Wrobel: EMBOA project evaluation report, Technical report of ETI Faculty, Politechnika Gdanska, 3/2022, [download].

Observational study of robot-assisted intervention supported with emotion recognition technologies with data analysis - phase 2 - guidelines evaluation

Observational studies have successfully validated the developed guidelines for emotion recognition in robot-supported interventions in autism.

Results

1. Updated dataset containing annotated recordings from observations of children's interactions with the robot, [download].