Affective loop in Socially Assistive Robotics as an intervention tool for children with autism
The project is complete. This site is its permanent archive.
About the project
About EMBOA project
EMBOA is an acronym for project ”Affective loop in Socially Assistive Robotics as an intervention tool for children with autism”. The project is conducted by an international consortium and aims at the development of guidelines and practical evaluation of applying emotion recognition technologies in robot-supported intervention in children with autism.
It combines three domains: autism therapy, social robots and automatic emotion recognition.
EMBOA is a mixed research and didactic project under EU Erasmus Plus Strategic Partnership for Higher Education Programme. Years: 2019-2022.
Goal
The EMBOA project goal is to confirm the possibility of the application (feasibility study) of Affective Computing in Social Robotics, and in particular, we aim at the identification of the best practices and obstacles in using the combination of these technologies. What we hope to obtain is a novel approach for creating an affective loop in child-robot interaction that would enhance interventions regarding emotional intelligence building in children with autism.
Main tasks
- Identify state-of-the-art of emotion recognition applied in human-robot interaction in autism intervention, based on systematic literature review;
- Performing feasibility study confirming the possibility of the application;
- Identification of the best practices and obstacles in using the combination of the technologies;
- Development of guidelines and practical evaluation of applying emotion recognition technologies in robot-supported intervention in children with autism;
- The guidelines will be shared in English, Polish, German, Turkish and Macedonian.
Autism therapy
Children with autism spectrum disorder (ASD) suffer from multiple deficits, and limited social and emotional skills are among those, that influence their ability to involve in interaction and communication. There are promising results in the use of robots in supporting the social and emotional development of children with autism.
We do not know, why children with autism are eager to interact with human-like looking robots and not with humans. Regardless of the reason, social robots proved to be a way to get through the social obstacles of a child and make him/her involved in the interaction. Once the interaction happens, we have a unique opportunity to engage a child in gradually building and practicing social and emotional skills.
Robot-assisted therapy in autism has been a growing area of research in recent years. Humanoid robots, e.g., Robota, Nao, Kaspar, Milo have been used with children with autism to help mediate interactions with peers and adults. The ASD practitioners expressed the robot Kaspar to be of added value to ASD objectives in domains such as communication, interpersonal interaction, social relations, and emotional wellbeing. In the project we use Kaspar robot to study child-robot interaction. Kaspar robot was not used in conjunction with emotion recognition technologies before.
Emotion recognition
There are several works on facial expressions in children with autism. Multiple studies suggest that participants with ASD display facial expressions less frequently and shortly, and they are less likely to share facial expressions with others. However, participants with ASD do not express emotions less intensely, nor is their reaction time of expression onset slower. None of the studies reported were based on interaction with a social robot, however some studies reported emotional expressions in autism analyzed with automatic recognition software. The approach of the EMBOA project is to combine social robots with automatic emotion recognition in the therapy of children with autism.
As in all Erasmus Plus projects, the findings of the project will be widely disseminated in higher education, in research, among autism therapy specialists as well as to the general public in the variety of forms:
- 2 short-term joint staff training events;
- 1 intensive program for higher education learners;
- 6 multiplier events for autism therapy professionals and caregivers in all participating countries;
- openly available guidelines;
- scientific and technical papers,
- papers, posters and presentations to general public.
Consortium
Partners of the consortium
The project consortium is multidisciplinary and combines partners with competence in interventions in autism, robotics, and automatic emotion recognition from Poland, UK, Germany, North Macedonia, and Turkey.
The University of Hertfordshire Higher Education Corporation
Macedonian Association for Applied Psychology
Results
Outcomes
Main deliverable
Guidelines for emotion recognition in robot-supported interventions in autism
Duygun Erol Barkana, Katrin D. Bartl-Pokorny, Hatice Köse, Agnieszka Landowska, Michał R. Wróbel, Ben Robins, Tatjana Zorcec. Technical report of the Faculty of ETI, Gdańsk University of Technology. Published in five languages.
Published as
Wróbel, M. R.; Bartl-Pokorny, K. D.; Zorcec, T.; Barkana, D. E.; Köse, H.; Milling, M.; Robins, B.; Schuller, B. W.; Landowska, A. “Guidelines for Emotion Recognition in Robot-Supported Interventions in Autism.” International Journal of Social Robotics 18, 68 (2026). doi:10.1007/s12369-026-01401-2
Intellectual outputs
IO 1
Systematic literature review and meta-analysis of emotion recognition in children with autism
The aim of the intellectual output was to investigate state-of-the-art in emotion recognition in children with autism.
The automatic emotion recognition domain brings new methods and technologies that might be used to enhance the therapy of children with autism. The study aimed at the exploration of methods and tools used to recognize emotions in children. It was performed as a literature review study using a systematic approach and PRISMA methodology for reporting quantitative and qualitative results. Diverse observation channels and modalities are used in the analyzed studies, including facial expressions, the prosody of speech, and physiological signals. Regarding representation models, the basic emotions are the most frequently recognized, especially happiness, fear, and sadness. Both single-channel and multichannel approaches are applied, with a preference for the first one. For multimodal recognition, early fusion was the most frequently applied. SVM and neural networks were the most popular for building classifiers. All channels are reported to be prone to some disturbance, and as a result, information on specific symptoms of emotions might be temporarily or permanently unavailable. The challenges of proper stimuli, labeling methods, and the creation of open datasets were also identified.
- Collection of papers Presentation of the collection of papers “Emotion recognition in children with autism. A collection of papers extracted from systematic literature review under Erasmus+ EMBOA project”, [download];
- Technical report Aleksandra Karpus, Agnieszka Landowska, Jakub Miler, Małgorzata Pykała: “SYSTEMATIC LITERATURE REVIEW – METHODS AND HINTS”, Technical report of ETI Faculty 1/2020, Politechnika Gdanska, 1/2020, [download];
- Journal publication Landowska, A.; Karpus, A.; Zawadzka, T.; Robins, B.; Erol Barkana, D.; Kose, H.; Zorcec, T.; Cummins, N. Automatic Emotion Recognition in Children with Autism: A Systematic Literature Review. Sensors 2022, 22, 1649, [download], https://doi.org/10.3390/s22041649.
IO 2
Systematic literature review and meta-analysis of interventions regarding Robots and emotional skills in children with autism
The aim of the intellectual output was to explore the latest developments in social robots in intervention for children with autism.
Children with autism spectrum disorder (ASD) have deficits in the socio-communicative domain and frequently face severe difficulties in the recognition and expression of emotions. Existing literature suggested that children with ASD benefit from robot-based interventions. However, studies varied considerably in participant characteristics, applied robots, and trained skills. Nao and ZECA were the most frequently used robots; recognition of basic emotions and getting into interaction were the most frequently trained skills, while happiness, sadness, fear, and anger were the most frequently trained emotions. The studies reported a wide range of challenges with respect to robot-based intervention, ranging from limitations for certain ASD subgroups and security aspects of the robots to efforts regarding the automatic recognition of the children’s emotional state by the robotic systems. Finally, we summarised and discussed recommendations regarding the application of robot-based interventions for children with ASD.
- Collection of papers Presentation of the collection of papers “Emotions in robot-based interventions in children with autism. A collection of papers extracted from systematic literature review under Erasmus+ EMBOA project”, [download];
- Technical report Katrin D. Bartl-Pokorny, Małgorzata Pykała, Duygun Erol Barkana, Alice Baird, Hatice Köse, Tatjana Zorcec, Ben Robins, Björn W. Schuller, Agnieszka Landowska: Systematic Literature Review - Robot-Based Intervention for Children with Autism Spectrum Disorder, Technical report of ETI Faculty 1/2021, Politechnika Gdanska, 1/2021, [download];
- Journal publication Bartl-Pokorny, K.D., Pykała, M., Uluer, P., Barkana, D.E., Baird, A., Kose, H., Zorcec, T., Robins, B., Schuller, B.W. and Landowska, A., 2021. Robot-based intervention for children with autism spectrum disorder: a systematic literature review. IEEE Access., 2021, [download], doi: 10.1109/ACCESS.2021.3132785.
IO 3
Observational study of robot-assisted intervention supported with emotion recognition technologies with data analysis
The intellectual output aimed to conduct research involving children with autism and social robots.
The observational studies revealed a number of insights into conducting therapy for children with ASD using social robots in terms of their ability to recognize emotions. The gathered information formed the basis for the development of the guidelines.
- Dataset Dataset containing annotated recordings from observations of children’s interactions with the robot [download];
- Technical report Buket Coşkun, Elif Toprak, Pınar Uluer, Duygun Erol Barkana, Hatice Köse: Analysis of the usability of the physiological data and detection of stress, Technical Report of Istanbul Technical University and Yeditepe University., 1/2022, [download];
- Technical report Manuel Milling, Katrin D. Bartl-Pokorny, Björn W. Schuller: Investigating automatic speech emotion recognition for children with autism spectrum disorder in interactive intervention session, Technical Report of University of Augsburg, [download];
- Technical report Agata Kołakowska, Jan Kowalina, Agnieszka Landowska, Michal R. Wrobel, Ihar Uzun: Analyzing the emotion recognition capabilities based on video recordings of the faces of children on the autism spectrum while interacting with a social robot, Technical report of ETI Faculty, Politechnika Gdanska, 1/2022, [download];
- Technical report Aleksandra Karpus, Michal R. Wrobel: An analysis of the eyetracker usability based on recordings of children with autism spectrum disorder while interacting with a social robot, Technical report of ETI Faculty, Politechnika Gdanska, 2/2022, [download];
- Journal publication Milling, M., A. Baird, K. D. Bartl-Pokorny, S. Liu, A. M. Alcorn, J. Shen, T. Tavassoli et al. “Evaluating the Impact of Voice Activity Detection on Speech Emotion Recognition for Autistic Children.” Frontiers in Computer Science 4 (2022), [download];
- Conference paper Landowska, A. and Robins, B., 2020, April. Robot eye perspective in perceiving facial expressions in interaction with children with autism. In Workshops of the International Conference on Advanced Information Networking and Applications (pp. 1287-1297). Springer, Cham, [download];
- Conference paper Biçer, E., Takır, Ş., Gürpınar, C., Uluer, P. and Köse, H., 2022, May. Masking and Compression Techniques for Efficient Action Unit Detection of Children for Social Robots. In 2022 30th Signal Processing and Communications Applications Conference (SIU) (pp. 1-4). IEEE, [download];
- Conference paper Coşkun B., Uluer P., Toprak E., Erol Barkana D., Köse H., Zorcec, T., Robins, B., Landowska, A., “Stress Detection of Children with Autism using Physiological Signals in Kaspar Robot-Based Intervention Studies”, 9th IEEE RAS/EMBS International Conference on Biomedical Robotics & Biomechatronics (BioRob 2022), 21-24 August, Seoul, Korea, [download];
- Conference paper Aktaş, S.N.B., Uluer, P., Coşkun, B., Toprak, E., Barkana, D.E., Köse, H., Zorcec, T., Robins, B. and Landowska, A., “Stress Detection of Children With ASD Using Physiological Signals”. In 2022 30th Signal Processing and Communications Applications Conference (SIU) (pp. 1-4)., IEEE, 2022 [download].
IO 5
Guidelines for application of emotion recognition algorithms in robot-supported interventions in autism
The aim of the intellectual output was to develop the guidelines on affective loop in robot-assisted intervention in children with autism.
The EMBOA project goal was to confirm the possibility of the application (feasibility study), and in particular, we aim at the identification of the best practices and obstacles in using the combination of the technologies. The lessons learned from observational studies supported by systematic literature reviews, summarized in the form of guidelines, might be used in higher education in all involved countries in robotics, computer science, and special pedagogy fields of study.
- Technical report Duygun Erol Barkana, Katrin D. Bartl-Pokorny, Hatice Kose, Agnieszka Landowska, Michal R. Wrobel, Ben Robins, Tatjana Zorcec: Guidelines for emotion recognition in robot-supported interventions in autism, Technical report of ETI Faculty, Politechnika Gdanska, [download: English, Turkish, Macedonian, Polish, German];
- Technical report Agnieszka Landowska, Michal R. Wrobel: EMBOA project evaluation report, Technical report of ETI Faculty, Politechnika Gdanska, 3/2022, [download].
- Journal publication Barkana DE, Bartl-Pokorny KD, Kose H, Landowska A, Milling M, Robins B, Schuller BW, Uluer P, Wróbel MR, Zorcec T. Challenges in Observing the Emotions of Children with Autism Interacting with a Social Robot. Int J of Soc Robotics 16, 2261–2276 (2024). [download]
- Journal publication Wróbel MR, Bartl-Pokorny KD, Zorcec T, Barkana DE, Kose H, Milling M, Robins B, Schuller BW, Landowska A. Guidelines for Emotion Recognition in Robot-Supported Interventions in Autism. Int J of Soc Robotics 18, 68 (2026). [download]
IO 6
Observational study of robot-assisted intervention supported with emotion recognition technologies with data analysis — phase 2 — guidelines evaluation
Observational studies have successfully validated the developed guidelines for emotion recognition in robot-supported interventions in autism.
- Dataset Updated dataset containing annotated recordings from observations of children’s interactions with the robot, [download].
Dissemination activities and trainings
- September 2019 Staff training event (LTT1) on affective computing technologies was held in Gdansk, Poland in September 2019. See gallery for more information.
- March 2020 Staff training event (LTT2) on social robot Kaspar was held in Hatfield, UK in March 2020. See gallery for more information.
- April/May 2022 Students training event (LTT3) enabling the sharing of knowledge developed within the project and enriching the curricula of selected majors took place in April/May 2022 in Gdansk, Poland. See gallery for more information.
- June–August 2022 Multiplier events (seminars) to disseminate the results of the project were held at the end of the project, from June to August in 2022, in each partner country. See gallery about meetings in Turkey, Poland.
Gallery
Meetings, trainings and observations
Eighty photographs from the project’s staff and student trainings, the observational studies with the Kaspar robot, and the multiplier events held in each partner country. Select a photo to view it full size.